Description
The complementary nature of radar and camera sensors offers a promising way to overcome the current limitations of robotic perception, particularly in poorly structured environments or those subject to weather disturbances. Although this synergy is recognized, research still focuses primarily on optimizing perception algorithms or decision-making systems, relegating the joint use of these modalities to the background. However, radar, thanks to its intrinsic robustness in adverse weather conditions, remains an ideal candidate for advanced integration with the rich semantic data provided by cameras.
This thesis proposes an innovative approach to multimodal fusion, exploiting the full radar spectrum — which is much more informative than point clouds alone — and the unification of spatial representations on the radar plane (aerial view, or Bird's Eye View). The latter offers the advantage of metric consistency (preserved distances and object sizes), which is particularly well suited to critical autonomous navigation tasks, where the perspective view of cameras introduces distortions that depend on the point of view.
The main challenge lies in the fundamental incompatibility of the coordinate systems and spatial interpretations between the two sensors: the camera produces an optical projection aligned with a vertical plane, while the radar generates a horizontal orthogonal representation. Existing work circumvents this problem by merging radar point clouds onto camera images (via deep neural networks), thereby optimizing object detection in perspective view. However, this approach limits applications to visual tasks and neglects the potential of the radar plane for localization, mapping, or semantic segmentation missions — areas that remain largely unexplored in this multimodal context.
The contributions of this work are therefore structured around two main themes: the use of raw radar spectrum, which carries latent information (speed, reflectivity, noise) that is absent from traditional point clouds; and the unification of modalities on the radar field, enabling consistent fusion for extended applications (object detection, segmentation, navigation).
These advances should significantly improve the performance of perception systems, while opening up prospects for tasks that have been underutilized until now, such as multimodal semantic segmentation or dynamic adaptation to complex environments.