  Panoptic Driving Perception (PDP) is critical for the autonomous navigation
of Unmanned Surface Vehicles (USVs). A PDP model typically integrates multiple
tasks, necessitating the simultaneous and robust execution of various
perception tasks to facilitate downstream path planning. The fusion of visual
and radar sensors is currently acknowledged as a robust and cost-effective
approach. However, most existing research has primarily focused on fusing
visual and radar features dedicated to object detection or utilizing a shared
feature space for multiple tasks, neglecting the individual representation
differences between various tasks. To address this gap, we propose a pair of
Asymmetric Fair Fusion (AFF) modules with favorable explainability designed to
efficiently interact with independent features from both visual and radar
modalities, tailored to the specific requirements of object detection and
semantic segmentation tasks. The AFF modules treat image and radar maps as
irregular point sets and transform these features into a crossed-shared feature
space for multitasking, ensuring equitable treatment of vision and radar point
cloud features. Leveraging AFF modules, we propose a novel and efficient PDP
model, ASY-VRNet, which processes image and radar features based on irregular
super-pixel point sets. Additionally, we propose an effective multitask
learning method specifically designed for PDP models. Compared to other
lightweight models, ASY-VRNet achieves state-of-the-art performance in object
detection, semantic segmentation, and drivable-area segmentation on the
WaterScenes benchmark. Our project is publicly available at
https://github.com/GuanRunwei/ASY-VRNet.
