  Perception systems for autonomous driving have seen significant advancements
in their performance over last few years. However, these systems struggle to
show robustness in extreme weather conditions because sensors like lidars and
cameras, which are the primary sensors in a sensor suite, see a decline in
performance under these conditions. In order to solve this problem,
camera-radar fusion systems provide a unique opportunity for all weather
reliable high quality perception. Cameras provides rich semantic information
while radars can work through occlusions and in all weather conditions. In this
work, we show that the state-of-the-art fusion methods perform poorly when
camera input is degraded, which essentially results in losing the all-weather
reliability they set out to achieve. Contrary to these approaches, we propose a
new method, RadSegNet, that uses a new design philosophy of independent
information extraction and truly achieves reliability in all conditions,
including occlusions and adverse weather. We develop and validate our proposed
system on the benchmark Astyx dataset and further verify these results on the
RADIATE dataset. When compared to state-of-the-art methods, RadSegNet achieves
a 27% improvement on Astyx and 41.46% increase on RADIATE, in average precision
score and maintains a significantly better performance in adverse weather
conditions
