  One of the fundamental challenges in the design of perception systems for
autonomous vehicles is validating the performance of each algorithm under a
comprehensive variety of operating conditions. In the case of vision-based
semantic segmentation, there are known issues when encountering new scenarios
that are sufficiently different to the training data. In addition, even small
variations in environmental conditions such as illumination and precipitation
can affect the classification performance of the segmentation model. Given the
reliance on visual information, these effects often translate into poor
semantic pixel classification which can potentially lead to catastrophic
consequences when driving autonomously. This paper presents a novel method for
analysing the robustness of semantic segmentation models and provides a number
of metrics to evaluate the classification performance over a variety of
environmental conditions. The process incorporates an additional sensor (lidar)
to automate the process, eliminating the need for labour-intensive hand
labelling of validation data. The system integrity can be monitored as the
performance of the vision sensors are validated against a different sensor
modality. This is necessary for detecting failures that are inherent to vision
technology. Experimental results are presented based on multiple datasets
collected at different times of the year with different environmental
conditions. These results show that the semantic segmentation performance
varies depending on the weather, camera parameters, existence of shadows, etc..
The results also demonstrate how the metrics can be used to compare and
validate the performance after making improvements to a model, and compare the
performance of different networks.
