  Autonomous driving is challenging in adverse road and weather conditions in
which there might not be lane lines, the road might be covered in snow and the
visibility might be poor. We extend the previous work on end-to-end learning
for autonomous steering to operate in these adverse real-life conditions with
multimodal data. We collected 28 hours of driving data in several road and
weather conditions and trained convolutional neural networks to predict the car
steering wheel angle from front-facing color camera images and lidar range and
reflectance data. We compared the CNN model performances based on the different
modalities and our results show that the lidar modality improves the
performances of different multimodal sensor-fusion models. We also performed
on-road tests with different models and they support this observation.
