  The primary focus of autonomous driving research is to improve driving
accuracy. While great progress has been made, state-of-the-art algorithms still
fail at times. Such failures may have catastrophic consequences. It therefore
is important that automated cars foresee problems ahead as early as possible.
This is also of paramount importance if the driver will be asked to take over.
We conjecture that failures do not occur randomly. For instance, driving models
may fail more likely at places with heavy traffic, at complex intersections,
and/or under adverse weather/illumination conditions. This work presents a
method to learn to predict the occurrence of these failures, i.e. to assess how
difficult a scene is to a given driving model and to possibly give the human
driver an early headsup. A camera-based driving model is developed and trained
over real driving datasets. The discrepancies between the model's predictions
and the human `ground-truth' maneuvers were then recorded, to yield the
`failure' scores. Experimental results show that the failure score can indeed
be learned and predicted. Thus, our prediction method is able to improve the
overall safety of an automated driving model by alerting the human driver
timely, leading to better human-vehicle collaborative driving.
