  Deep learning-based models are widely deployed in autonomous driving areas,
especially the increasingly noticed end-to-end solutions. However, the
black-box property of these models raises concerns about their trustworthiness
and safety for autonomous driving, and how to debug the causality has become a
pressing concern. Despite some existing research on the explainability of
autonomous driving, there is currently no systematic solution to help
researchers debug and identify the key factors that lead to the final predicted
action of end-to-end autonomous driving. In this work, we propose a
comprehensive approach to explore and analyze the causality of end-to-end
autonomous driving. First, we validate the essential information that the final
planning depends on by using controlled variables and counterfactual
interventions for qualitative analysis. Then, we quantitatively assess the
factors influencing model decisions by visualizing and statistically analyzing
the response of key model inputs. Finally, based on the comprehensive study of
the multi-factorial end-to-end autonomous driving system, we have developed a
strong baseline and a tool for exploring causality in the close-loop simulator
CARLA. It leverages the essential input sources to obtain a well-designed
model, resulting in highly competitive capabilities. As far as we know, our
work is the first to unveil the mystery of end-to-end autonomous driving and
turn the black box into a white one. Thorough close-loop experiments
demonstrate that our method can be applied to end-to-end autonomous driving
solutions for causality debugging. Code will be available at
https://github.com/bdvisl/DriveInsight.
