  Vision Transformers (ViTs) have achieved state-of-the-art results on various
computer vision tasks, including 3D object detection. However, their end-to-end
implementation also makes ViTs less explainable, which can be a challenge for
deploying them in safety-critical applications, such as autonomous driving,
where it is important for authorities, developers, and users to understand the
model's reasoning behind its predictions. In this paper, we propose a novel
method for generating saliency maps for a DetR-like ViT with multiple camera
inputs used for 3D object detection. Our method is based on the raw attention
and is more efficient than gradient-based methods. We evaluate the proposed
method on the nuScenes dataset using extensive perturbation tests and show that
it outperforms other explainability methods in terms of visual quality and
quantitative metrics. We also demonstrate the importance of aggregating
attention across different layers of the transformer. Our work contributes to
the development of explainable AI for ViTs, which can help increase trust in AI
applications by establishing more transparency regarding the inner workings of
AI models.
