  In Autonomous Driving (AD) transparency and safety are paramount, as mistakes
are costly. However, neural networks used in AD systems are generally
considered black boxes. As a countermeasure, we have methods of explainable AI
(XAI), such as feature relevance estimation and dimensionality reduction.
Coarse graining techniques can also help reduce dimensionality and find
interpretable global patterns. A specific coarse graining method is
Renormalization Groups from statistical physics. It has previously been applied
to Restricted Boltzmann Machines (RBMs) to interpret unsupervised learning. We
refine this technique by building a transparent backbone model for
convolutional variational autoencoders (VAE) that allows mapping latent values
to input features and has performance comparable to trained black box VAEs.
Moreover, we propose a custom feature map visualization technique to analyze
the internal convolutional layers in the VAE to explain internal causes of poor
reconstruction that may lead to dangerous traffic scenarios in AD applications.
In a second key contribution, we propose explanation and evaluation techniques
for the internal dynamics and feature relevance of prediction networks. We test
a long short-term memory (LSTM) network in the computer vision domain to
evaluate the predictability and in future applications potentially safety of
prediction models. We showcase our methods by analyzing a VAE-LSTM world model
that predicts pedestrian perception in an urban traffic situation.
