  Explainability and uncertainty quantification are two pillars of trustable
artificial intelligence. However, the reasoning behind uncertainty estimates is
generally left unexplained. Identifying the drivers of uncertainty complements
explanations of point predictions in recognizing model limitations and enhances
trust in decisions and their communication. So far, explanations of
uncertainties have been rarely studied. The few exceptions rely on Bayesian
neural networks or technically intricate approaches, such as auxiliary
generative models, thereby hindering their broad adoption. We present a simple
approach to explain predictive aleatoric uncertainties. We estimate uncertainty
as predictive variance by adapting a neural network with a Gaussian output
distribution. Subsequently, we apply out-of-the-box explainers to the model's
variance output. This approach can explain uncertainty influences more reliably
than literature baselines, which we evaluate in a synthetic setting with a
known data-generating process. We further adapt multiple metrics from
conventional XAI research to uncertainty explanations. We quantify our findings
with a nuanced benchmark analysis that includes real-world datasets. Finally,
we apply our approach to an age regression model and discover reasonable
sources of uncertainty. Overall, we explain uncertainty estimates with little
modifications to the model architecture and demonstrate that our approach
competes effectively with more intricate methods.
