  Analysis of 3D segmentation models, especially in the context of medical
imaging, is often limited to segmentation performance metrics that overlook the
crucial aspect of explainability and bias. Currently, effectively explaining
these models with saliency maps is challenging due to the high dimensions of
input images multiplied by the ever-growing number of segmented class labels.
To this end, we introduce Agg^2Exp, a methodology for aggregating fine-grained
voxel attributions of the segmentation model's predictions. Unlike classical
explanation methods that primarily focus on the local feature attribution,
Agg^2Exp enables a more comprehensive global view on the importance of
predicted segments in 3D images. Our benchmarking experiments show that
gradient-based voxel attributions are more faithful to the model's predictions
than perturbation-based explanations. As a concrete use-case, we apply Agg^2Exp
to discover knowledge acquired by the Swin UNEt TRansformer model trained on
the TotalSegmentator v2 dataset for segmenting anatomical structures in
computed tomography medical images. Agg^2Exp facilitates the explanatory
analysis of large segmentation models beyond their predictive performance.
