  Image-based brain cancer prediction models, based on radiomics, quantify the
radiologic phenotype from magnetic resonance imaging (MRI). However, these
features are difficult to reproduce because of variability in acquisition and
preprocessing pipelines. Despite evidence of intra-tumor phenotypic
heterogeneity, the spatial diversity between different slices within an MRI
scan has been relatively unexplored using such methods. In this work, we
propose a deep anchor attention aggregation strategy with a Vision Transformer
to predict survival risk for brain cancer patients. A Deep Anchor Attention
Learning (DAAL) algorithm is proposed to assign different weights to
slice-level representations with trainable distance measurements. We evaluated
our method on N = 326 MRIs. Our results outperformed attention multiple
instance learning-based techniques. DAAL highlights the importance of critical
slices and corroborates the clinical intuition that inter-slice spatial
diversity can reflect disease severity and is implicated in outcome.
