  Vision Transformers are at the heart of the current surge of interest in
foundation models for histopathology. They process images by breaking them into
smaller patches following a regular grid, regardless of their content. Yet, not
all parts of an image are equally relevant for its understanding. This is
particularly true in computational pathology where background is completely
non-informative and may introduce artefacts that could mislead predictions. To
address this issue, we propose a novel method that explicitly masks background
in Vision Transformers' attention mechanism. This ensures tokens corresponding
to background patches do not contribute to the final image representation,
thereby improving model robustness and interpretability. We validate our
approach using prostate cancer grading from whole-slide images as a case study.
Our results demonstrate that it achieves comparable performance with plain
self-attention while providing more accurate and clinically meaningful
attention heatmaps.
