  Mechanistic interpretability has been explored in detail for large language
models (LLMs). For the first time, we provide a preliminary investigation with
similar interpretability methods for medical imaging. Specifically, we analyze
the features from a ViT-Small encoder obtained from a pathology Foundation
Model via application to two datasets: one dataset of pathology images, and one
dataset of pathology images paired with spatial transcriptomics. We discover an
interpretable representation of cell and tissue morphology, along with gene
expression within the model embedding space. Our work paves the way for further
exploration around interpretable feature dimensions and their utility for
medical and clinical applications.
