  Digital pathology has significantly advanced disease detection and
pathologist efficiency through the analysis of gigapixel whole-slide images
(WSI). In this process, WSIs are first divided into patches, for which a
feature extractor model is applied to obtain feature vectors, which are
subsequently processed by an aggregation model to predict the respective WSI
label. With the rapid evolution of representation learning, numerous new
feature extractor models, often termed foundational models, have emerged.
Traditional evaluation methods rely on a static downstream aggregation model
setup, encompassing a fixed architecture and hyperparameters, a practice we
identify as potentially biasing the results. Our study uncovers a sensitivity
of feature extractor models towards aggregation model configurations,
indicating that performance comparability can be skewed based on the chosen
configurations. By accounting for this sensitivity, we find that the
performance of many current feature extractor models is notably similar. We
support this insight by evaluating seven feature extractor models across three
different datasets with 162 different aggregation model configurations. This
comprehensive approach provides a more nuanced understanding of the feature
extractors' sensitivity to various aggregation model configurations, leading to
a fairer and more accurate assessment of new foundation models in digital
pathology.
