  Scarcity of labeled histopathology data limits the applicability of deep
learning methods to under-profiled cancer types and labels. Transfer learning
allows researchers to overcome the limitations of small datasets by
pre-training machine learning models on larger datasets similar to the small
target dataset. However, similarity between datasets is often determined
heuristically. In this paper, we propose a principled notion of distance
between histopathology datasets based on a hierarchical generalization of
optimal transport distances. Our method does not require any training, is
agnostic to model type, and preserves much of the hierarchical structure in
histopathology datasets imposed by tiling. We apply our method to H&E stained
slides from The Cancer Genome Atlas from six different cancer types. We show
that our method outperforms a baseline distance in a cancer-type prediction
task. Our results also show that our optimal transport distance predicts
difficulty of transferability in a tumor vs.normal prediction setting.
