  The notion of task similarity is at the core of various machine learning
paradigms, such as domain adaptation and meta-learning. Current methods to
quantify it are often heuristic, make strong assumptions on the label sets
across the tasks, and many are architecture-dependent, relying on task-specific
optimal parameters (e.g., require training a model on each dataset). In this
work we propose an alternative notion of distance between datasets that (i) is
model-agnostic, (ii) does not involve training, (iii) can compare datasets even
if their label sets are completely disjoint and (iv) has solid theoretical
footing. This distance relies on optimal transport, which provides it with rich
geometry awareness, interpretable correspondences and well-understood
properties. Our results show that this novel distance provides meaningful
comparison of datasets, and correlates well with transfer learning hardness
across various experimental settings and datasets.
