  This paper prescribes a distance between learning tasks modeled as joint
distributions on data and labels. Using tools in information geometry, the
distance is defined to be the length of the shortest weight trajectory on a
Riemannian manifold as a classifier is fitted on an interpolated task. The
interpolated task evolves from the source to the target task using an optimal
transport formulation. This distance, which we call the "coupled transfer
distance" can be compared across different classifier architectures. We develop
an algorithm to compute the distance which iteratively transports the marginal
on the data of the source task to that of the target task while updating the
weights of the classifier to track this evolving data distribution. We develop
theory to show that our distance captures the intuitive idea that a good
transfer trajectory is the one that keeps the generalization gap small during
transfer, in particular at the end on the target task. We perform thorough
empirical validation and analysis across diverse image classification datasets
to show that the coupled transfer distance correlates strongly with the
difficulty of fine-tuning.
