  Federated Learning is increasingly used in domains such as healthcare to
facilitate collaborative model training without data-sharing. However, datasets
located in different sites are often non-identically distributed, leading to
degradation of model performance in FL. Most existing methods for assessing
these distribution shifts are limited by being dataset or task-specific.
Moreover, these metrics can only be calculated by exchanging data, a practice
restricted in many FL scenarios. To address these challenges, we propose a
novel metric for assessing dataset similarity. Our metric exhibits several
desirable properties for FL: it is dataset-agnostic, is calculated in a
privacy-preserving manner, and is computationally efficient, requiring no model
training. In this paper, we first establish a theoretical connection between
our metric and training dynamics in FL. Next, we extensively evaluate our
metric on a range of datasets including synthetic, benchmark, and medical
imaging datasets. We demonstrate that our metric shows a robust and
interpretable relationship with model performance and can be calculated in
privacy-preserving manner. As the first federated dataset similarity metric, we
believe this metric can better facilitate successful collaborations between
sites.
