  We study the problem of (learning) algorithm comparison, where the goal is to
find differences between models trained with two different learning algorithms.
We begin by formalizing this goal as one of finding distinguishing feature
transformations, i.e., input transformations that change the predictions of
models trained with one learning algorithm but not the other. We then present
ModelDiff, a method that leverages the datamodels framework (Ilyas et al.,
2022) to compare learning algorithms based on how they use their training data.
We demonstrate ModelDiff through three case studies, comparing models trained
with/without data augmentation, with/without pre-training, and with different
SGD hyperparameters. Our code is available at
https://github.com/MadryLab/modeldiff .
