  The goal of data attribution is to trace model predictions back to training
data. Despite a long line of work towards this goal, existing approaches to
data attribution tend to force users to choose between computational
tractability and efficacy. That is, computationally tractable methods can
struggle with accurately attributing model predictions in non-convex settings
(e.g., in the context of deep neural networks), while methods that are
effective in such regimes require training thousands of models, which makes
them impractical for large models or datasets.
  In this work, we introduce TRAK (Tracing with the Randomly-projected After
Kernel), a data attribution method that is both effective and computationally
tractable for large-scale, differentiable models. In particular, by leveraging
only a handful of trained models, TRAK can match the performance of attribution
methods that require training thousands of models. We demonstrate the utility
of TRAK across various modalities and scales: image classifiers trained on
ImageNet, vision-language models (CLIP), and language models (BERT and mT5). We
provide code for using TRAK (and reproducing our work) at
https://github.com/MadryLab/trak .
