  The goal of data attribution for text-to-image models is to identify the
training images that most influence the generation of a new image. We can
define "influence" by saying that, for a given output, if a model is retrained
from scratch without that output's most influential images, the model should
then fail to generate that output image. Unfortunately, directly searching for
these influential images is computationally infeasible, since it would require
repeatedly retraining from scratch. We propose a new approach that efficiently
identifies highly-influential images. Specifically, we simulate unlearning the
synthesized image, proposing a method to increase the training loss on the
output image, without catastrophic forgetting of other, unrelated concepts.
Then, we find training images that are forgotten by proxy, identifying ones
with significant loss deviations after the unlearning process, and label these
as influential. We evaluate our method with a computationally intensive but
"gold-standard" retraining from scratch and demonstrate our method's advantages
over previous methods.
