  Identifying the training datasets that influence a language model's outputs
is essential for minimizing the generation of harmful content and enhancing its
performance. Ideally, we can measure the influence of each dataset by removing
it from training; however, it is prohibitively expensive to retrain a model
multiple times. This paper presents UnTrac: unlearning traces the influence of
a training dataset on the model's performance. UnTrac is extremely simple; each
training dataset is unlearned by gradient ascent, and we evaluate how much the
model's predictions change after unlearning. Furthermore, we propose a more
scalable approach, UnTrac-Inv, which unlearns a test dataset and evaluates the
unlearned model on training datasets. UnTrac-Inv resembles UnTrac, while being
efficient for massive training datasets. In the experiments, we examine if our
methods can assess the influence of pretraining datasets on generating toxic,
biased, and untruthful content. Our methods estimate their influence much more
accurately than existing methods while requiring neither excessive memory space
nor multiple checkpoints.
