  Multi-stage training and knowledge transfer, from a large-scale pretraining
task to various finetuning tasks, have revolutionized natural language
processing and computer vision resulting in state-of-the-art performance
improvements. In this paper, we develop a multi-stage influence function score
to track predictions from a finetuned model all the way back to the pretraining
data. With this score, we can identify the pretraining examples in the
pretraining task that contribute most to a prediction in the finetuning task.
The proposed multi-stage influence function generalizes the original influence
function for a single model in (Koh & Liang, 2017), thereby enabling influence
computation through both pretrained and finetuned models. We study two
different scenarios with the pretrained embeddings fixed or updated in the
finetuning tasks. We test our proposed method in various experiments to show
its effectiveness and potential applications.
