  Learning dynamics, which describes how the learning of specific training
examples influences the model's prediction of other examples, give us a
powerful tool for understanding the behavior of deep learning systems. We study
the learning dynamics of large language models during finetuning, by analyzing
the step-wise decomposition and accumulated influence among different
responses. Our framework allows a uniform interpretation of many interesting
observations about the training of popular algorithms for both instruction
tuning and preference tuning. The analysis not only explains where the benefits
of these methods come from but also inspires a simple, effective method to
further improve the alignment performance. Code for experiments is available at
https://github.com/Joshua-Ren/Learning_dynamics_LLM.
