  This paper explores the connection between learning trajectories of Deep
Neural Networks (DNNs) and their generalization capabilities when optimized
using (stochastic) gradient descent algorithms. Instead of concentrating solely
on the generalization error of the DNN post-training, we present a novel
perspective for analyzing generalization error by investigating the
contribution of each update step to the change in generalization error. This
perspective allows for a more direct comprehension of how the learning
trajectory influences generalization error. Building upon this analysis, we
propose a new generalization bound that incorporates more extensive trajectory
information. Our proposed generalization bound depends on the complexity of
learning trajectory and the ratio between the bias and diversity of training
set. Experimental findings reveal that our method effectively captures the
generalization error throughout the training process. Furthermore, our approach
can also track changes in generalization error when adjustments are made to
learning rates and label noise levels. These results demonstrate that learning
trajectory information is a valuable indicator of a model's generalization
capabilities.
