  Understanding the training dynamics of deep neural networks is challenging
due to their high-dimensional nature and intricate loss landscapes. Recent
studies have revealed that, along the training trajectory, the gradient
approximately aligns with a low-rank top eigenspace of the training loss
Hessian, referred to as the dominant subspace. Given this alignment, this paper
explores whether neural networks can be trained within the dominant subspace,
which, if feasible, could lead to more efficient training methods. Our primary
observation is that when the SGD update is projected onto the dominant
subspace, the training loss does not decrease further. This suggests that the
observed alignment between the gradient and the dominant subspace is spurious.
Surprisingly, projecting out the dominant subspace proves to be just as
effective as the original update, despite removing the majority of the original
update component. Similar observations are made for the large learning rate
regime (also known as Edge of Stability) and Sharpness-Aware Minimization. We
discuss the main causes and implications of this spurious alignment, shedding
light on the intricate dynamics of neural network training.
