  Expressiveness and generalization of deep models was recently addressed via
the connection between neural networks (NNs) and kernel learning, where
first-order dynamics of NN during a gradient-descent (GD) optimization were
related to gradient similarity kernel, also known as Neural Tangent Kernel
(NTK). In the majority of works this kernel is considered to be time-invariant,
with its properties being defined entirely by NN architecture and independent
of the learning task at hand. In contrast, in this paper we empirically explore
these properties along the optimization and show that in practical applications
the NTK changes in a very dramatic and meaningful way, with its top
eigenfunctions aligning toward the target function learned by NN. Moreover,
these top eigenfunctions serve as basis functions for NN output - a function
represented by NN is spanned almost completely by them for the entire
optimization process. Further, since the learning along top eigenfunctions is
typically fast, their alignment with the target function improves the overall
optimization performance. In addition, we study how the neural spectrum is
affected by learning rate decay, typically done by practitioners, showing
various trends in the kernel behavior. We argue that the presented phenomena
may lead to a more complete theoretical understanding behind NN learning.
