  This paper focuses on predicting the occurrence of grokking in neural
networks, a phenomenon in which perfect generalization emerges long after signs
of overfitting or memorization are observed. It has been reported that grokking
can only be observed with certain hyper-parameters. This makes it critical to
identify the parameters that lead to grokking. However, since grokking occurs
after a large number of epochs, searching for the hyper-parameters that lead to
it is time-consuming. In this paper, we propose a low-cost method to predict
grokking without training for a large number of epochs. In essence, by studying
the learning curve of the first few epochs, we show that one can predict
whether grokking will occur later on. Specifically, if certain oscillations
occur in the early epochs, one can expect grokking to occur if the model is
trained for a much longer period of time. We propose using the spectral
signature of a learning curve derived by applying the Fourier transform to
quantify the amplitude of low-frequency components to detect the presence of
such oscillations. We also present additional experiments aimed at explaining
the cause of these oscillations and characterizing the loss landscape.
