  Spectral bias is an important observation of neural network training, stating
that the network will learn a low frequency representation of the target
function before converging to higher frequency components. This property is
interesting due to its link to good generalization in over-parameterized
networks. However, in low dimensional settings, a severe spectral bias occurs
that obstructs convergence to high frequency components entirely. In order to
overcome this limitation, one can encode the inputs using a high frequency
sinusoidal encoding. Previous works attempted to explain this phenomenon using
Neural Tangent Kernel (NTK) and Fourier analysis. However, NTK does not capture
real network dynamics, and Fourier analysis only offers a global perspective on
the network properties that induce this bias. In this paper, we provide a novel
approach towards understanding spectral bias by directly studying ReLU MLP
training dynamics. Specifically, we focus on the connection between the
computations of ReLU networks (activation regions), and the speed of gradient
descent convergence. We study these dynamics in relation to the spatial
information of the signal to understand how they influence spectral bias. We
then use this formulation to study the severity of spectral bias in low
dimensional settings, and how positional encoding overcomes this.
