  We present a neural network architecture designed to naturally learn a
positional embedding and overcome the spectral bias towards lower frequencies
faced by conventional implicit neural representation networks. Our proposed
architecture, SPDER, is a simple MLP that uses an activation function composed
of a sinusoidal multiplied by a sublinear function, called the damping
function. The sinusoidal enables the network to automatically learn the
positional embedding of an input coordinate while the damping passes on the
actual coordinate value by preventing it from being projected down to within a
finite range of values. Our results indicate that SPDERs speed up training by
10x and converge to losses 1,500-50,000x lower than that of the
state-of-the-art for image representation. SPDER is also state-of-the-art in
audio representation. The superior representation capability allows SPDER to
also excel on multiple downstream tasks such as image super-resolution and
video frame interpolation. We provide intuition as to why SPDER significantly
improves fitting compared to that of other INR methods while requiring no
hyperparameter tuning or preprocessing.
