  It has been observed \citep{zhang2016understanding} that deep neural networks
can memorize: they achieve 100\% accuracy on training data. Recent theoretical
results explained such behavior in highly overparametrized regimes, where the
number of neurons in each layer is larger than the number of training samples.
In this paper, we show that neural networks can be trained to memorize training
data perfectly in a mildly overparametrized regime, where the number of
parameters is just a constant factor more than the number of training samples,
and the number of neurons is much smaller.
