  We construct an experimental setup in which changing the scale of
initialization strongly impacts the implicit regularization induced by SGD,
interpolating from good generalization performance to completely memorizing the
training set while making little progress on the test set. Moreover, we find
that the extent and manner in which generalization ability is affected depends
on the activation and loss function used, with $\sin$ activation demonstrating
extreme memorization. In the case of the homogeneous ReLU activation, we show
that this behavior can be attributed to the loss function. Our empirical
investigation reveals that increasing the scale of initialization correlates
with misalignment of representations and gradients across examples in the same
class. This insight allows us to devise an alignment measure over gradients and
representations which can capture this phenomenon. We demonstrate that our
alignment measure correlates with generalization of deep models trained on
image classification tasks.
