  We investigate the ability of deep neural networks to identify the support of
the target function. Our findings reveal that mini-batch SGD effectively learns
the support in the first layer of the network by shrinking to zero the weights
associated with irrelevant components of input. In contrast, we demonstrate
that while vanilla GD also approximates the target function, it requires an
explicit regularization term to learn the support in the first layer. We prove
that this property of mini-batch SGD is due to a second-order implicit
regularization effect which is proportional to $\eta / b$ (step size / batch
size). Our results are not only another proof that implicit regularization has
a significant impact on training optimization dynamics but they also shed light
on the structure of the features that are learned by the network. Additionally,
they suggest that smaller batches enhance feature interpretability and reduce
dependency on initialization.
