  A number of competing hypotheses have been proposed to explain why
small-batch Stochastic Gradient Descent (SGD)leads to improved generalization
over the full-batch regime, with recent work crediting the implicit
regularization of various quantities throughout training. However, to date,
empirical evidence assessing the explanatory power of these hypotheses is
lacking. In this paper, we conduct an extensive empirical evaluation, focusing
on the ability of various theorized mechanisms to close the small-to-large
batch generalization gap. Additionally, we characterize how the quantities that
SGD has been claimed to (implicitly) regularize change over the course of
training. By using micro-batches, i.e. disjoint smaller subsets of each
mini-batch, we empirically show that explicitly penalizing the gradient norm or
the Fisher Information Matrix trace, averaged over micro-batches, in the
large-batch regime recovers small-batch SGD generalization, whereas
Jacobian-based regularizations fail to do so. This generalization performance
is shown to often be correlated with how well the regularized model's gradient
norms resemble those of small-batch SGD. We additionally show that this
behavior breaks down as the micro-batch size approaches the batch size.
Finally, we note that in this line of inquiry, positive experimental findings
on CIFAR10 are often reversed on other datasets like CIFAR100, highlighting the
need to test hypotheses on a wider collection of datasets.
