  Neural networks typically generalize well when fitting the data perfectly,
even though they are heavily overparameterized. Many factors have been pointed
out as the reason for this phenomenon, including an implicit bias of stochastic
gradient descent (SGD) and a possible simplicity bias arising from the neural
network architecture. The goal of this paper is to disentangle the factors that
influence generalization stemming from optimization and architectural choices
by studying random and SGD-optimized networks that achieve zero training error.
We experimentally show, in the low sample regime, that overparameterization in
terms of increasing width is beneficial for generalization, and this benefit is
due to the bias of SGD and not due to an architectural bias. In contrast, for
increasing depth, overparameterization is detrimental for generalization, but
random and SGD-optimized networks behave similarly, so this can be attributed
to an architectural bias. For more information, see
https://bias-sgd-or-architecture.github.io .
