  We take a geometrical viewpoint and present a unifying view on supervised
deep learning with the Bregman divergence loss function - this entails frequent
classification and prediction tasks. Motivated by simulations we suggest that
there is principally no implicit bias of vanilla stochastic gradient descent
training of deep models towards "simpler" functions. Instead, we show that good
generalization may be instigated by bounded spectral products over layers
leading to a novel geometric regularizer. It is revealed that in deep enough
models such a regularizer enables both, extreme accuracy and generalization, to
be reached. We associate popular regularization techniques like weight decay,
drop out, batch normalization, and early stopping with this perspective. Backed
up by theory we further demonstrate that "generalization by design" is
practically possible and that good generalization may be encoded into the
structure of the network. We design two such easy-to-use structural
regularizers that insert an additional \textit{generalization layer} into a
model architecture, one with a skip connection and another one with drop-out.
We verify our theoretical results in experiments on various feedforward and
convolutional architectures, including ResNets, and datasets (MNIST, CIFAR10,
synthetic data). We believe this work opens up new avenues of research towards
better generalizing architectures.
