  Training deep neural networks is a very demanding task, especially
challenging is how to adapt architectures to improve the performance of trained
models. We can find that sometimes, shallow networks generalize better than
deep networks, and the addition of more layers results in higher training and
test errors. The deep residual learning framework addresses this degradation
problem by adding skip connections to several neural network layers. It would
at first seem counter-intuitive that such skip connections are needed to train
deep networks successfully as the expressivity of a network would grow
exponentially with depth. In this paper, we first analyze the flow of
information through neural networks. We introduce and evaluate the
batch-entropy which quantifies the flow of information through each layer of a
neural network. We prove empirically and theoretically that a positive
batch-entropy is required for gradient descent-based training approaches to
optimize a given loss function successfully. Based on those insights, we
introduce batch-entropy regularization to enable gradient descent-based
training algorithms to optimize the flow of information through each hidden
layer individually. With batch-entropy regularization, gradient descent
optimizers can transform untrainable networks into trainable networks. We show
empirically that we can therefore train a "vanilla" fully connected network and
convolutional neural network -- no skip connections, batch normalization,
dropout, or any other architectural tweak -- with 500 layers by simply adding
the batch-entropy regularization term to the loss function. The effect of
batch-entropy regularization is not only evaluated on vanilla neural networks,
but also on residual networks, autoencoders, and also transformer models over a
wide range of computer vision as well as natural language processing tasks.
