  It is important to understand how dropout, a popular regularization method,
aids in achieving a good generalization solution during neural network
training. In this work, we present a theoretical derivation of an implicit
regularization of dropout, which is validated by a series of experiments.
Additionally, we numerically study two implications of the implicit
regularization, which intuitively rationalizes why dropout helps
generalization. Firstly, we find that input weights of hidden neurons tend to
condense on isolated orientations trained with dropout. Condensation is a
feature in the non-linear learning process, which makes the network less
complex. Secondly, we experimentally find that the training with dropout leads
to the neural network with a flatter minimum compared with standard gradient
descent training, and the implicit regularization is the key to finding flat
solutions. Although our theory mainly focuses on dropout used in the last
hidden layer, our experiments apply to general dropout in training neural
networks. This work points out a distinct characteristic of dropout compared
with stochastic gradient descent and serves as an important basis for fully
understanding dropout.
