  Training a very deep neural network is a challenging task, as the deeper a
neural network is, the more non-linear it is. We compare the performances of
various preconditioned Langevin algorithms with their non-Langevin counterparts
for the training of neural networks of increasing depth. For shallow neural
networks, Langevin algorithms do not lead to any improvement, however the
deeper the network is and the greater are the gains provided by Langevin
algorithms. Adding noise to the gradient descent allows to escape from local
traps, which are more frequent for very deep neural networks. Following this
heuristic we introduce a new Langevin algorithm called Layer Langevin, which
consists in adding Langevin noise only to the weights associated to the deepest
layers. We then prove the benefits of Langevin and Layer Langevin algorithms
for the training of popular deep residual architectures for image
classification.
