  This report describes the difficulties of training neural networks and in
particular deep neural networks. It then provides a literature review of
training methods for deep neural networks, with a focus on pre-training. It
focuses on Deep Belief Networks composed of Restricted Boltzmann Machines and
Stacked Autoencoders and provides an outreach on further and alternative
approaches. It also includes related practical recommendations from the
literature on training them. In the second part, initial experiments using some
of the covered methods are performed on two databases. In particular,
experiments are performed on the MNIST hand-written digit dataset and on facial
emotion data from a Kaggle competition. The results are discussed in the
context of results reported in other research papers. An error rate lower than
the best contribution to the Kaggle competition is achieved using an optimized
Stacked Autoencoder.
