  This work presents a new algorithm for training recurrent neural networks
(although ideas are applicable to feedforward networks as well). The algorithm
is derived from a theory in nonconvex optimization related to the diffusion
equation. The contributions made in this work are two fold. First, we show how
some seemingly disconnected mechanisms used in deep learning such as smart
initialization, annealed learning rate, layerwise pretraining, and noise
injection (as done in dropout and SGD) arise naturally and automatically from
this framework, without manually crafting them into the algorithms. Second, we
present some preliminary results on comparing the proposed method against SGD.
It turns out that the new algorithm can achieve similar level of generalization
accuracy of SGD in much fewer number of epochs.
