  Recurrent Neural Networks (RNNs) have long been recognized for their
potential to model complex time series. However, it remains to be determined
what optimization techniques and recurrent architectures can be used to best
realize this potential. The experiments presented take a deep look into Hessian
free optimization, a powerful second order optimization method that has shown
promising results, but still does not enjoy widespread use. This algorithm was
used to train to a number of RNN architectures including standard RNNs, long
short-term memory, multiplicative RNNs, and stacked RNNs on the task of
character prediction. The insights from these experiments led to the creation
of a new multiplicative LSTM hybrid architecture that outperformed both LSTM
and multiplicative RNNs. When tested on a larger scale, multiplicative LSTM
achieved character level modelling results competitive with the state of the
art for RNNs using very different methodology.
