  Recurrent neural networks (RNNs) with Long Short-Term memory cells currently
hold the best known results in unconstrained handwriting recognition. We show
that their performance can be greatly improved using dropout - a recently
proposed regularization method for deep architectures. While previous works
showed that dropout gave superior performance in the context of convolutional
networks, it had never been applied to RNNs. In our approach, dropout is
carefully used in the network so that it does not affect the recurrent
connections, hence the power of RNNs in modeling sequence is preserved.
Extensive experiments on a broad range of handwritten databases confirm the
effectiveness of dropout on deep architectures even when the network mainly
consists of recurrent and shared connections.
