  Previous RNN architectures have largely been superseded by LSTM, or "Long
Short-Term Memory". Since its introduction, there have been many variations on
this simple design. However, it is still widely used and we are not aware of a
gated-RNN architecture that outperforms LSTM in a broad sense while still being
as simple and efficient. In this paper we propose a modified LSTM-like
architecture. Our architecture is still simple and achieves better performance
on the tasks that we tested on. We also introduce a new RNN performance
benchmark that uses the handwritten digits and stresses several important
network capabilities.
