  Recurrent neural networks (RNNs) are difficult to train on sequence
processing tasks, not only because input noise may be amplified through
feedback, but also because any inaccuracy in the weights has similar
consequences as input noise. We describe a method for denoising the hidden
state during training to achieve more robust representations thereby improving
generalization performance. Attractor dynamics are incorporated into the hidden
state to `clean up' representations at each step of a sequence. The attractor
dynamics are trained through an auxillary denoising loss to recover previously
experienced hidden states from noisy versions of those states. This
state-denoised recurrent neural network {SDRNN} performs multiple steps of
internal processing for each external sequence step. On a range of tasks, we
show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN
with attractor dynamics on the hidden state but without the auxillary loss. We
argue that attractor dynamics---and corresponding connectivity
constraints---are an essential component of the deep learning arsenal and
should be invoked not only for recurrent networks but also for improving deep
feedforward nets and intertask transfer.
