  The key challenge of sequence representation learning is to capture the
long-range temporal dependencies. Typical methods for supervised sequence
representation learning are built upon recurrent neural networks to capture
temporal dependencies. One potential limitation of these methods is that they
only model one-order information interactions explicitly between adjacent time
steps in a sequence, hence the high-order interactions between nonadjacent time
steps are not fully exploited. It greatly limits the capability of modeling the
long-range temporal dependencies since the temporal features learned by
one-order interactions cannot be maintained for a long term due to temporal
information dilution and gradient vanishing. To tackle this limitation, we
propose the Non-local Recurrent Neural Memory (NRNM) for supervised sequence
representation learning, which performs non-local operations \MR{by means of
self-attention mechanism} to learn full-order interactions within a sliding
temporal memory block and models global interactions between memory blocks in a
gated recurrent manner. Consequently, our model is able to capture long-range
dependencies. Besides, the latent high-level features contained in high-order
interactions can be distilled by our model. We validate the effectiveness and
generalization of our NRNM on three types of sequence applications across
different modalities, including sequence classification, step-wise sequential
prediction and sequence similarity learning. Our model compares favorably
against other state-of-the-art methods specifically designed for each of these
sequence applications.
