  Despite a lot of research efforts devoted in recent years, how to efficiently
learn long-term dependencies from sequences still remains a pretty challenging
task. As one of the key models for sequence learning, recurrent neural network
(RNN) and its variants such as long short term memory (LSTM) and gated
recurrent unit (GRU) are still not powerful enough in practice. One possible
reason is that they have only feedforward connections, which is different from
the biological neural system that is typically composed of both feedforward and
feedback connections. To address this problem, this paper proposes a
biologically-inspired deep network, called shuttleNet\footnote{Our code is
available at \url{https://github.com/shiyemin/shuttlenet}}. Technologically,
the shuttleNet consists of several processors, each of which is a GRU while
associated with multiple groups of cells and states. Unlike traditional RNNs,
all processors inside shuttleNet are loop connected to mimic the brain's
feedforward and feedback connections, in which they are shared across multiple
pathways in the loop connection. Attention mechanism is then employed to select
the best information flow pathway. Extensive experiments conducted on two
benchmark datasets (i.e UCF101 and HMDB51) show that we can beat
state-of-the-art methods by simply embedding shuttleNet into a CNN-RNN
framework.
