  The Long Short-Term Memory (LSTM) recurrent neural network is capable of
processing complex sequential information since it utilizes special gating
schemes for learning representations from long input sequences. It has the
potential to model any sequential time-series data, where the current hidden
state has to be considered in the context of the past hidden states. This
property makes LSTM an ideal choice to learn the complex dynamics present in
long sequences. Unfortunately, the conventional LSTMs do not consider the
impact of spatio-temporal dynamics corresponding to the given salient motion
patterns, when they gate the information that ought to be memorized through
time. To address this problem, we propose a differential gating scheme for the
LSTM neural network, which emphasizes on the change in information gain caused
by the salient motions between the successive video frames. This change in
information gain is quantified by Derivative of States (DoS), and thus the
proposed LSTM model is termed as differential Recurrent Neural Network (dRNN).
In addition, the original work used the hidden state at the last time-step to
model the entire video sequence. Based on the energy profiling of DoS, we
further propose to employ the State Energy Profile (SEP) to search for salient
dRNN states and construct more informative representations. The effectiveness
of the proposed model was demonstrated by automatically recognizing human
actions from the real-world 2D and 3D single-person action datasets. We point
out that LSTM is a special form of dRNN. As a result, we have introduced a new
family of LSTMs. Our study is one of the first works towards demonstrating the
potential of learning complex time-series representations via high-order
derivatives of states.
