  Online action detection (OAD) is a task that receives video segments within a
streaming video as inputs and identifies ongoing actions within them. It is
important to retain past information associated with a current action. However,
long short-term memory (LSTM), a popular recurrent unit for modeling temporal
information from videos, accumulates past information from the previous hidden
and cell states and the extracted visual features at each timestep without
considering the relationships between the past and current information.
Consequently, the forget gate of the original LSTM can lose the accumulated
information relevant to the current action because it determines which
information to forget without considering the current action. We introduce a
novel information elevation unit (IEU) that lifts up and accumulate the past
information relevant to the current action in order to model the past
information that is especially relevant to the current action. To the best of
our knowledge, our IEN is the first attempt that considers the computational
overhead for the practical use of OAD. Through ablation studies, we design an
efficient and effective OAD network using IEUs, called an information elevation
network (IEN). Our IEN uses visual features extracted by a fast action
recognition network taking only RGB frames because extracting optical flows
requires heavy computation overhead. On two OAD benchmark datasets, THUMOS-14
and TVSeries, our IEN outperforms state-of-the-art OAD methods using only RGB
frames. Furthermore, on the THUMOS-14 dataset, our IEN outperforms the
state-of-the-art OAD methods using two-stream features based on RGB frames and
optical flows.
