  Action recognition is a fundamental problem in computer vision with a lot of
potential applications such as video surveillance, human computer interaction,
and robot learning. Given pre-segmented videos, the task is to recognize
actions happening within videos. Historically, hand crafted video features were
used to address the task of action recognition. With the success of Deep
ConvNets as an image analysis method, a lot of extensions of standard ConvNets
were purposed to process variable length video data. In this work, we propose a
novel recurrent ConvNet architecture called recurrent residual networks to
address the task of action recognition. The approach extends ResNet, a state of
the art model for image classification. While the original formulation of
ResNet aims at learning spatial residuals in its layers, we extend the approach
by introducing recurrent connections that allow to learn a spatio-temporal
residual. In contrast to fully recurrent networks, our temporal connections
only allow a limited range of preceding frames to contribute to the output for
the current frame, enabling efficient training and inference as well as
limiting the temporal context to a reasonable local range around each frame. On
a large-scale action recognition dataset, we show that our model improves over
both, the standard ResNet architecture and a ResNet extended by a fully
recurrent layer.
