  Video predictive understanding encompasses a wide range of efforts that are
concerned with the anticipation of the unobserved future from the current as
well as historical video observations. Action prediction is a major sub-area of
video predictive understanding and is the focus of this review. This sub-area
has two major subdivisions: early action recognition and future action
prediction. Early action recognition is concerned with recognizing an ongoing
action as soon as possible. Future action prediction is concerned with the
anticipation of actions that follow those previously observed. In either case,
the \textbf{\textit{causal}} relationship between the past, current, and
potential future information is the main focus. Various mathematical tools such
as Markov Chains, Gaussian Processes, Auto-Regressive modeling, and Bayesian
recursive filtering are widely adopted jointly with computer vision techniques
for these two tasks. However, these approaches face challenges such as the
curse of dimensionality, poor generalization, and constraints from
domain-specific knowledge. Recently, structures that rely on deep convolutional
neural networks and recurrent neural networks have been extensively proposed
for improving the performance of existing vision tasks, in general, and action
prediction tasks, in particular. However, they have their own shortcomings, \eg
reliance on massive training data and lack of strong theoretical underpinnings.
In this survey, we start by introducing the major sub-areas of the broad area
of video predictive understanding, which recently have received intensive
attention and proven to have practical value. Next, a thorough review of
various early action recognition and future action prediction algorithms are
provided with suitably organized divisions. Finally, we conclude our discussion
with future research directions.
