  We present a new architecture for human action forecasting from videos. A
temporal recurrent encoder captures temporal information of input videos while
a self-attention model is used to attend on relevant feature dimensions of the
input space. To handle temporal variations in observed video data, a feature
masking techniques is employed. We classify observed actions accurately using
an auxiliary classifier which helps to understand what has happened so far.
Then the decoder generates actions for the future based on the output of the
recurrent encoder and the self-attention model. Experimentally, we validate
each component of our architecture where we see that the impact of
self-attention to identify relevant feature dimensions, temporal masking, and
observed auxiliary classifier. We evaluate our method on two standard action
forecasting benchmarks and obtain state-of-the-art results.
