  Endowing visual agents with predictive capability is a key step towards video
intelligence at scale. The predominant modeling paradigm for this is sequence
learning, mostly implemented through LSTMs. Feed-forward Transformer
architectures have replaced recurrent model designs in ML applications of
language processing and also partly in computer vision. In this paper we
investigate on the competitiveness of Transformer-style architectures for video
predictive tasks. To do so we propose HORST, a novel higher order recurrent
layer design whose core element is a spatial-temporal decomposition of
self-attention for video. HORST achieves state of the art competitive
performance on Something-Something early action recognition and EPIC-Kitchens
action anticipation, showing evidence of predictive capability that we
attribute to our recurrent higher order design of self-attention.
