  Capturing spatiotemporal dynamics is an essential topic in video recognition.
In this paper, we present learnable higher-order operations as a generic family
of building blocks for capturing spatiotemporal dynamics from RGB input video
space. Similar to higher-order functions, the weights of higher-order
operations are themselves derived from the data with learnable parameters.
Classical architectures such as residual learning and network-in-network are
first-order operations where weights are directly learned from the data.
Higher-order operations make it easier to capture context-sensitive patterns,
such as motion. Self-attention models are also higher-order operations, but the
attention weights are mostly computed from an affine operation or dot product.
Learnable higher-order operations can be more generic and flexible.
Experimentally, we show that on the task of video recognition, our higher-order
models can achieve results on par with or better than the existing
state-of-the-art methods on Something-Something (V1 and V2), Kinetics and
Charades datasets.
