  In this paper, we propose a convolutional layer inspired by optical flow
algorithms to learn motion representations. Our representation flow layer is a
fully-differentiable layer designed to capture the `flow' of any representation
channel within a convolutional neural network for action recognition. Its
parameters for iterative flow optimization are learned in an end-to-end fashion
together with the other CNN model parameters, maximizing the action recognition
performance. Furthermore, we newly introduce the concept of learning `flow of
flow' representations by stacking multiple representation flow layers. We
conducted extensive experimental evaluations, confirming its advantages over
previous recognition models using traditional optical flows in both
computational speed and performance. Code/models available here:
https://piergiaj.github.io/rep-flow-site/
