  In this work we study the benefits of using tracking and 3D poses for action
recognition. To achieve this, we take the Lagrangian view on analysing actions
over a trajectory of human motion rather than at a fixed point in space. Taking
this stand allows us to use the tracklets of people to predict their actions.
In this spirit, first we show the benefits of using 3D pose to infer actions,
and study person-person interactions. Subsequently, we propose a Lagrangian
Action Recognition model by fusing 3D pose and contextualized appearance over
tracklets. To this end, our method achieves state-of-the-art performance on the
AVA v2.2 dataset on both pose only settings and on standard benchmark settings.
When reasoning about the action using only pose cues, our pose model achieves
+10.0 mAP gain over the corresponding state-of-the-art while our fused model
has a gain of +2.8 mAP over the best state-of-the-art model. Code and results
are available at: https://brjathu.github.io/LART
