  Correspondences between frames encode rich information about dynamic content
in videos. However, it is challenging to effectively capture and learn those
due to their irregular structure and complex dynamics. In this paper, we
propose a novel neural network that learns video representations by aggregating
information from potential correspondences. This network, named $CPNet$, can
learn evolving 2D fields with temporal consistency. In particular, it can
effectively learn representations for videos by mixing appearance and
long-range motion with an RGB-only input. We provide extensive ablation
experiments to validate our model. CPNet shows stronger performance than
existing methods on Kinetics and achieves the state-of-the-art performance on
Something-Something and Jester. We provide analysis towards the behavior of our
model and show its robustness to errors in proposals.
