  State-of-the-art methods for self-supervised sequential action alignment rely
on deep networks that find correspondences across videos in time. They either
learn frame-to-frame mapping across sequences, which does not leverage temporal
information, or assume monotonic alignment between each video pair, which
ignores variations in the order of actions. As such, these methods are not able
to deal with common real-world scenarios that involve background frames or
videos that contain non-monotonic sequence of actions.
  In this paper, we propose an approach to align sequential actions in the wild
that involve diverse temporal variations. To this end, we propose an approach
to enforce temporal priors on the optimal transport matrix, which leverages
temporal consistency, while allowing for variations in the order of actions.
Our model accounts for both monotonic and non-monotonic sequences and handles
background frames that should not be aligned. We demonstrate that our approach
consistently outperforms the state-of-the-art in self-supervised sequential
action representation learning on four different benchmark datasets.
