  Self-supervised video representation learning aimed at maximizing similarity
between different temporal segments of one video, in order to enforce feature
persistence over time. This leads to loss of pertinent information related to
temporal relationships, rendering actions such as `enter' and `leave' to be
indistinguishable. To mitigate this limitation, we propose Latent Time
Navigation (LTN), a time-parameterized contrastive learning strategy that is
streamlined to capture fine-grained motions. Specifically, we maximize the
representation similarity between different video segments from one video,
while maintaining their representations time-aware along a subspace of the
latent representation code including an orthogonal basis to represent temporal
changes. Our extensive experimental analysis suggests that learning video
representations by LTN consistently improves performance of action
classification in fine-grained and human-oriented tasks (e.g., on Toyota
Smarthome dataset). In addition, we demonstrate that our proposed model, when
pre-trained on Kinetics-400, generalizes well onto the unseen real world video
benchmark datasets UCF101 and HMDB51, achieving state-of-the-art performance in
action recognition.
