  We introduce LAVITI, a novel approach to learning language, video, and
temporal representations in long-form videos via contrastive learning.
Different from pre-training on video-text pairs like EgoVLP, LAVITI aims to
align language, video, and temporal features by extracting meaningful moments
in untrimmed videos. Our model employs a set of learnable moment queries to
decode clip-level visual, language, and temporal features. In addition to
vision and language alignment, we introduce relative temporal embeddings (TE)
to represent timestamps in videos, which enables contrastive learning of time.
Significantly different from traditional approaches, the prediction of a
particular timestamp is transformed by computing the similarity score between
the predicted TE and all TEs. Furthermore, existing approaches for video
understanding are mainly designed for short videos due to high computational
complexity and memory footprint. Our method can be trained on the Ego4D dataset
with only 8 NVIDIA RTX-3090 GPUs in a day. We validated our method on
CharadesEgo action recognition, achieving state-of-the-art results.
