  Unsupervised video representation learning has made remarkable achievements
in recent years. However, most existing methods are designed and optimized for
video classification. These pre-trained models can be sub-optimal for temporal
localization tasks due to the inherent discrepancy between video-level
classification and clip-level localization. To bridge this gap, we make the
first attempt to propose a self-supervised pretext task, coined as Pseudo
Action Localization (PAL) to Unsupervisedly Pre-train feature encoders for
Temporal Action Localization tasks (UP-TAL). Specifically, we first randomly
select temporal regions, each of which contains multiple clips, from one video
as pseudo actions and then paste them onto different temporal positions of the
other two videos. The pretext task is to align the features of pasted pseudo
action regions from two synthetic videos and maximize the agreement between
them. Compared to the existing unsupervised video representation learning
approaches, our PAL adapts better to downstream TAL tasks by introducing a
temporal equivariant contrastive learning paradigm in a temporally dense and
scale-aware manner. Extensive experiments show that PAL can utilize large-scale
unlabeled video data to significantly boost the performance of existing TAL
methods. Our codes and models will be made publicly available at
https://github.com/zhang-can/UP-TAL.
