  Existing video self-supervised learning methods mainly rely on trimmed videos
for model training. However, trimmed datasets are manually annotated from
untrimmed videos. In this sense, these methods are not really self-supervised.
In this paper, we propose a novel self-supervised method, referred to as
Exploring Relations in Untrimmed Videos (ERUV), which can be straightforwardly
applied to untrimmed videos (real unlabeled) to learn spatio-temporal features.
ERUV first generates single-shot videos by shot change detection. Then a
designed sampling strategy is used to model relations for video clips. The
strategy is saved as our self-supervision signals. Finally, the network learns
representations by predicting the category of relations between the video
clips. ERUV is able to compare the differences and similarities of videos,
which is also an essential procedure for action and video related tasks. We
validate our learned models with action recognition and video retrieval tasks
with three kinds of 3D CNNs. Experimental results show that ERUV is able to
learn richer representations and it outperforms state-of-the-art
self-supervised methods with significant margins.
