  We propose a new self-supervised CNN pre-training technique based on a novel
auxiliary task called "odd-one-out learning". In this task, the machine is
asked to identify the unrelated or odd element from a set of otherwise related
elements. We apply this technique to self-supervised video representation
learning where we sample subsequences from videos and ask the network to learn
to predict the odd video subsequence. The odd video subsequence is sampled such
that it has wrong temporal order of frames while the even ones have the correct
temporal order. Therefore, to generate a odd-one-out question no manual
annotation is required. Our learning machine is implemented as multi-stream
convolutional neural network, which is learned end-to-end. Using odd-one-out
networks, we learn temporal representations for videos that generalizes to
other related tasks such as action recognition.
  On action classification, our method obtains 60.3\% on the UCF101 dataset
using only UCF101 data for training which is approximately 10% better than
current state-of-the-art self-supervised learning methods. Similarly, on HMDB51
dataset we outperform self-supervised state-of-the art methods by 12.7% on
action classification task.
