  We present a new method to learn video representations from unlabeled data.
Given large-scale unlabeled video data, the objective is to benefit from such
data by learning a generic and transferable representation space that can be
directly used for a new task such as zero/few-shot learning. We formulate our
unsupervised representation learning as a multi-modal, multi-task learning
problem, where the representations are also shared across different modalities
via distillation. Further, we also introduce the concept of finding a better
loss function to train such multi-task multi-modal representation space using
an evolutionary algorithm; our method automatically searches over different
combinations of loss functions capturing multiple (self-supervised) tasks and
modalities. Our formulation allows for the distillation of audio, optical flow
and temporal information into a single, RGB-based convolutional neural network.
We also compare the effects of using additional unlabeled video data and
evaluate our representation learning on standard public video datasets.
