  Inspired by the fact that human eyes continue to develop tracking ability in
early and middle childhood, we propose to use tracking as a proxy task for a
computer vision system to learn the visual representations. Modelled on the
Catch game played by the children, we design a Catch-the-Patch (CtP) game for a
3D-CNN model to learn visual representations that would help with video-related
tasks. In the proposed pretraining framework, we cut an image patch from a
given video and let it scale and move according to a pre-set trajectory. The
proxy task is to estimate the position and size of the image patch in a
sequence of video frames, given only the target bounding box in the first
frame. We discover that using multiple image patches simultaneously brings
clear benefits. We further increase the difficulty of the game by randomly
making patches invisible. Extensive experiments on mainstream benchmarks
demonstrate the superior performance of CtP against other video pretraining
methods. In addition, CtP-pretrained features are less sensitive to domain gaps
than those trained by a supervised action recognition task. When both trained
on Kinetics-400, we are pleasantly surprised to find that CtP-pretrained
representation achieves much higher action classification accuracy than its
fully supervised counterpart on Something-Something dataset. Code is available
online: github.com/microsoft/CtP.
