  Humans view the world through many sensory channels, e.g., the
long-wavelength light channel, viewed by the left eye, or the high-frequency
vibrations channel, heard by the right ear. Each view is noisy and incomplete,
but important factors, such as physics, geometry, and semantics, tend to be
shared between all views (e.g., a "dog" can be seen, heard, and felt). We
investigate the classic hypothesis that a powerful representation is one that
models view-invariant factors. We study this hypothesis under the framework of
multiview contrastive learning, where we learn a representation that aims to
maximize mutual information between different views of the same scene but is
otherwise compact. Our approach scales to any number of views, and is
view-agnostic. We analyze key properties of the approach that make it work,
finding that the contrastive loss outperforms a popular alternative based on
cross-view prediction, and that the more views we learn from, the better the
resulting representation captures underlying scene semantics. Our approach
achieves state-of-the-art results on image and video unsupervised learning
benchmarks. Code is released at: http://github.com/HobbitLong/CMC/.
