  Multiview self-supervised representation learning roots in exploring semantic
consistency across data of complex intra-class variation. Such variation is not
directly accessible and therefore simulated by data augmentations. However,
commonly adopted augmentations are handcrafted and limited to simple
geometrical and color changes, which are unable to cover the abundant
intra-class variation. In this paper, we propose to extract the underlying data
variation from datasets and construct a novel augmentation operator, named
local manifold augmentation (LMA). LMA is achieved by training an
instance-conditioned generator to fit the distribution on the local manifold of
data and sampling multiview data using it. LMA shows the ability to create an
infinite number of data views, preserve semantics, and simulate complicated
variations in object pose, viewpoint, lighting condition, background etc.
Experiments show that with LMA integrated, self-supervised learning methods
such as MoCov2 and SimSiam gain consistent improvement on prevalent benchmarks
including CIFAR10, CIFAR100, STL10, ImageNet100, and ImageNet. Furthermore, LMA
leads to representations that obtain more significant invariance to the
viewpoint, object pose, and illumination changes and stronger robustness to
various real distribution shifts reflected by ImageNet-V2, ImageNet-R, ImageNet
Sketch etc.
