  Recent Self-Supervised Learning (SSL) methods are able to learn feature
representations that are invariant to different data augmentations, which can
then be transferred to downstream tasks of interest. However, different
downstream tasks require different invariances for their best performance, so
the optimal choice of augmentations for SSL depends on the target task. In this
paper, we aim to learn self-supervised features that generalize well across a
variety of downstream tasks (e.g., object classification, detection and
instance segmentation) without knowing any task information beforehand. We do
so by Masked Augmentation Subspace Training (or MAST) to encode in the single
feature space the priors from different data augmentations in a factorized way.
Specifically, we disentangle the feature space into separate subspaces, each
induced by a learnable mask that selects relevant feature dimensions to model
invariance to a specific augmentation. We show the success of MAST in jointly
capturing generalizable priors from different augmentations, using both unique
and shared features across the subspaces. We further show that MAST benefits
from uncertainty modeling to reweight ambiguous samples from strong
augmentations that may cause similarity mismatch in each subspace. Experiments
demonstrate that MAST consistently improves generalization on various
downstream tasks, while being task-agnostic and efficient during SSL. We also
provide interesting insights about how different augmentations are related and
how uncertainty reflects learning difficulty.
