  Self-supervised learning aims to learn a embedding space where semantically
similar samples are close. Contrastive learning methods pull views of samples
together and push different samples away, which utilizes semantic invariance of
augmentation but ignores the relationship between samples. To better exploit
the power of augmentation, we observe that semantically similar samples are
more likely to have similar augmented views. Therefore, we can take the
augmented views as a special description of a sample. In this paper, we model
such a description as the augmentation distribution and we call it augmentation
feature. The similarity in augmentation feature reflects how much the views of
two samples overlap and is related to their semantical similarity. Without
computational burdens to explicitly estimate values of the augmentation
feature, we propose Augmentation Component Analysis (ACA) with a
contrastive-like loss to learn principal components and an on-the-fly
projection loss to embed data. ACA equals an efficient dimension reduction by
PCA and extracts low-dimensional embeddings, theoretically preserving the
similarity of augmentation distribution between samples. Empirical results show
our method can achieve competitive results against various traditional
contrastive learning methods on different benchmarks.
