  As a newly emerging unsupervised learning paradigm, self-supervised learning
(SSL) recently gained widespread attention, which usually introduces a pretext
task without manual annotation of data. With its help, SSL effectively learns
the feature representation beneficial for downstream tasks. Thus the pretext
task plays a key role. However, the study of its design, especially its essence
currently is still open. In this paper, we borrow a multi-view perspective to
decouple a class of popular pretext tasks into a combination of view data
augmentation (VDA) and view label classification (VLC), where we attempt to
explore the essence of such pretext task while providing some insights into its
design. Specifically, a simple multi-view learning framework is specially
designed (SSL-MV), which assists the feature learning of downstream tasks
(original view) through the same tasks on the augmented views. SSL-MV focuses
on VDA while abandons VLC, empirically uncovering that it is VDA rather than
generally considered VLC that dominates the performance of such SSL.
Additionally, thanks to replacing VLC with VDA tasks, SSL-MV also enables an
integrated inference combining the predictions from the augmented views,
further improving the performance. Experiments on several benchmark datasets
demonstrate its advantages.
