  Self-supervised learning aims to extract meaningful features from unlabeled
data for further downstream tasks. In this paper, we consider classification as
a downstream task in phase 2 and develop rigorous theories to realize the
factors that implicitly influence the general loss of this classification task.
Our theories signify that sharpness-aware feature extractors benefit the
classification task in phase 2 and the existing data shift between the ideal
(i.e., the ideal one used in theory development) and practical (i.e., the
practical one used in implementation) distributions to generate positive pairs
also remarkably affects this classification task. Further harvesting these
theoretical findings, we propose to minimize the sharpness of the feature
extractor and a new Fourier-based data augmentation technique to relieve the
data shift in the distributions generating positive pairs, reaching Sharpness &
Shift-Aware Contrastive Learning (SSA-CLR). We conduct extensive experiments to
verify our theoretical findings and demonstrate that sharpness & shift-aware
contrastive learning can remarkably boost the performance as well as obtaining
more robust extracted features compared with the baselines.
