  Contrastive self-supervised learning (CSL) has managed to match or surpass
the performance of supervised learning in image and video classification.
However, it is still largely unknown if the nature of the representations
induced by the two learning paradigms is similar. We investigate this under the
lens of adversarial robustness. Our analysis of the problem reveals that CSL
has intrinsically higher sensitivity to perturbations over supervised learning.
We identify the uniform distribution of data representation over a unit
hypersphere in the CSL representation space as the key contributor to this
phenomenon. We establish that this is a result of the presence of false
negative pairs in the training process, which increases model sensitivity to
input perturbations. Our finding is supported by extensive experiments for
image and video classification using adversarial perturbations and other input
corruptions. We devise a strategy to detect and remove false negative pairs
that is simple, yet effective in improving model robustness with CSL training.
We close up to 68% of the robustness gap between CSL and its supervised
counterpart. Finally, we contribute to adversarial learning by incorporating
our method in CSL. We demonstrate an average gain of about 5% over two
different state-of-the-art methods in this domain.
