  Contrastive learning (CL) has recently emerged as an effective approach to
learning representation in a range of downstream tasks. Central to this
approach is the selection of positive (similar) and negative (dissimilar) sets
to provide the model the opportunity to `contrast' between data and class
representation in the latent space. In this paper, we investigate CL for
improving model robustness using adversarial samples. We first designed and
performed a comprehensive study to understand how adversarial vulnerability
behaves in the latent space. Based on this empirical evidence, we propose an
effective and efficient supervised contrastive learning to achieve model
robustness against adversarial attacks. Moreover, we propose a new sample
selection strategy that optimizes the positive/negative sets by removing
redundancy and improving correlation with the anchor. Extensive experiments
show that our Adversarial Supervised Contrastive Learning (ASCL) approach
achieves comparable performance with the state-of-the-art defenses while
significantly outperforms other CL-based defense methods by using only $42.8\%$
positives and $6.3\%$ negatives.
