  Contrastive learning (CL) has recently been applied to adversarial learning
tasks. Such practice considers adversarial samples as additional positive views
of an instance, and by maximizing their agreements with each other, yields
better adversarial robustness. However, this mechanism can be potentially
flawed, since adversarial perturbations may cause instance-level identity
confusion, which can impede CL performance by pulling together different
instances with separate identities. To address this issue, we propose to treat
adversarial samples unequally when contrasted, with an asymmetric InfoNCE
objective ($A-InfoNCE$) that allows discriminating considerations of
adversarial samples. Specifically, adversaries are viewed as inferior positives
that induce weaker learning signals, or as hard negatives exhibiting higher
contrast to other negative samples. In the asymmetric fashion, the adverse
impacts of conflicting objectives between CL and adversarial learning can be
effectively mitigated. Experiments show that our approach consistently
outperforms existing Adversarial CL methods across different finetuning schemes
without additional computational cost. The proposed A-InfoNCE is also a generic
form that can be readily extended to other CL methods. Code is available at
https://github.com/yqy2001/A-InfoNCE.
