  This study reveals the inherent tolerance of contrastive learning (CL)
towards sampling bias, wherein negative samples may encompass similar semantics
(\eg labels). However, existing theories fall short in providing explanations
for this phenomenon. We bridge this research gap by analyzing CL through the
lens of distributionally robust optimization (DRO), yielding several key
insights: (1) CL essentially conducts DRO over the negative sampling
distribution, thus enabling robust performance across a variety of potential
distributions and demonstrating robustness to sampling bias; (2) The design of
the temperature $\tau$ is not merely heuristic but acts as a Lagrange
Coefficient, regulating the size of the potential distribution set; (3) A
theoretical connection is established between DRO and mutual information, thus
presenting fresh evidence for ``InfoNCE as an estimate of MI'' and a new
estimation approach for $\phi$-divergence-based generalized mutual information.
We also identify CL's potential shortcomings, including over-conservatism and
sensitivity to outliers, and introduce a novel Adjusted InfoNCE loss (ADNCE) to
mitigate these issues. It refines potential distribution, improving performance
and accelerating convergence. Extensive experiments on various domains (image,
sentence, and graphs) validate the effectiveness of the proposal. The code is
available at \url{https://github.com/junkangwu/ADNCE}.
