  How can you sample good negative examples for contrastive learning? We argue
that, as with metric learning, contrastive learning of representations benefits
from hard negative samples (i.e., points that are difficult to distinguish from
an anchor point). The key challenge toward using hard negatives is that
contrastive methods must remain unsupervised, making it infeasible to adopt
existing negative sampling strategies that use true similarity information. In
response, we develop a new family of unsupervised sampling methods for
selecting hard negative samples where the user can control the hardness. A
limiting case of this sampling results in a representation that tightly
clusters each class, and pushes different classes as far apart as possible. The
proposed method improves downstream performance across multiple modalities,
requires only few additional lines of code to implement, and introduces no
computational overhead.
