  Standard contrastive learning approaches usually require a large number of
negatives for effective unsupervised learning and often exhibit slow
convergence. We suspect this behavior is due to the suboptimal selection of
negatives used for offering contrast to the positives. We counter this
difficulty by taking inspiration from support vector machines (SVMs) to present
max-margin contrastive learning (MMCL). Our approach selects negatives as the
sparse support vectors obtained via a quadratic optimization problem, and
contrastiveness is enforced by maximizing the decision margin. As SVM
optimization can be computationally demanding, especially in an end-to-end
setting, we present simplifications that alleviate the computational burden. We
validate our approach on standard vision benchmark datasets, demonstrating
better performance in unsupervised representation learning over
state-of-the-art, while having better empirical convergence properties.
