  In representation learning, regression has traditionally received less
attention than classification. Directly applying representation learning
techniques designed for classification to regression often results in
fragmented representations in the latent space, yielding sub-optimal
performance. In this paper, we argue that the potential of contrastive learning
for regression has been overshadowed due to the neglect of two crucial aspects:
ordinality-awareness and hardness. To address these challenges, we advocate
"mixup your own contrastive pairs for supervised contrastive regression",
instead of relying solely on real/augmented samples. Specifically, we propose
Supervised Contrastive Learning for Regression with Mixup (SupReMix). It takes
anchor-inclusive mixtures (mixup of the anchor and a distinct negative sample)
as hard negative pairs and anchor-exclusive mixtures (mixup of two distinct
negative samples) as hard positive pairs at the embedding level. This strategy
formulates harder contrastive pairs by integrating richer ordinal information.
Through extensive experiments on six regression datasets including 2D images,
volumetric images, text, tabular data, and time-series signals, coupled with
theoretical analysis, we demonstrate that SupReMix pre-training fosters
continuous ordered representations of regression data, resulting in significant
improvement in regression performance. Furthermore, SupReMix is superior to
other approaches in a range of regression challenges including transfer
learning, imbalanced training data, and scenarios with fewer training samples.
