  Contrastive Learning has emerged as a powerful representation learning method
and facilitates various downstream tasks especially when supervised data is
limited. How to construct efficient contrastive samples through data
augmentation is key to its success. Unlike vision tasks, the data augmentation
method for contrastive learning has not been investigated sufficiently in
language tasks. In this paper, we propose a novel approach to construct
contrastive samples for language tasks using text summarization. We use these
samples for supervised contrastive learning to gain better text representations
which greatly benefit text classification tasks with limited annotations. To
further improve the method, we mix up samples from different classes and add an
extra regularization, named Mixsum, in addition to the cross-entropy-loss.
Experiments on real-world text classification datasets (Amazon-5, Yelp-5, AG
News, and IMDb) demonstrate the effectiveness of the proposed contrastive
learning framework with summarization-based data augmentation and Mixsum
regularization.
