  This paper presents miCSE, a mutual information-based contrastive learning
framework that significantly advances the state-of-the-art in few-shot sentence
embedding. The proposed approach imposes alignment between the attention
pattern of different views during contrastive learning. Learning sentence
embeddings with miCSE entails enforcing the structural consistency across
augmented views for every sentence, making contrastive self-supervised learning
more sample efficient. As a result, the proposed approach shows strong
performance in the few-shot learning domain. While it achieves superior results
compared to state-of-the-art methods on multiple benchmarks in few-shot
learning, it is comparable in the full-shot scenario. This study opens up
avenues for efficient self-supervised learning methods that are more robust
than current contrastive methods for sentence embedding.
