  Recent advancements in contrastive learning have revolutionized
self-supervised representation learning and achieved state-of-the-art
performance on benchmark tasks. While most existing methods focus on applying
contrastive learning to input data modalities such as images, natural language
sentences, or networks, they overlook the potential of utilizing outputs from
previously trained encoders. In this paper, we introduce SIMSKIP, a novel
contrastive learning framework that specifically refines input embeddings for
downstream tasks. Unlike traditional unsupervised learning approaches, SIMSKIP
takes advantage of the output embeddings of encoder models as its input.
Through theoretical analysis, we provide evidence that applying SIMSKIP does
not result in larger upper bounds on downstream task errors than those of the
original embeddings, which serve as SIMSKIP's input. Experimental results on
various open datasets demonstrate that the embeddings produced by SIMSKIP
improve performance on downstream tasks.
