  Pre-trained Language Models (PLMs) have achieved remarkable performance for
various language understanding tasks in IR systems, which require the
fine-tuning process based on labeled training data. For low-resource scenarios,
prompt-based learning for PLMs exploits prompts as task guidance and turns
downstream tasks into masked language problems for effective few-shot
fine-tuning. In most existing approaches, the high performance of prompt-based
learning heavily relies on handcrafted prompts and verbalizers, which may limit
the application of such approaches in real-world scenarios. To solve this
issue, we present CP-Tuning, the first end-to-end Contrastive Prompt Tuning
framework for fine-tuning PLMs without any manual engineering of task-specific
prompts and verbalizers. It is integrated with the task-invariant continuous
prompt encoding technique with fully trainable prompt parameters. We further
propose the pair-wise cost-sensitive contrastive learning procedure to optimize
the model in order to achieve verbalizer-free class mapping and enhance the
task-invariance of prompts. It explicitly learns to distinguish different
classes and makes the decision boundary smoother by assigning different costs
to easy and hard cases. Experiments over a variety of language understanding
tasks used in IR systems and different PLMs show that CP-Tuning outperforms
state-of-the-art methods.
