  Recently, prefix-tuning was proposed to efficiently adapt pre-trained
language models to a broad spectrum of natural language classification tasks.
It leverages soft prefix as task-specific indicators and language verbalizers
as categorical-label mentions to narrow the formulation gap from pre-training
language models. However, when the label space increases considerably (i.e.,
many-class classification), such a tuning technique suffers from a verbalizer
ambiguity problem since the many-class labels are represented by
semantic-similar verbalizers in short language phrases. To overcome this,
inspired by the human-decision process that the most ambiguous classes would be
mulled over for each instance, we propose a brand-new prefix-tuning method,
Counterfactual Contrastive Prefix-tuning (CCPrefix), for many-class
classification. Basically, an instance-dependent soft prefix, derived from
fact-counterfactual pairs in the label space, is leveraged to complement the
language verbalizers in many-class classification. We conduct experiments on
many-class benchmark datasets in both the fully supervised setting and the
few-shot setting, which indicates that our model outperforms former baselines.
