  Large pre-trained language models (LMs) such as GPT-3 have acquired a
surprising ability to perform zero-shot learning. For example, to classify
sentiment without any training examples, we can "prompt" the LM with the review
and the label description "Does the user like this movie?", and ask whether the
next word is "yes" or "no". However, the next word prediction training
objective is still misaligned with the target zero-shot learning objective. To
address this weakness, we propose meta-tuning, which directly optimizes the
zero-shot learning objective by fine-tuning pre-trained language models on a
collection of datasets. We focus on classification tasks, and construct the
meta-dataset by aggregating 43 existing datasets and annotating 441 label
descriptions in a question-answering (QA) format. When evaluated on unseen
tasks, meta-tuned models outperform a same-sized QA model and the previous SOTA
zero-shot learning system based on natural language inference. Additionally,
increasing parameter count from 220M to 770M improves AUC-ROC scores by 6.3%,
and we forecast that even larger models would perform better. Therefore,
measuring zero-shot learning performance on language models out-of-the-box
might underestimate their true potential, and community-wide efforts on
aggregating datasets and unifying their formats can help build models that
answer prompts better.
