  Recent works on instruction tuning (IT) have achieved great performance with
zero-shot generalizability to unseen tasks. With additional context (e.g., task
definition, examples) provided to models for fine-tuning, they achieved much
higher performance than untuned models. Despite impressive performance gains,
what models learn from IT remains understudied. In this work, we analyze how
models utilize instructions during IT by comparing model training with altered
vs. original instructions. Specifically, we create simplified task definitions
by removing all semantic components and only leaving the output space
information, and delusive examples that contain incorrect input-output mapping.
Our experiments show that models trained on simplified task definition or
delusive examples can achieve comparable performance to the ones trained on the
original instructions and examples. Furthermore, we introduce a random baseline
to perform zeroshot classification tasks, and find it achieves similar
performance (42.6% exact-match) as IT does (43% exact-match) in low resource
setting, while both methods outperform naive T5 significantly (30% per
exact-match). Our analysis provides evidence that the impressive performance
gain of current IT models can come from picking up superficial patterns, such
as learning the output format and guessing. Our study highlights the urgent
need for more reliable IT methods and evaluation.
