  Large language models (LLMs) have shown impressive performance in following
natural language instructions to solve unseen tasks. However, it remains
unclear whether models truly understand task definitions and whether the
human-written definitions are optimal. In this paper, we systematically study
the role of task definitions in instruction learning. We first conduct an
ablation analysis informed by human annotations to understand which parts of a
task definition are most important, and find that model performance only drops
substantially when removing contents describing the task output, in particular
label information. Next, we propose an automatic algorithm to compress task
definitions to a minimal supporting set of tokens, and find that 60\% of tokens
can be removed while maintaining or even improving model performance. Based on
these results, we propose two strategies to help models better leverage task
instructions: (1) providing only key information for tasks in a common
structured format, and (2) adding a meta-tuning stage to help the model better
understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L
improvement over 119 unseen test tasks.
