  Traditional supervised learning mostly works on individual tasks and requires
training on a large set of task-specific examples. This paradigm seriously
hinders the development of task generalization since preparing a task-specific
example set is costly. To build a system that can quickly and easily generalize
to new tasks, task instructions have been adopted as an emerging trend of
supervision recently. These instructions give the model the definition of the
task and allow the model to output the appropriate answer based on the
instructions and inputs. However, task instructions are often expressed in
different forms, which can be interpreted from two threads: first, some
instructions are short sentences and are pretrained language model (PLM)
oriented, such as prompts, while other instructions are paragraphs and are
human-oriented, such as those in Amazon MTurk; second, different end-users very
likely explain the same task with instructions of different textual
expressions. A robust system for task generalization should be able to handle
any new tasks regardless of the variability of instructions.
  However, the system robustness in dealing with instruction-driven task
generalization is still unexplored. This work investigates the system
robustness when the instructions of new tasks are (i) manipulated, (ii)
paraphrased, or (iii) from different levels of conciseness. To our knowledge,
this is the first work that systematically studies how robust a PLM is when it
is supervised by instructions with different factors of variability.
