  Recent NLP models have shown the remarkable ability to effectively generalise
`zero-shot' to new tasks using only natural language instructions as guidance.
However, many of these approaches suffer from high computational costs due to
their reliance on concatenating lengthy instructions with every input example,
resulting in costly reprocessing of the instruction. To avoid this, we
introduce Hypernetworks for INstruction Tuning (HINT), which convert task
instructions and examples into parameter-efficient modules inserted into an
underlying model using a pretrained text encoder, eliminating the need to
include instructions in the model input. The hypernetwork in HINT also produces
an encoded instruction, which we concatenate with encoded inputs during
decoding to further improve performance. HINT models outperform strong
state-of-the-art baselines by over 10% when controlling for compute (measured
in FLOPs). By converting instructions into modules, HINT models can effectively
disregard the length of instructions and few-shot example inputs in terms of
compute usage. As a result, HINT can enhance its performance by up to 25% by
incorporating additional few-shot data, while utilizing only up to 5% more
compute. This combines the strengths of parameter-efficient fine-tuning and
in-context learning.
