  We investigate input-conditioned hypernetworks for multi-tasking in NLP,
generating parameter-efficient adaptations for a decoder using a hypernetwork
conditioned on the output of an encoder. This approach produces a unique
decoder adaptation for every input instance, allowing the network a larger
degree of flexibility than prior work that only produces one decoder adaptation
per task. We apply our method to sequence classification tasks, extractive QA,
and summarisation and find that it surpasses previous parameter efficient
fine-tuning methods and often outperforms fully finetuning the underlying
model. An analysis of the embeddings used by our hypernetwork shows that they
are sensitive to output label and type, suggesting that our approach better
maps from encoder representations to output labels. Our code is publicly
available at https://github.com/allenai/hyperdecoders.
