  The state of the art on many NLP tasks is currently achieved by large
pre-trained language models, which require a considerable amount of
computation. We explore a setting where many different predictions are made on
a single piece of text. In that case, some of the computational cost during
inference can be amortized over the different tasks using a shared text
encoder. We compare approaches for training such an encoder and show that
encoders pre-trained over multiple tasks generalize well to unseen tasks. We
also compare ways of extracting fixed- and limited-size representations from
this encoder, including different ways of pooling features extracted from
multiple layers or positions. Our best approach compares favorably to knowledge
distillation, achieving higher accuracy and lower computational cost once the
system is handling around 7 tasks. Further, we show that through binary
quantization, we can reduce the size of the extracted representations by a
factor of 16 making it feasible to store them for later use. The resulting
method offers a compelling solution for using large-scale pre-trained models at
a fraction of the computational cost when multiple tasks are performed on the
same text.
