  Multi-Task Learning (MTL) models have shown their robustness, effectiveness,
and efficiency for transferring learned knowledge across tasks. In real
industrial applications such as web content classification, multiple
classification tasks are predicted from the same input text such as a web
article. However, at the serving time, the existing multitask transformer
models such as prompt or adaptor based approaches need to conduct N forward
passes for N tasks with O(N) computation cost. To tackle this problem, we
propose a scalable method that can achieve stronger performance with close to
O(1) computation cost via only one forward pass. To illustrate real application
usage, we release a multitask dataset on news topic and style classification.
Our experiments show that our proposed method outperforms strong baselines on
both the GLUE benchmark and our news dataset. Our code and dataset are publicly
available at https://bit.ly/mtop-code.
