  Multi-task learning (MTL) significantly pre-dates the deep learning era, and
it has seen a resurgence in the past few years as researchers have been
applying MTL to deep learning solutions for natural language tasks. While
steady MTL research has always been present, there is a growing interest driven
by the impressive successes published in the related fields of transfer
learning and pre-training, such as BERT, and the release of new challenge
problems, such as GLUE and the NLP Decathlon (decaNLP). These efforts place
more focus on how weights are shared across networks, evaluate the re-usability
of network components and identify use cases where MTL can significantly
outperform single-task solutions. This paper strives to provide a comprehensive
survey of the numerous recent MTL contributions to the field of natural
language processing and provide a forum to focus efforts on the hardest
unsolved problems in the next decade. While novel models that improve
performance on NLP benchmarks are continually produced, lasting MTL challenges
remain unsolved which could hold the key to better language understanding,
knowledge discovery and natural language interfaces.
