  Text-to-text transformers have shown remarkable success in the task of
multi-task transfer learning, especially in natural language processing (NLP).
However, while there have been several attempts to train transformers on
different domains, there is usually a clear relationship between these domains,
e.g.,, code summarization, where the natural language summary describes the
code. There have been very few attempts to study how multi-task transfer
learning works on tasks in significantly different domains. In this project, we
investigated the behavior of multi-domain, multi-task learning using
multi-domain text-to-text transfer transformers (MD-T5) on four tasks across
two domains - Python Code and Chess. We carried out extensive experiments using
three popular training strategies: Bert-style joint pretraining + successive
finetuning, GPT-style joint pretraining + successive finetuning, and GPT-style
joint pretraining + joint finetuning. Also, we evaluate the model on four
metrics - Play Score, Eval Score, BLEU Score, and Multi-Domain Learning Score
(MDLS). These metrics measure performance across the various tasks and
multi-domain learning. We show that while negative knowledge transfer and
catastrophic forgetting are still considerable challenges for all the models,
the GPT-style joint pretraining + joint finetuning strategy showed the most
promise in multi-domain, multi-task learning as it performs well across all
four tasks while still keeping its multi-domain knowledge.
