  Improving end-to-end speech recognition by incorporating external text data
has been a longstanding research topic. There has been a recent focus on
training E2E ASR models that get the performance benefits of external text data
without incurring the extra cost of evaluating an external language model at
inference time. In this work, we propose training ASR model jointly with a set
of text-to-text auxiliary tasks with which it shares a decoder and parts of the
encoder. When we jointly train ASR and masked language model with the 960-hour
Librispeech and Opensubtitles data respectively, we observe WER reductions of
16% and 20% on test-other and test-clean respectively over an ASR-only baseline
without any extra cost at inference time, and reductions of 6% and 8% compared
to a stronger MUTE-L baseline which trains the decoder with the same text data
as our model. We achieve further improvements when we train masked language
model on Librispeech data or when we use machine translation as the auxiliary
task, without significantly sacrificing performance on the task itself.
