  Attention-based sequence-to-sequence modeling provides a powerful and elegant
solution for applications that need to map one sequence to a different
sequence. Its success heavily relies on the availability of large amounts of
training data. This presents a challenge for speech applications where labelled
speech data is very expensive to obtain, such as automatic speech recognition
(ASR) and speech translation (ST). In this study, we propose a general
multi-task learning framework to leverage text data for ASR and ST tasks. Two
auxiliary tasks, a denoising autoencoder task and machine translation task, are
proposed to be co-trained with ASR and ST tasks respectively. We demonstrate
that representing text input as phoneme sequences can reduce the difference
between speech and text inputs, and enhance the knowledge transfer from text
corpora to the speech to text tasks. Our experiments show that the proposed
method achieves a relative 10~15% word error rate reduction on the English
Librispeech task compared with our baseline, and improves the speech
translation quality on the MuST-C tasks by 3.6~9.2 BLEU.
