  In this paper, we propose a novel multi-modal multi-task encoder-decoder
pre-training framework (MMSpeech) for Mandarin automatic speech recognition
(ASR), which employs both unlabeled speech and text data. The main difficulty
in speech-text joint pre-training comes from the significant difference between
speech and text modalities, especially for Mandarin speech and text. Unlike
English and other languages with an alphabetic writing system, Mandarin uses an
ideographic writing system where character and sound are not tightly mapped to
one another. Therefore, we propose to introduce the phoneme modality into
pre-training, which can help capture modality-invariant information between
Mandarin speech and text. Specifically, we employ a multi-task learning
framework including five self-supervised and supervised tasks with speech and
text data. For end-to-end pre-training, we introduce self-supervised
speech-to-pseudo-codes (S2C) and phoneme-to-text (P2T) tasks utilizing
unlabeled speech and text data, where speech-pseudo-codes pairs and
phoneme-text pairs are a supplement to the supervised speech-text pairs. To
train the encoder to learn better speech representation, we introduce
self-supervised masked speech prediction (MSP) and supervised phoneme
prediction (PP) tasks to learn to map speech into phonemes. Besides, we
directly add the downstream supervised speech-to-text (S2T) task into the
pre-training process, which can further improve the pre-training performance
and achieve better recognition results even without fine-tuning. Experiments on
AISHELL-1 show that our proposed method achieves state-of-the-art performance,
with a more than 40% relative improvement compared with other pre-training
methods.
