  Recently, representation learning for text and speech has successfully
improved many language related tasks. However, all existing methods suffer from
two limitations: (a) they only learn from one input modality, while a unified
representation for both speech and text is needed by tasks such as end-to-end
speech translation, and as a result,(b) they can not exploit various
large-scale text and speech data and their performance is limited by the
scarcity of parallel speech translation data.To address these problems, we
propose a Fused Acoustic and Text Masked Language Model (FAT-MLM) which jointly
learns a unified representation for both acoustic and text input from various
types of corpora including parallel data for speech recognition and machine
translation, and even pure speech and text data. Within this cross-modal
representation learning framework, we further present an end-to-end model for
Fused Acoustic and Text Speech Translation (FAT-ST). Experiments on three
translation directions show that by fine-tuning from FAT-MLM, our proposed
speech translation models substantially improve translation quality by up to
+5.9 BLEU.
