  We present a method for introducing a text encoder into pre-trained
end-to-end speech translation systems. It enhances the ability of adapting one
modality (i.e., source-language speech) to another (i.e., source-language
text). Thus, the speech translation model can learn from both unlabeled and
labeled data, especially when the source-language text data is abundant. Beyond
this, we present a denoising method to build a robust text encoder that can
deal with both normal and noisy text data. Our system sets new
state-of-the-arts on the MuST-C En-De, En-Fr, and LibriSpeech En-Fr tasks.
