  The advances in attention-based encoder-decoder (AED) networks have brought
great progress to end-to-end (E2E) automatic speech recognition (ASR). One way
to further improve the performance of AED-based E2E ASR is to introduce an
extra text encoder for leveraging extensive text data and thus capture more
context-aware linguistic information. However, this approach brings a mismatch
problem between the speech encoder and the text encoder due to the different
units used for modeling. In this paper, we propose an embedding aligner and
modality switch training to better align the speech and text latent spaces. The
embedding aligner is a shared linear projection between text encoder and speech
encoder trained by masked language modeling (MLM) loss and connectionist
temporal classification (CTC), respectively. The modality switch training
randomly swaps speech and text embeddings based on the forced alignment result
to learn a joint representation space. Experimental results show that our
proposed approach achieves a relative 14% to 19% word error rate (WER)
reduction on Librispeech ASR task. We further verify its effectiveness on
spoken language understanding (SLU), i.e., an absolute 2.5% to 2.8% F1 score
improvement on SNIPS slot filling task.
