  Recently, to mitigate the confusion between different languages in
code-switching (CS) automatic speech recognition (ASR), the conditionally
factorized models, such as the language-aware encoder (LAE), explicitly
disregard the contextual information between different languages. However, this
information may be helpful for ASR modeling. To alleviate this issue, we
propose the LAE-ST-MoE framework. It incorporates speech translation (ST) tasks
into LAE and utilizes ST to learn the contextual information between different
languages. It introduces a task-based mixture of expert modules, employing
separate feed-forward networks for the ASR and ST tasks. Experimental results
on the ASRU 2019 Mandarin-English CS challenge dataset demonstrate that,
compared to the LAE-based CTC, the LAE-ST-MoE model achieves a 9.26% mix error
reduction on the CS test with the same decoding parameter. Moreover, the
well-trained LAE-ST-MoE model can perform ST tasks from CS speech to Mandarin
or English text.
