  Mixture-of-experts based models, which use language experts to extract
language-specific representations effectively, have been well applied in
code-switching automatic speech recognition. However, there is still
substantial space to improve as similar pronunciation across languages may
result in ineffective multi-language modeling and inaccurate language boundary
estimation. To eliminate these drawbacks, we propose a cross-layer language
adapter and a boundary-aware training method, namely Boundary-Aware
Mixture-of-Experts (BA-MoE). Specifically, we introduce language-specific
adapters to separate language-specific representations and a unified gating
layer to fuse representations within each encoder layer. Second, we compute
language adaptation loss of the mean output of each language-specific adapter
to improve the adapter module's language-specific representation learning.
Besides, we utilize a boundary-aware predictor to learn boundary
representations for dealing with language boundary confusion. Our approach
achieves significant performance improvement, reducing the mixture error rate
by 16.55\% compared to the baseline on the ASRU 2019 Mandarin-English
code-switching challenge dataset.
