  With the massive developments of end-to-end (E2E) neural networks, recent
years have witnessed unprecedented breakthroughs in automatic speech
recognition (ASR). However, the codeswitching phenomenon remains a major
obstacle that hinders ASR from perfection, as the lack of labeled data and the
variations between languages often lead to degradation of ASR performance. In
this paper, we focus exclusively on improving the acoustic encoder of E2E ASR
to tackle the challenge caused by the codeswitching phenomenon. Our main
contributions are threefold: First, we introduce a novel disentanglement loss
to enable the lower-layer of the encoder to capture inter-lingual acoustic
information while mitigating linguistic confusion at the higher-layer of the
encoder. Second, through comprehensive experiments, we verify that our proposed
method outperforms the prior-art methods using pretrained dual-encoders,
meanwhile having access only to the codeswitching corpus and consuming half of
the parameterization. Third, the apparent differentiation of the encoders'
output features also corroborates the complementarity between the
disentanglement loss and the mixture-of-experts (MoE) architecture.
