  Low-resource accented speech recognition is one of the important challenges
faced by current ASR technology in practical applications. In this study, we
propose a Conformer-based architecture, called Aformer, to leverage both the
acoustic information from large non-accented and limited accented training
data. Specifically, a general encoder and an accent encoder are designed in the
Aformer to extract complementary acoustic information. Moreover, we propose to
train the Aformer in a multi-pass manner, and investigate three
cross-information fusion methods to effectively combine the information from
both general and accent encoders. All experiments are conducted on both the
accented English and Mandarin ASR tasks. Results show that our proposed methods
outperform the strong Conformer baseline by relative 10.2% to 24.5%
word/character error rate reduction on six in-domain and out-of-domain accented
test sets.
