  End-to-end models are favored in automatic speech recognition (ASR) because
of its simplified system structure and superior performance. Among these
models, recurrent neural network transducer (RNN-T) has achieved significant
progress in streaming on-device speech recognition because of its high-accuracy
and low-latency. RNN-T adopts a prediction network to enhance language
information, but its language modeling ability is limited because it still
needs paired speech-text data to train. Further strengthening the language
modeling ability through extra text data, such as shallow fusion with an
external language model, only brings a small performance gain. In view of the
fact that Mandarin Chinese is a character-based language and each character is
pronounced as a tonal syllable, this paper proposes a novel cascade RNN-T
approach to improve the language modeling ability of RNN-T. Our approach
firstly uses an RNN-T to transform acoustic feature into syllable sequence, and
then converts the syllable sequence into character sequence through an
RNN-T-based syllable-to-character converter. Thus a rich text repository can be
easily used to strengthen the language model ability. By introducing several
important tricks, the cascade RNN-T approach surpasses the character-based
RNN-T by a large margin on several Mandarin test sets, with much higher
recognition quality and similar latency.
