  In recent years, the evolution of end-to-end (E2E) automatic speech
recognition (ASR) models has been remarkable, largely due to advances in deep
learning architectures like transformer. On top of E2E systems, researchers
have achieved substantial accuracy improvement by rescoring E2E model's N-best
hypotheses with a phoneme-based model. This raises an interesting question
about where the improvements come from other than the system combination
effect. We examine the underlying mechanisms driving these gains and propose an
efficient joint training approach, where E2E models are trained jointly with
diverse modeling units. This methodology does not only align the strengths of
both phoneme and grapheme-based models but also reveals that using these
diverse modeling units in a synergistic way can significantly enhance model
accuracy. Our findings offer new insights into the optimal integration of
heterogeneous modeling units in the development of more robust and accurate ASR
systems.
