  The idea of combining multiple languages' recordings to train a single
automatic speech recognition (ASR) model brings the promise of the emergence of
universal speech representation. Recently, a Transformer encoder-decoder model
has been shown to leverage multilingual data well in IPA transcriptions of
languages presented during training. However, the representations it learned
were not successful in zero-shot transfer to unseen languages. Because that
model lacks an explicit factorization of the acoustic model (AM) and language
model (LM), it is unclear to what degree the performance suffered from
differences in pronunciation or the mismatch in phonotactics. To gain more
insight into the factors limiting zero-shot ASR transfer, we replace the
encoder-decoder with a hybrid ASR system consisting of a separate AM and LM.
Then, we perform an extensive evaluation of monolingual, multilingual, and
crosslingual (zero-shot) acoustic and language models on a set of 13
phonetically diverse languages. We show that the gain from modeling
crosslingual phonotactics is limited, and imposing a too strong model can hurt
the zero-shot transfer. Furthermore, we find that a multilingual LM hurts a
multilingual ASR system's performance, and retaining only the target language's
phonotactic data in LM training is preferable.
