  State-of-the-art large-scale universal speech models (USMs) show a decent
automatic speech recognition (ASR) performance across multiple domains and
languages. However, it remains a challenge for these models to recognize
overlapped speech, which is often seen in meeting conversations. We propose an
approach to adapt USMs for multi-talker ASR. We first develop an enhanced
version of serialized output training to jointly perform multi-talker ASR and
utterance timestamp prediction. That is, we predict the ASR hypotheses for all
speakers, count the speakers, and estimate the utterance timestamps at the same
time. We further introduce a lightweight adapter module to maintain the
multilingual property of the USMs even when we perform the adaptation with only
a single language. Experimental results obtained using the AMI and AliMeeting
corpora show that our proposed approach effectively transfers the USMs to a
strong multilingual multi-talker ASR model with timestamp prediction
capability.
