  Multi-talker speech recognition and target-talker speech recognition, both
involve transcription in multi-talker contexts, remain significant challenges.
However, existing methods rarely attempt to simultaneously address both tasks.
In this study, we propose a pioneering approach to empower Whisper, which is a
speech foundation model, to tackle joint multi-talker and target-talker speech
recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar
separator into its encoder to separate mixed embedding for multiple talkers;
(ii) a Target Talker Identifier is introduced to identify the embedding flow of
the target talker on the fly, requiring only three-second enrollment speech as
a cue; (iii) soft prompt tuning for decoder is explored for better task
adaptation. Our method outperforms previous methods on two- and three-talker
LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable
zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset.
