  Multi-speaker automatic speech recognition (ASR) is crucial for many
real-world applications, but it requires dedicated modeling techniques.
Existing approaches can be divided into modular and end-to-end methods. Modular
approaches separate speakers and recognize each of them with a single-speaker
ASR system. End-to-end models process overlapped speech directly in a single,
powerful neural network. This work proposes a middle-ground approach that
leverages explicit speech separation similarly to the modular approach but also
incorporates mixture speech information directly into the ASR module in order
to mitigate the propagation of errors made by the speech separator. We also
explore a way to exchange cross-speaker context information through a layer
that combines information of the individual speakers. Our system is optimized
through separate and joint training stages and achieves a relative improvement
of 7% in word error rate over a purely modular setup on the SMS-WSJ task.
