  In this paper, we propose a novel approach for the transcription of speech
conversations with natural speaker overlap, from single channel speech
recordings. The proposed model is a combination of a speaker diarization system
and a hybrid automatic speech recognition (ASR) system. The speaker conditioned
acoustic model (SCAM) in the ASR system consists of a series of embedding
layers which use the speaker activity inputs from the diarization system to
derive speaker specific embeddings. The output of the SCAM are speaker specific
senones that are used for decoding the transcripts for each speaker in the
conversation. In this work, we experiment with the automatic speaker activity
decisions generated using an end-to-end speaker diarization system. A joint
learning approach is also proposed where the diarization model and the ASR
acoustic model are jointly optimized. The experiments are performed on the
mixed-channel two speaker recordings from the Switchboard corpus of telephone
conversations. In these experiments, we show that the proposed acoustic model,
incorporating speaker activity decisions and joint optimization, improves
significantly over the ASR system with explicit source filtering (relative
improvements of 12% in word error rate (WER) over the baseline system).
