  Many of the recent advances in speech separation are primarily aimed at
synthetic mixtures of short audio utterances with high degrees of overlap. Most
of these approaches need an additional stitching step to stitch the separated
speech chunks for long form audio. Since most of the approaches involve
Permutation Invariant training (PIT), the order of separated speech chunks is
nondeterministic and leads to difficulty in accurately stitching homogenous
speaker chunks for downstream tasks like Automatic Speech Recognition (ASR).
Also, most of these models are trained with synthetic mixtures and do not
generalize to real conversational data. In this paper, we propose a speaker
conditioned separator trained on speaker embeddings extracted directly from the
mixed signal using an over-clustering based approach. This model naturally
regulates the order of the separated chunks without the need for an additional
stitching step. We also introduce a data sampling strategy with real and
synthetic mixtures which generalizes well to real conversation speech. With
this model and data sampling technique, we show significant improvements in
speaker-attributed word error rate (SA-WER) on Hub5 data.
