  This paper introduces a practical approach for leveraging a real-time deep
learning model to alternate between speech enhancement and joint speech
enhancement and separation depending on whether the input mixture contains one
or two active speakers. Scale-invariant signal-to-distortion ratio (SI-SDR) has
shown to be a highly effective training measure in time-domain speech
separation. However, the SI-SDR metric is ill-defined for zero-energy target
signals, which is a problem when training a speech separation model using
utterances with varying numbers of talkers. Unlike existing solutions that
focus on modifying the loss function to accommodate zero-energy target signals,
the proposed approach circumvents this problem by training the model to extract
speech on both its output channels regardless if the input is a single or
dual-talker mixture. A lightweight speaker overlap detection (SOD) module is
also introduced to differentiate between single and dual-talker segments in
real-time. The proposed module takes advantage of the new formulation by
operating directly on the separated masks, given by the separation model,
instead of the original mixture, thus effectively simplifying the detection
task. Experimental results show that the proposed training approach outperforms
existing solutions, and the SOD module exhibits high accuracy.
