  As the performance of single-channel speech separation systems has improved,
there has been a desire to move to more challenging conditions than the clean,
near-field speech that initial systems were developed on. When training deep
learning separation models, a need for ground truth leads to training on
synthetic mixtures. As such, training in noisy conditions requires either using
noise synthetically added to clean speech, preventing the use of in-domain data
for a noisy-condition task, or training using mixtures of noisy speech,
requiring the network to additionally separate the noise. We demonstrate the
relative inseparability of noise and that this noisy speech paradigm leads to
significant degradation of system performance. We also propose an
SI-SDR-inspired training objective that tries to exploit the inseparability of
noise to implicitly partition the signal and discount noise separation errors,
enabling the training of better separation systems with noisy oracle sources.
