  One solution to automatic speech recognition (ASR) of overlapping speakers is
to separate speech and then perform ASR on the separated signals. Commonly, the
separator produces artefacts which often degrade ASR performance. Addressing
this issue typically requires reference transcriptions to jointly train the
separation and ASR networks. This is often not viable for training on
real-world in-domain audio where reference transcript information is not always
available. This paper proposes a transcription-free method for joint training
using only audio signals. The proposed method uses embedding differences of
pre-trained ASR encoders as a loss with a proposed modification to permutation
invariant training (PIT) called guided PIT (GPIT). The method achieves a 6.4%
improvement in word error rate (WER) measures over a signal-level loss and also
shows enhancement improvements in perceptual measures such as short-time
objective intelligibility (STOI).
