  Voice-controlled house-hold devices, like Amazon Echo or Google Home, face
the problem of performing speech recognition of device-directed speech in the
presence of interfering background speech, i.e., background noise and
interfering speech from another person or media device in proximity need to be
ignored. We propose two end-to-end models to tackle this problem with
information extracted from the "anchored segment". The anchored segment refers
to the wake-up word part of an audio stream, which contains valuable speaker
information that can be used to suppress interfering speech and background
noise. The first method is called "Multi-source Attention" where the attention
mechanism takes both the speaker information and decoder state into
consideration. The second method directly learns a frame-level mask on top of
the encoder output. We also explore a multi-task learning setup where we use
the ground truth of the mask to guide the learner. Given that audio data with
interfering speech is rare in our training data set, we also propose a way to
synthesize "noisy" speech from "clean" speech to mitigate the mismatch between
training and test data. Our proposed methods show up to 15% relative reduction
in WER for Amazon Alexa live data with interfering background speech without
significantly degrading on clean speech.
