  We present a training scheme for streaming automatic speech recognition (ASR)
based on recurrent neural network transducers (RNN-T) which allows the encoder
network to learn to exploit context audio from a stream, using segmented or
partially labeled sequences of the stream during training. We show that the use
of context audio during training and inference can lead to word error rate
reductions of more than 6% in a realistic production setting for a voice
assistant ASR system. We investigate the effect of the proposed training
approach on acoustically challenging data containing background speech and
present data points which indicate that this approach helps the network learn
both speaker and environment adaptation. To gain further insight into the
ability of a long short-term memory (LSTM) based ASR encoder to exploit
long-term context, we also visualize RNN-T loss gradients with respect to the
input.
