  Current ASR systems are mainly trained and evaluated at the utterance level.
Long range cross utterance context can be incorporated. A key task is to derive
a suitable compact representation of the most relevant history contexts. In
contrast to previous researches based on either LSTM-RNN encoded histories that
attenuate the information from longer range contexts, or frame level
concatenation of transformer context embeddings, in this paper compact
low-dimensional cross utterance contextual features are learned in the
Conformer-Transducer Encoder using specially designed attention pooling layers
that are applied over efficiently cached preceding utterances history vectors.
Experiments on the 1000-hr Gigaspeech corpus demonstrate that the proposed
contextualized streaming Conformer-Transducers outperform the baseline using
utterance internal context only with statistically significant WER reductions
of 0.7% to 0.5% absolute (4.3% to 3.1% relative) on the dev and test data.
