  End-to-end (E2E) speech recognition architectures assemble all components of
traditional speech recognition system into a single model. Although it
simplifies ASR system, it introduces contextual ASR drawback: the E2E model has
worse performance on utterances containing infrequent proper nouns. In this
work, we propose to add a contextual bias attention (CBA) module to attention
based encoder decoder (AED) model to improve its ability of recognizing the
contextual phrases. Specifically, CBA utilizes the context vector of source
attention in decoder to attend to a specific bias embedding. Jointly learned
with the basic AED parameters, CBA can tell the model when and where to bias
its output probability distribution. At inference stage, a list of bias phrases
is preloaded and we adapt the posterior distributions of both CTC and attention
decoder according to the attended bias phrase of CBA. We evaluate the proposed
method on GigaSpeech and achieve a consistent relative improvement on recall
rate of bias phrases ranging from 15% to 28% compared to the baseline model.
Meanwhile, our method shows a strong anti-bias ability as the performance on
general tests only degrades 1.7% even 2,000 bias phrases are present.
