  End-to-End (E2E) automatic speech recognition (ASR) systems used in voice
assistants often have difficulties recognizing infrequent words personalized to
the user, such as names and places. Rare words often have non-trivial
pronunciations, and in such cases, human knowledge in the form of a
pronunciation lexicon can be useful. We propose a PROnunCiation-aware
conTextual adaptER (PROCTER) that dynamically injects lexicon knowledge into an
RNN-T model by adding a phonemic embedding along with a textual embedding. The
experimental results show that the proposed PROCTER architecture outperforms
the baseline RNN-T model by improving the word error rate (WER) by 44% and 57%
when measured on personalized entities and personalized rare entities,
respectively, while increasing the model size (number of trainable parameters)
by only 1%. Furthermore, when evaluated in a zero-shot setting to recognize
personalized device names, we observe 7% WER improvement with PROCTER, as
compared to only 1% WER improvement with text-only contextual attention
