  We propose a novel neural network architecture based on conformer transducer
that adds contextual information flow to the ASR systems. Our method improves
the accuracy of recognizing uncommon words while not harming the word error
rate of regular words. We explore the uncommon words accuracy improvement when
we use the new model and/or shallow fusion with context language model. We
found that combination of both provides cumulative gain in uncommon words
recognition accuracy.
