  Improving the representation of contextual information is key to unlocking
the potential of end-to-end (E2E) automatic speech recognition (ASR). In this
work, we present a novel and simple approach for training an ASR context
mechanism with difficult negative examples. The main idea is to focus on proper
nouns (e.g., unique entities such as names of people and places) in the
reference transcript, and use phonetically similar phrases as negative
examples, encouraging the neural model to learn more discriminative
representations. We apply our approach to an end-to-end contextual ASR model
that jointly learns to transcribe and select the correct context items, and
show that our proposed method gives up to $53.1\%$ relative improvement in word
error rate (WER) across several benchmarks.
