  Automatic Speech Recognition (ASR) still face challenges when recognizing
time-variant rare-phrases. Contextual biasing (CB) modules bias ASR model
towards such contextually-relevant phrases. During training, a list of biasing
phrases are selected from a large pool of phrases following a sampling
strategy. In this work we firstly analyse different sampling strategies to
provide insights into the training of CB for ASR with correlation plots between
the bias embeddings among various training stages. Secondly, we introduce a
neighbourhood attention (NA) that localizes self attention (SA) to the nearest
neighbouring frames to further refine the CB output. The results show that this
proposed approach provides on average a 25.84% relative WER improvement on
LibriSpeech sets and rare-word evaluation compared to the baseline.
