  Contextualized end-to-end automatic speech recognition has been an active
research area, with recent efforts focusing on the implicit learning of
contextual phrases based on the final loss objective. However, these approaches
ignore the useful contextual knowledge encoded in the intermediate layers. We
hypothesize that employing explicit biasing loss as an auxiliary task in the
encoder intermediate layers may better align text tokens or audio frames with
the desired objectives. Our proposed intermediate biasing loss brings more
regularization and contextualization to the network. Our method outperforms a
conventional contextual biasing baseline on the LibriSpeech corpus, achieving a
relative improvement of 22.5% in biased word error rate (B-WER) and up to 44%
compared to the non-contextual baseline with a biasing list size of 100.
Moreover, employing RNN-transducer-driven joint decoding further reduces the
unbiased word error rate (U-WER), resulting in a more robust network.
