  Connectionist Temporal Classification (CTC) models are popular for their
balance between speed and performance for Automatic Speech Recognition (ASR).
However, these CTC models still struggle in other areas, such as
personalization towards custom words. A recent approach explores Contextual
Adapters, wherein an attention-based biasing model for CTC is used to improve
the recognition of custom entities. While this approach works well with enough
data, we showcase that it isn't an effective strategy for low-resource
languages. In this work, we propose a supervision loss for smoother training of
the Contextual Adapters. Further, we explore a multilingual strategy to improve
performance with limited training data. Our method achieves 48% F1 improvement
in retrieving unseen custom entities for a low-resource language.
Interestingly, as a by-product of training the Contextual Adapters, we see a
5-11% Word Error Rate (WER) reduction in the performance of the base CTC model
as well.
