  Despite improvements to the generalization performance of automated speech
recognition (ASR) models, specializing ASR models for downstream tasks remains
a challenging task, primarily due to reduced data availability (necessitating
increased data collection), and rapidly shifting data distributions (requiring
more frequent model fine-tuning). In this work, we investigate the potential of
leveraging external knowledge, particularly through off-policy key-value stores
generated with text-to-speech methods, to allow for flexible post-training
adaptation to new data distributions. In our approach, audio embeddings
captured from text-to-speech, along with semantic text embeddings, are used to
bias ASR via an approximate k-nearest-neighbor (KNN) based attentive fusion
step. Our experiments on LibiriSpeech and in-house voice assistant/search
datasets show that the proposed approach can reduce domain adaptation time by
up to 1K GPU-hours while providing up to 3% WER improvement compared to a
fine-tuning baseline, suggesting a promising approach for adapting production
ASR systems in challenging zero and few-shot scenarios.
