  The recognition of rare named entities, such as personal names and
terminologies, is challenging for automatic speech recognition (ASR) systems,
especially when they are not frequently observed in the training data. In this
paper, we introduce keyword spotting enhanced Whisper (KWS-Whisper), a novel
ASR system that leverages the Whisper model and performs open-vocabulary
keyword spotting (OV-KWS) on the hidden states of the Whisper encoder to
recognize user-defined named entities. These entities serve as prompts for the
Whisper decoder. To optimize the model, we propose a multitask training
approach that learns OV-KWS and contextual-ASR tasks. We evaluate our approach
on Chinese Aishell hot word subsets and two internal code-switching test sets
and show that it significantly improves the entity recall compared to the
original Whisper model. Moreover, we demonstrate that the OV-KWS can be a
plug-and-play module to enhance the ASR error correction methods and frozen
Whisper models.
