  Conversational agents commonly utilize keyword spotting (KWS) to initiate
voice interaction with the user. For user experience and privacy
considerations, existing approaches to KWS largely focus on accuracy, which can
often come at the expense of introduced latency. To address this tradeoff, we
propose a novel approach to control KWS model latency and which generalizes to
any loss function without explicit knowledge of the keyword endpoint. Through a
single, tunable hyperparameter, our approach enables one to balance detection
latency and accuracy for the targeted application. Empirically, we show that
our approach gives superior performance under latency constraints when compared
to existing methods. Namely, we make a substantial 25\% relative false accepts
improvement for a fixed latency target when compared to the baseline
state-of-the-art. We also show that when our approach is used in conjunction
with a max-pooling loss, we are able to improve relative false accepts by 25 %
at a fixed latency when compared to cross entropy loss.
