  In this work we explore the latency and accuracy of keyword spotting (KWS)
models in streaming and non-streaming modes on mobile phones. NN model
conversion from non-streaming mode (model receives the whole input sequence and
then returns the classification result) to streaming mode (model receives
portion of the input sequence and classifies it incrementally) may require
manual model rewriting. We address this by designing a Tensorflow/Keras based
library which allows automatic conversion of non-streaming models to streaming
ones with minimum effort. With this library we benchmark multiple KWS models in
both streaming and non-streaming modes on mobile phones and demonstrate
different tradeoffs between latency and accuracy. We also explore novel KWS
models with multi-head attention which reduce the classification error over the
state-of-art by 10% on Google speech commands data sets V2. The streaming
library with all experiments is open-sourced.
