  In this work, we present Slimmable Neural Networks applied to the problem of
small-footprint keyword spotting. We show that slimmable neural networks allow
us to create super-nets from Convolutioanl Neural Networks and Transformers,
from which sub-networks of different sizes can be extracted. We demonstrate the
usefulness of these models on in-house Alexa data and Google Speech Commands,
and focus our efforts on models for the on-device use case, limiting ourselves
to less than 250k parameters. We show that slimmable models can match (and in
some cases, outperform) models trained from scratch. Slimmable neural networks
are therefore a class of models particularly useful when the same functionality
is to be replicated at different memory and compute budgets, with different
accuracy requirements.
