  Keyword Spotting (KWS) models are becoming increasingly integrated into
various systems, e.g. voice assistants. To achieve satisfactory performance,
these models typically rely on a large amount of labelled data, limiting their
applications only to situations where such data is available. Self-supervised
Learning (SSL) methods can mitigate such a reliance by leveraging
readily-available unlabelled data. Most SSL methods for speech have primarily
been studied for large models, whereas this is not ideal, as compact KWS models
are generally required. This paper explores the effectiveness of SSL on small
models for KWS and establishes that SSL can enhance the performance of small
KWS models when labelled data is scarce. We pretrain three compact
transformer-based KWS models using Data2Vec, and fine-tune them on a
label-deficient setup of the Google Speech Commands data set. It is found that
Data2Vec pretraining leads to a significant increase in accuracy, with
label-deficient scenarios showing an improvement of 8.22% 11.18% absolute
accuracy.
