  Evaluation of keyword spotting (KWS) systems that detect keywords in speech
is a challenging task under realistic privacy constraints. The KWS is designed
to only collect data when the keyword is present, limiting the availability of
hard samples that may contain false negatives, and preventing direct estimation
of model recall from production data. Alternatively, complementary data
collected from other sources may not be fully representative of the real
application. In this work, we propose an evaluation technique which we call
AB/BA analysis. Our framework evaluates a candidate KWS model B against a
baseline model A, using cross-dataset offline decoding for relative recall
estimation, without requiring negative examples. Moreover, we propose a
formulation with assumptions that allow estimation of relative false positive
rate between models with low variance even when the number of false positives
is small. Finally, we propose to leverage machine-generated soft labels, in a
technique we call Semi-Supervised AB/BA analysis, that improves the analysis
time, privacy, and cost. Experiments with both simulation and real data show
that AB/BA analysis is successful at measuring recall improvement in
conjunction with the trade-off in relative false positive rate.
