  Non-Intrusive Load Monitoring (NILM) is a field of research focused on
segregating constituent electrical loads in a system based only on their
aggregated signal. Significant computational resources and research time are
spent training models, often using as much data as possible, perhaps driven by
the preconception that more data equates to more accurate models and better
performing algorithms. When has enough prior training been done? When has a
NILM algorithm encountered new, unseen data? This work applies the notion of
Bayesian surprise to answer these questions which are important for both
supervised and unsupervised algorithms. We quantify the degree of surprise
between the predictive distribution (termed postdictive surprise), as well as
the transitional probabilities (termed transitional surprise), before and after
a window of observations. We compare the performance of several benchmark NILM
algorithms supported by NILMTK, in order to establish a useful threshold on the
two combined measures of surprise. We validate the use of transitional surprise
by exploring the performance of a popular Hidden Markov Model as a function of
surprise threshold. Finally, we explore the use of a surprise threshold as a
regularization technique to avoid overfitting in cross-dataset performance.
Although the generality of the specific surprise threshold discussed herein may
be suspect without further testing, this work provides clear evidence that a
point of diminishing returns of model performance with respect to dataset size
exists. This has implications for future model development, dataset
acquisition, as well as aiding in model flexibility during deployment.
