  To assess the performance of load disaggregation algorithms it is common
practise to train a candidate algorithm on data from one or multiple households
and subsequently apply cross-validation by evaluating the classification and
energy estimation performance on unseen portions of the dataset derived from
the same households. With an emerging discussion of transferability in
Non-Intrusive Load Monitoring (NILM), there is a need for domain-specific
metrics to assess the performance of NILM algorithms on new test scenarios
being unseen buildings. In this paper, we discuss several metrics to assess the
generalisation ability of NILM algorithms. These metrics target different
aspects of performance evaluation in NILM and are meant to complement the
traditional performance evaluation approach. We demonstrate how our metrics can
be utilised to evaluate NILM algorithms by means of two case studies. We
conduct our studies on several energy consumption datasets and take into
consideration five state-of-the-art as well as four baseline NILM solutions.
Finally, we formulate research challenges for future work.
