  Non-Intrusive Load Monitoring (NILM) comprises of a set of techniques that
provide insights into the energy consumption of households and industrial
facilities. Latest contributions show significant improvements in terms of
accuracy and generalisation abilities. Despite all progress made concerning
disaggregation techniques, performance evaluation and comparability remains an
open research question. The lack of standardisation and consensus on evaluation
procedures makes reproducibility and comparability extremely difficult. In this
paper, we draw attention to comparability in NILM with a focus on highlighting
the considerable differences amongst common energy datasets used to test the
performance of algorithms. We divide discussion on comparability into data
aspects, performance metrics, and give a close view on evaluation processes.
Detailed information on pre-processing as well as data cleaning methods, the
importance of unified performance reporting, and the need for complexity
measures in load disaggregation are found to be the most urgent issues in
NILM-related research. In addition, our evaluation suggests that datasets
should be chosen carefully. We conclude by formulating suggestions for future
work to enhance comparability.
