  In machine learning, the choice of a learning algorithm that is suitable for
the application domain is critical. The performance metric used to compare
different algorithms must also reflect the concerns of users in the application
domain under consideration. In this work, we propose a novel probability-based
performance metric called Relevance Score for evaluating supervised learning
algorithms. We evaluate the proposed metric through empirical analysis on a
dataset gathered from an intelligent lighting pilot installation. In comparison
to the commonly used Classification Accuracy metric, the Relevance Score proves
to be more appropriate for a certain class of applications.
