  Several performance measures can be used for evaluating classification
results: accuracy, F-measure, and many others. Can we say that some of them are
better than others, or, ideally, choose one measure that is best in all
situations? To answer this question, we conduct a systematic analysis of
classification performance measures: we formally define a list of desirable
properties and theoretically analyze which measures satisfy which properties.
We also prove an impossibility theorem: some desirable properties cannot be
simultaneously satisfied. Finally, we propose a new family of measures
satisfying all desirable properties except one. This family includes the
Matthews Correlation Coefficient and a so-called Symmetric Balanced Accuracy
that was not previously used in classification literature. We believe that our
systematic approach gives an important tool to practitioners for adequately
evaluating classification results.
