  Scoring functions are used to evaluate and compare partially probabilistic
forecasts. We investigate the use of rank-sum functions such as empirical Area
Under the Curve (AUC), a widely-used measure of classification performance, as
a scoring function for the prediction of probabilities of a set of binary
outcomes. It is shown that the AUC is not generally a proper scoring function,
that is, under certain circumstances it is possible to improve on the expected
AUC by modifying the quoted probabilities from their true values. However with
some restrictions, or with certain modifications, it can be made proper.
