  The area under the ROC curve is a common measure that is often used to rank
the relative performance of different binary classifiers. However, as has been
also previously noted, it can be a measure that ill-captures the benefits of
different classifiers when either the true class values or misclassification
costs are highly unbalanced between the two classes. We introduce a third
dimension to capture these costs, and lift the ROC curve to a ROC surface in a
natural way. We study both this surface and introduce the VOROS, the volume
over this ROC surface, as a 3D generalization of the 2D area under the ROC
curve. For problems where there are only bounds on the expected costs or class
imbalances, we restrict consideration to the volume of the appropriate
subregion of the ROC surface. We show how the VOROS can better capture the
costs of different classifiers on both a classical and a modern example
dataset.
