  The vast majority of statistical theory on binary classification
characterizes performance in terms of accuracy. However, accuracy is known in
many cases to poorly reflect the practical consequences of classification
error, most famously in imbalanced binary classification, where data are
dominated by samples from one of two classes. The first part of this paper
derives a novel generalization of the Bayes-optimal classifier from accuracy to
any performance metric computed from the confusion matrix. Specifically, this
result (a) demonstrates that stochastic classifiers sometimes outperform the
best possible deterministic classifier and (b) removes an empirically
unverifiable absolute continuity assumption that is poorly understood but
pervades existing results. We then demonstrate how to use this generalized
Bayes classifier to obtain regret bounds in terms of the error of estimating
regression functions under uniform loss. Finally, we use these results to
develop some of the first finite-sample statistical guarantees specific to
imbalanced binary classification. Specifically, we demonstrate that optimal
classification performance depends on properties of class imbalance, such as a
novel notion called Uniform Class Imbalance, that have not previously been
formalized. We further illustrate these contributions numerically in the case
of $k$-nearest neighbor classification
