  This paper explores connections between margin-based loss functions and
consistency in binary classification and regression applications. It is shown
that a large class of margin-based loss functions for binary
classification/regression result in estimating scores equivalent to
log-likelihood scores weighted by an even function. A simple characterization
for conformable (consistent) loss functions is given, which allows for
straightforward comparison of different losses, including exponential loss,
logistic loss, and others. The characterization is used to construct a new
Huber-type loss function for the logistic model. A simple relation between the
margin and standardized logistic regression residuals is derived, demonstrating
that all margin-based loss can be viewed as loss functions of squared
standardized logistic regression residuals. The relation provides new,
straightforward interpretations for exponential and logistic loss, and aids in
understanding why exponential loss is sensitive to outliers. In particular, it
is shown that minimizing empirical exponential loss is equivalent to minimizing
the sum of squared standardized logistic regression residuals. The relation
also provides new insight into the AdaBoost algorithm.
