  Surrogate risk minimization is an ubiquitous paradigm in supervised machine
learning, wherein a target problem is solved by minimizing a surrogate loss on
a dataset. Surrogate regret bounds, also called excess risk bounds, are a
common tool to prove generalization rates for surrogate risk minimization.
While surrogate regret bounds have been developed for certain classes of loss
functions, such as proper losses, general results are relatively sparse. We
provide two general results. The first gives a linear surrogate regret bound
for any polyhedral (piecewise-linear and convex) surrogate, meaning that
surrogate generalization rates translate directly to target rates. The second
shows that for sufficiently non-polyhedral surrogates, the regret bound is a
square root, meaning fast surrogate generalization rates translate to slow
rates for the target. Together, these results suggest polyhedral surrogates are
optimal in many cases.
