  We formalize and study the natural approach of designing convex surrogate
loss functions via embeddings, for problems such as classification, ranking, or
structured prediction. In this approach, one embeds each of the finitely many
predictions (e.g.\ rankings) as a point in $\mathbb{R}^d$, assigns the original
loss values to these points, and "convexifies" the loss in some way to obtain a
surrogate. We establish a strong connection between this approach and
polyhedral (piecewise-linear convex) surrogate losses. Given any polyhedral
loss $L$, we give a construction of a link function through which $L$ is a
consistent surrogate for the loss it embeds. Conversely, we show how to
construct a consistent polyhedral surrogate for any given discrete loss. Our
framework yields succinct proofs of consistency or inconsistency of various
polyhedral surrogates in the literature, and for inconsistent surrogates, it
further reveals the discrete losses for which these surrogates are consistent.
We show some additional structure of embeddings, such as the equivalence of
embedding and matching Bayes risks, and the equivalence of various notions of
non-redudancy. Using these results, we establish that indirect elicitation, a
necessary condition for consistency, is also sufficient when working with
polyhedral surrogates.
