  We introduce UniLoss, a unified framework to generate surrogate losses for
training deep networks with gradient descent, reducing the amount of manual
design of task-specific surrogate losses. Our key observation is that in many
cases, evaluating a model with a performance metric on a batch of examples can
be refactored into four steps: from input to real-valued scores, from scores to
comparisons of pairs of scores, from comparisons to binary variables, and from
binary variables to the final performance metric. Using this refactoring we
generate differentiable approximations for each non-differentiable step through
interpolation. Using UniLoss, we can optimize for different tasks and metrics
using one unified framework, achieving comparable performance compared with
task-specific losses. We validate the effectiveness of UniLoss on three tasks
and four datasets. Code is available at
https://github.com/princeton-vl/uniloss.
