  The minimization of loss functions is the heart and soul of Machine Learning.
In this paper, we propose an off-the-shelf optimization approach that can
minimize virtually any non-differentiable and non-decomposable loss function
(e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation
Coefficient, etc.) seamlessly. Our strategy learns smooth relaxation versions
of the true losses by approximating them through a surrogate neural network.
The proposed loss networks are set-wise models which are invariant to the order
of mini-batch instances. Ultimately, the surrogate losses are learned jointly
with the prediction model via bilevel optimization. Empirical results on
multiple datasets with diverse real-life loss functions compared with
state-of-the-art baselines demonstrate the efficiency of learning surrogate
losses.
