  Evaluation metrics in machine learning are often hardly taken as loss
functions, as they could be non-differentiable and non-decomposable, e.g.,
average precision and F1 score. This paper aims to address this problem by
revisiting the surrogate loss learning, where a deep neural network is employed
to approximate the evaluation metrics. Instead of pursuing an exact recovery of
the evaluation metric through a deep neural network, we are reminded of the
purpose of the existence of these evaluation metrics, which is to distinguish
whether one model is better or worse than another. In this paper, we show that
directly maintaining the relation of models between surrogate losses and
metrics suffices, and propose a rank correlation-based optimization method to
maximize this relation and learn surrogate losses. Compared to previous works,
our method is much easier to optimize and enjoys significant efficiency and
performance gains. Extensive experiments show that our method achieves
improvements on various tasks including image classification and neural machine
translation, and even outperforms state-of-the-art methods on human pose
estimation and machine reading comprehension tasks. Code is available at:
https://github.com/hunto/ReLoss.
