  Model evolution and constant availability of data are two common phenomena in
large-scale real-world machine learning applications, e.g. ads and
recommendation systems. To adapt, the real-world system typically retrain with
all available data and online learn with recently available data to update the
models periodically with the goal of better serving performance. In this paper,
we propose a novel framework, named Confidence Ranking, which designs the
optimization objective as a ranking function with two different models. Our
confidence ranking loss allows direct optimization of the logits output for
different convex surrogate functions of metrics, e.g. AUC and Accuracy
depending on the target task and dataset. Armed with our proposed methods, our
experiments show that the introduction of confidence ranking loss can
outperform all baselines on the CTR prediction tasks of public and industrial
datasets. This framework has been deployed in the advertisement system of
JD.com to serve the main traffic in the fine-rank stage.
