  Industrial ranking systems, such as advertising systems, rank items by
aggregating multiple objectives into one final objective to satisfy user demand
and commercial intent. Cascade architecture, composed of retrieval,
pre-ranking, and ranking stages, is usually adopted to reduce the computational
cost. Each stage may employ various models for different objectives and
calculate the final objective by aggregating these models' outputs. The
multi-stage ranking strategy causes a new problem - the ranked lists of the
ranking stage and previous stages may be inconsistent. For example, items that
should be ranked at the top of the ranking stage may be ranked at the bottom of
previous stages. In this paper, we focus on the \textbf{ranking consistency}
between the pre-ranking and ranking stages. Specifically, we formally define
the problem of ranking consistency and propose the Ranking Consistency Score
(RCS) metric for evaluation. We demonstrate that ranking consistency has a
direct impact on online performance. Compared with the traditional evaluation
manner that mainly focuses on the individual ranking quality of every
objective, RCS considers the ranking consistency of the fused final objective,
which is more proper for evaluation. Finally, to improve the ranking
consistency, we propose several methods from the perspective of sample
selection and learning algorithms. Experimental results on one of the biggest
industrial E-commerce platforms in China validate the efficacy of the proposed
metrics and methods.
