  E-commerce search systems such as Taobao Search, the largest e-commerce
searching system in China, aim at providing users with the most preferred items
(e.g., products). Due to the massive data and limited time for response, a
typical industrial ranking system consists of three or more modules, including
matching, pre-ranking, and ranking. The pre-ranking is widely considered a
mini-ranking module, as it needs to rank hundreds of times more items than the
ranking under limited latency. Existing researches focus on building a lighter
model that imitates the ranking model. As such, the metric of a pre-ranking
model follows the ranking model using Area Under ROC (AUC) for offline
evaluation. However, such a metric is inconsistent with online A/B tests in
practice, so engineers have to perform costly online tests to reach a
convincing conclusion. In our work, we rethink the role of the pre-ranking. We
argue that the primary goal of the pre-ranking stage is to return an optimal
unordered set rather than an ordered list of items because it is the ranking
that determines the final exposures. Since AUC measures the quality of an
ordered item list, it is not suitable for evaluating the quality of the output
unordered set. This paper proposes a new evaluation metric called All-Scenario
Hitrate (ASH) for pre-ranking. ASH is proven effective in the offline
evaluation and consistent with online A/B tests based on numerous experiments
in Taobao Search. We also introduce an all-scenario-based multi-objective
learning framework (ASMOL), which improves the ASH significantly. Surprisingly,
the new pre-ranking model can outperforms the ranking model when outputting
thousands of items. The phenomenon validates that the pre-ranking stage should
not imitate the ranking blindly. With the improvements in ASH consistently
translating to online improvement, it makes a 1.2% GMV improvement on Taobao
Search.
