  Learning-to-rank (LTR) has become a key technology in E-commerce
applications. Most existing LTR approaches follow a supervised learning
paradigm from offline labeled data collected from the online system. However,
it has been noticed that previous LTR models can have a good validation
performance over offline validation data but have a poor online performance,
and vice versa, which implies a possible large inconsistency between the
offline and online evaluation. We investigate and confirm in this paper that
such inconsistency exists and can have a significant impact on AliExpress
Search. Reasons for the inconsistency include the ignorance of item context
during the learning, and the offline data set is insufficient for learning the
context. Therefore, this paper proposes an evaluator-generator framework for
LTR with item context. The framework consists of an evaluator that generalizes
to evaluate recommendations involving the context, and a generator that
maximizes the evaluator score by reinforcement learning, and a discriminator
that ensures the generalization of the evaluator. Extensive experiments in
simulation environments and AliExpress Search online system show that, firstly,
the classic data-based metrics on the offline dataset can show significant
inconsistency with online performance, and can even be misleading. Secondly,
the proposed evaluator score is significantly more consistent with the online
performance than common ranking metrics. Finally, as the consequence, our
method achieves a significant improvement (\textgreater$2\%$) in terms of
Conversion Rate (CR) over the industrial-level fine-tuned model in online A/B
tests.
