  Evaluation metrics are an essential part of a ranking system, and in the past
many evaluation metrics have been proposed in information retrieval and Web
search. Discounted Cumulated Gains (DCG) has emerged as one of the evaluation
metrics widely adopted for evaluating the performance of ranking functions used
in Web search. However, the two sets of parameters, gain values and discount
factors, used in DCG are determined in a rather ad-hoc way. In this paper we
first show that DCG is generally not coherent, meaning that comparing the
performance of ranking functions using DCG very much depends on the particular
gain values and discount factors used. We then propose a novel methodology that
can learn the gain values and discount factors from user preferences over
rankings. Numerical simulations illustrate the effectiveness of our proposed
methods. Please contact the authors for the full version of this work.
