  Ranking evaluation metrics are a fundamental element of design and
improvement efforts in information retrieval. We observe that most popular
metrics disregard information portrayed in the scores used to derive rankings,
when available. This may pose a numerical scaling problem, causing an under- or
over-estimation of the evaluation depending on the degree of divergence between
the scores of ranked items. The purpose of this work is to propose a principled
way of quantifying multi-graded relevance judgments of items and enable a more
accurate penalization of ordering errors in rankings. We propose a data-driven
generation of relevance functions based on the degree of the divergence amongst
a set of items' scores and its application in the evaluation metric Normalized
Discounted Cumulative Gain (nDCG). We use synthetic data to demonstrate the
interest of our proposal and a combination of data on news items from Google
News and their respective popularity in Twitter to show its performance in
comparison to the standard nDCG. Results show that our proposal is capable of
providing a more fine-grained evaluation of rankings when compared to the
standard nDCG, and that the latter frequently under- or over-estimates its
evaluation scores in light of the divergence of items' scores.
