  Evaluation of search engines relies on assessments of search results for
selected test queries, from which we would ideally like to draw conclusions in
terms of relevance of the results for general (e.g., future, unknown) users. In
practice however, most evaluation scenarios only allow us to conclusively
determine the relevance towards the particular assessor that provided the
judgments. A factor that cannot be ignored when extending conclusions made from
assessors towards users, is the possible disagreement on relevance, assuming
that a single gold truth label does not exist. This paper presents and analyzes
the Predicted Relevance Model (PRM), which allows predicting a particular
result's relevance for a random user, based on an observed assessment and
knowledge on the average disagreement between assessors. With the PRM, existing
evaluation metrics designed to measure binary assessor relevance, can be
transformed into more robust and effectively graded measures that evaluate
relevance towards a random user. It also leads to a principled way of
quantifying multiple graded or categorical relevance levels for use as gains in
established graded relevance measures, such as normalized discounted cumulative
gain (nDCG), which nowadays often use heuristic and data-independent gain
values. Given a set of test topics with graded relevance judgments, the PRM
allows evaluating systems on different scenarios, such as their capability of
retrieving top results, or how well they are able to filter out non-relevant
ones. Its use in actual evaluation scenarios is illustrated on several
information retrieval test collections.
