  In this position paper we argue that certain aspects of relevance assessment
in the evaluation of IR systems are oversimplified and that human assessments
represented by qrels should be augmented to take account of contextual factors
and the subjectivity of the task at hand. We propose enhancing test collections
used in evaluation with information related to human assessors and their
interpretation of the task. Such augmented collections would provide a more
realistic and user-focused evaluation, enabling us to better understand the
evaluation process, the performance of systems and user interactions. A first
step is to conduct user studies to examine in more detail what people actually
do when we ask them to judge the relevance of a document.
