  There are many existing retrieval and question answering datasets. However,
most of them either focus on ranked list evaluation or single-candidate
question answering. This divide makes it challenging to properly evaluate
approaches concerned with ranking documents and providing snippets or answers
for a given query. In this work, we present FiRA: a novel dataset of
Fine-Grained Relevance Annotations. We extend the ranked retrieval annotations
of the Deep Learning track of TREC 2019 with passage and word level graded
relevance annotations for all relevant documents. We use our newly created data
to study the distribution of relevance in long documents, as well as the
attention of annotators to specific positions of the text. As an example, we
evaluate the recently introduced TKL document ranking model. We find that
although TKL exhibits state-of-the-art retrieval results for long documents, it
misses many relevant passages.
