  Unfortunately, the official English (sub)task results reported in the
NTCIR-14 WWW-2, NTCIR-15 WWW-3, and NTCIR-16 WWW-4 overview papers are
incorrect due to noise in the official qrels files; this paper reports results
based on the corrected qrels files. The noise is due to a fatal bug in the
backend of our relevance assessment interface. More specifically, at WWW-2,
WWW-3, and WWW-4, two versions of pool files were created for each English
topic: a PRI ("prioritised") file, which uses the NTCIRPOOL script to
prioritise likely relevant documents, and a RND ("randomised") file, which
randomises the pooled documents. This was done for the purpose of studying the
effect of document ordering for relevance assessors. However, the programmer
who wrote the interface backend assumed that a combination of a topic ID and a
document rank in the pool file uniquely determines a document ID; this is
obviously incorrect as we have two versions of pool files. The outcome is that
all the PRI-based relevance labels for the WWW-2 test collection are incorrect
(while all the RND-based relevance labels are correct), and all the RND-based
relevance labels for the WWW-3 and WWW-4 test collections are incorrect (while
all the PRI-based relevance labels are correct). This bug was finally
discovered at the NTCIR-16 WWW-4 task when the first seven authors of this
paper served as Gold assessors (i.e., topic creators who define what is
relevant) and closely examined the disagreements with Bronze assessors (i.e.,
non-topic-creators; non-experts). We would like to apologise to the WWW
participants and the NTCIR chairs for the inconvenience and confusion caused
due to this bug.
