  Despite its troubled past, the AOL Query Log continues to be an important
resource to the research community -- particularly for tasks like search
personalisation. When using the query log these ranking experiments, little
attention is usually paid to the document corpus. Recent work typically uses a
corpus containing versions of the documents collected long after the log was
produced. Given that web documents are prone to change over time, we study the
differences present between a version of the corpus containing documents as
they appeared in 2017 (which has been used by several recent works) and a new
version we construct that includes documents close to as they appeared at the
time the query log was produced (2006). We demonstrate that this new version of
the corpus has a far higher coverage of documents present in the original log
(93%) than the 2017 version (55%). Among the overlapping documents, the content
often differs substantially. Given these differences, we re-conduct session
search experiments that originally used the 2017 corpus and find that when
using our corpus for training or evaluation, system performance improves. We
place the results in context by introducing recent adhoc ranking baselines. We
also confirm the navigational nature of the queries in the AOL corpus by
showing that including the URL substantially improves performance across a
variety of models. Our version of the corpus can be easily reconstructed by
other researchers and is included in the ir-datasets package.
