  Document ranking experiments should be repeatable. However, the interaction
between multi-threaded indexing and score ties during retrieval may yield
non-deterministic rankings, making repeatability not as trivial as one might
imagine. In the context of the open-source Lucene search engine, score ties are
broken by internal document ids, which are assigned at index time. Due to
multi-threaded indexing, which makes experimentation with large modern document
collections practical, internal document ids are not assigned consistently
between different index instances of the same collection, and thus score ties
are broken unpredictably. This short paper examines the effectiveness impact of
such score ties, quantifying the variability that can be attributed to this
phenomenon. The obvious solution to this non-determinism and to ensure
repeatable document ranking is to break score ties using external collection
document ids. This approach, however, comes with measurable efficiency costs
due to the necessity of consulting external identifiers during query
evaluation.
