  This paper reports on an effort of reproducing the organizers' baseline as
well as the top performing participant submission at the 2021 edition of the
TREC Conversational Assistance track. TREC systems are commonly regarded as
reference points for effectiveness comparison. Yet, the papers accompanying
them have less strict requirements than peer-reviewed publications, which can
make reproducibility challenging. Our results indicate that key practical
information is indeed missing. While the results can be reproduced within a 19%
relative margin with respect to the main evaluation measure, the relative
difference between the baseline and the top performing approach shrinks from
the reported 18% to 5%. Additionally, we report on a new set of experiments
aimed at understanding the impact of various pipeline components. We show that
end-to-end system performance can indeed benefit from advanced retrieval
techniques in either stage of a two-stage retrieval pipeline. We also measure
the impact of the dataset used for fine-tuning the query rewriter and find that
employing different query rewriting methods in different stages of the
retrieval pipeline might be beneficial. Moreover, these results are shown to
generalize across the 2020 and 2021 editions of the track. We conclude our
study with a list of lessons learned and practical suggestions.
