  Unbiased learning-to-rank (ULTR) is a well-established framework for learning
from user clicks, which are often biased by the ranker collecting the data.
While theoretically justified and extensively tested in simulation, ULTR
techniques lack empirical validation, especially on modern search engines. The
Baidu-ULTR dataset released for the WSDM Cup 2023, collected from Baidu's
search engine, offers a rare opportunity to assess the real-world performance
of prominent ULTR techniques. Despite multiple submissions during the WSDM Cup
2023 and the subsequent NTCIR ULTRE-2 task, it remains unclear whether the
observed improvements stem from applying ULTR or other learning techniques.
  In this work, we revisit and extend the available experiments on the
Baidu-ULTR dataset. We find that standard unbiased learning-to-rank techniques
robustly improve click predictions but struggle to consistently improve ranking
performance, especially considering the stark differences obtained by choice of
ranking loss and query-document features. Our experiments reveal that gains in
click prediction do not necessarily translate to enhanced ranking performance
on expert relevance annotations, implying that conclusions strongly depend on
how success is measured in this benchmark.
