  Click-through rate (CTR) prediction is a critical task for many applications,
as its accuracy has a direct impact on user experience and platform revenue. In
recent years, CTR prediction has been widely studied in both academia and
industry, resulting in a wide variety of CTR prediction models. Unfortunately,
there is still a lack of standardized benchmarks and uniform evaluation
protocols for CTR prediction research. This leads to non-reproducible or even
inconsistent experimental results among existing studies, which largely limits
the practical value and potential impact of their research. In this work, we
aim to perform open benchmarking for CTR prediction and present a rigorous
comparison of different models in a reproducible manner. To this end, we ran
over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate
24 existing models on multiple datasets and settings. Surprisingly, our
experiments show that with sufficient hyper-parameter search and model tuning,
many deep models have smaller differences than expected. The results also
reveal that making real progress on the modeling of CTR prediction is indeed a
very challenging research task. We believe that our benchmarking work could not
only allow researchers to gauge the effectiveness of new models conveniently
but also make them fairly compare with the state of the arts. We have publicly
released the benchmarking code, evaluation protocols, and hyper-parameter
settings of our work to promote reproducible research in this field.
