  Numerous web applications rely on solving combinatorial optimization
problems, such as energy cost-aware scheduling, budget allocation on web
advertising, and graph matching on social networks. However, many optimization
problems involve unknown coefficients, and improper predictions of these
factors may lead to inferior decisions which may cause energy wastage,
inefficient resource allocation, inappropriate matching in social networks,
etc. Such a research topic is referred to as "Predict-Then-Optimize (PTO)"
which considers the performance of prediction and decision-making in a unified
system. A noteworthy recent development is the end-to-end methods by directly
optimizing the ultimate decision quality which claims to yield better results
in contrast to the traditional two-stage approach. However, the evaluation
benchmarks in this field are fragmented and the effectiveness of various models
in different scenarios remains unclear, hindering the comprehensive assessment
and fast deployment of these methods. To address these issues, we provide a
comprehensive categorization of current approaches and integrate existing
experimental scenarios to establish a unified benchmark, elucidating the
circumstances under which end-to-end training yields improvements, as well as
the contexts in which it performs ineffectively. We also introduce a new
dataset for the industrial combinatorial advertising problem for inclusive
finance to open-source. We hope the rethinking and benchmarking of PTO could
facilitate more convenient evaluation and deployment, and inspire further
improvements both in the academy and industry within this field.
