  Automated heuristic design (AHD) has gained considerable attention for its
potential to automate the development of effective heuristics. The recent
advent of large language models (LLMs) has paved a new avenue for AHD, with
initial efforts focusing on framing AHD as an evolutionary program search (EPS)
problem. However, inconsistent benchmark settings, inadequate baselines, and a
lack of detailed component analysis have left the necessity of integrating LLMs
with search strategies and the true progress achieved by existing LLM-based EPS
methods to be inadequately justified. This work seeks to fulfill these research
queries by conducting a large-scale benchmark comprising four LLM-based EPS
methods and four AHD problems across nine LLMs and five independent runs. Our
extensive experiments yield meaningful insights, providing empirical grounding
for the importance of evolutionary search in LLM-based AHD approaches, while
also contributing to the advancement of future EPS algorithmic development. To
foster accessibility and reproducibility, we have fully open-sourced our
benchmark and corresponding results.
