  Recently, using automatic configuration tuning to improve the performance of
modern database management systems (DBMSs) has attracted increasing interest
from the database community. This is embodied with a number of systems
featuring advanced tuning capabilities being developed. However, it remains a
challenge to select the best solution for database configuration tuning,
considering the large body of algorithm choices. In addition, beyond the
applications on database systems, we could find more potential algorithms
designed for configuration tuning. To this end, this paper provides a
comprehensive evaluation of configuration tuning techniques from a broader
perspective, hoping to better benefit the database community. In particular, we
summarize three key modules of database configuration tuning systems and
conduct extensive ablation studies using various challenging cases. Our
evaluation demonstrates that the hyper-parameter optimization algorithms can be
borrowed to further enhance the database configuration tuning. Moreover, we
identify the best algorithm choices for different modules. Beyond the
comprehensive evaluations, we offer an efficient and unified database
configuration tuning benchmark via surrogates that reduces the evaluation cost
to a minimum, allowing for extensive runs and analysis of new techniques.
