DACBench: A Benchmark Library for Dynamic Algorithm Configuration
  Dynamic Algorithm Configuration (DAC) aims to dynamically control a target
algorithm's hyperparameters in order to improve its performance. Several
theoretical and empirical results have demonstrated the benefits of dynamically
controlling hyperparameters in domains like evolutionary computation, AI
Planning or deep learning. Replicating these results, as well as studying new
methods for DAC, however, is difficult since existing benchmarks are often
specialized and incompatible with the same interfaces. To facilitate
benchmarking and thus research on DAC, we propose DACBench, a benchmark library
that seeks to collect and standardize existing DAC benchmarks from different AI
domains, as well as provide a template for new ones. For the design of
DACBench, we focused on important desiderata, such as (i) flexibility, (ii)
reproducibility, (iii) extensibility and (iv) automatic documentation and
visualization. To show the potential, broad applicability and challenges of
DAC, we explore how a set of six initial benchmarks compare in several
dimensions of difficulty.
