  Automating algorithm configuration is growing increasingly necessary as
algorithms come with more and more tunable parameters. It is common to tune
parameters using machine learning, optimizing performance metrics such as
runtime and solution quality. The training set consists of problem instances
from the specific domain at hand. We investigate a fundamental question about
these techniques: how large should the training set be to ensure that a
parameter's average empirical performance over the training set is close to its
expected, future performance? We answer this question for algorithm
configuration problems that exhibit a widely-applicable structure: the
algorithm's performance as a function of its parameters can be approximated by
a "simple" function. We show that if this approximation holds under the
L-infinity norm, we can provide strong sample complexity bounds. On the flip
side, if the approximation holds only under the L-p norm for p smaller than
infinity, it is not possible to provide meaningful sample complexity bounds in
the worst case. We empirically evaluate our bounds in the context of integer
programming, one of the most powerful tools in computer science. Via
experiments, we obtain sample complexity bounds that are up to 700 times
smaller than the previously best-known bounds.
