  Hyperparameter optimization can be formulated as a bilevel optimization
problem, where the optimal parameters on the training set depend on the
hyperparameters. We aim to adapt regularization hyperparameters for neural
networks by fitting compact approximations to the best-response function, which
maps hyperparameters to optimal weights and biases. We show how to construct
scalable best-response approximations for neural networks by modeling the
best-response as a single network whose hidden units are gated conditionally on
the regularizer. We justify this approximation by showing the exact
best-response for a shallow linear network with L2-regularized Jacobian can be
represented by a similar gating mechanism. We fit this model using a
gradient-based hyperparameter optimization algorithm which alternates between
approximating the best-response around the current hyperparameters and
optimizing the hyperparameters using the approximate best-response function.
Unlike other gradient-based approaches, we do not require differentiating the
training loss with respect to the hyperparameters, allowing us to tune discrete
hyperparameters, data augmentation hyperparameters, and dropout probabilities.
Because the hyperparameters are adapted online, our approach discovers
hyperparameter schedules that can outperform fixed hyperparameter values.
Empirically, our approach outperforms competing hyperparameter optimization
methods on large-scale deep learning problems. We call our networks, which
update their own hyperparameters online during training, Self-Tuning Networks
(STNs).
