  Gradient-based optimization has been critical to the success of machine
learning, updating a single set of parameters to minimize a single loss. A
growing number of applications rely on a generalization of this, where we have
a bilevel or nested optimization of which subsets of parameters update on
different objectives nested inside each other. We focus on motivating examples
of hyperparameter optimization and generative adversarial networks. However,
naively applying classical methods often fails when we look at solving these
nested problems on a large scale. In this thesis, we build tools for nested
optimization that scale to deep learning setups.
