  Bi-level optimization, especially the gradient-based category, has been
widely used in the deep learning community including hyperparameter
optimization and meta-knowledge extraction. Bi-level optimization embeds one
problem within another and the gradient-based category solves the outer-level
task by computing the hypergradient, which is much more efficient than
classical methods such as the evolutionary algorithm. In this survey, we first
give a formal definition of the gradient-based bi-level optimization. Next, we
delineate criteria to determine if a research problem is apt for bi-level
optimization and provide a practical guide on structuring such problems into a
bi-level optimization framework, a feature particularly beneficial for those
new to this domain. More specifically, there are two formulations: the
single-task formulation to optimize hyperparameters such as regularization
parameters and the distilled data, and the multi-task formulation to extract
meta-knowledge such as the model initialization. With a bi-level formulation,
we then discuss four bi-level optimization solvers to update the outer variable
including explicit gradient update, proxy update, implicit function update, and
closed-form update. Finally, we wrap up the survey by highlighting two
prospective future directions: (1) Effective Data Optimization for Science
examined through the lens of task formulation. (2) Accurate Explicit Proxy
Update analyzed from an optimization standpoint.
