  Recent studies on Learning to Optimize (L2O) suggest a promising path to
automating and accelerating the optimization procedure for complicated tasks.
Existing L2O models parameterize optimization rules by neural networks, and
learn those numerical rules via meta-training. However, they face two common
pitfalls: (1) scalability: the numerical rules represented by neural networks
create extra memory overhead for applying L2O models, and limit their
applicability to optimizing larger tasks; (2) interpretability: it is unclear
what an L2O model has learned in its black-box optimization rule, nor is it
straightforward to compare different L2O models in an explainable way. To avoid
both pitfalls, this paper proves the concept that we can "kill two birds by one
stone", by introducing the powerful tool of symbolic regression to L2O. In this
paper, we establish a holistic symbolic representation and analysis framework
for L2O, which yields a series of insights for learnable optimizers. Leveraging
our findings, we further propose a lightweight L2O model that can be
meta-trained on large-scale problems and outperformed human-designed and tuned
optimizers. Our work is set to supply a brand-new perspective to L2O research.
Codes are available at:
https://github.com/VITA-Group/Symbolic-Learning-To-Optimize.
