  In the learning to learn (L2L) framework, we cast the design of optimization
algorithms as a machine learning problem and use deep neural networks to learn
the update rules. In this paper, we extend the L2L framework to zeroth-order
(ZO) optimization setting, where no explicit gradient information is available.
Our learned optimizer, modeled as recurrent neural network (RNN), first
approximates gradient by ZO gradient estimator and then produces parameter
update utilizing the knowledge of previous iterations. To reduce high variance
effect due to ZO gradient estimator, we further introduce another RNN to learn
the Gaussian sampling rule and dynamically guide the query direction sampling.
Our learned optimizer outperforms hand-designed algorithms in terms of
convergence rate and final solution on both synthetic and practical ZO
optimization tasks (in particular, the black-box adversarial attack task, which
is one of the most widely used tasks of ZO optimization). We finally conduct
extensive analytical experiments to demonstrate the effectiveness of our
proposed optimizer.
