  In this study, we investigate learning rate adaption at different levels
based on the hyper-gradient descent framework and propose a method that
adaptively learns the optimizer parameters by combining multiple levels of
learning rates with hierarchical structures. Meanwhile, we show the
relationship between regularizing over-parameterized learning rates and
building combinations of adaptive learning rates at different levels. The
experiments on several network architectures, including feed-forward networks,
LeNet-5 and ResNet-18/34, show that the proposed multi-level adaptive approach
can outperform baseline adaptive methods in a variety of circumstances.
