  This paper addresses the challenge of optimizing meta-parameters (i.e.,
hyperparameters) in machine learning algorithms, a critical factor influencing
training efficiency and model performance. Moving away from the computationally
expensive traditional meta-parameter search methods, we introduce MetaOptimize
framework that dynamically adjusts meta-parameters, particularly step sizes
(also known as learning rates), during training. More specifically,
MetaOptimize can wrap around any first-order optimization algorithm, tuning
step sizes on the fly to minimize a specific form of regret that accounts for
long-term effect of step sizes on training, through a discounted sum of future
losses. We also introduce low complexity variants of MetaOptimize that, in
conjunction with its adaptability to multiple optimization algorithms,
demonstrate performance competitive to those of best hand-crafted learning rate
schedules across various machine learning applications.
