  Motivated by the problem of tuning hyperparameters in machine learning, we
present a new approach for gradually and adaptively optimizing an unknown
function using estimated gradients. We validate the empirical performance of
the proposed idea on both low and high dimensional problems. The experimental
results demonstrate the advantages of our approach for tuning high dimensional
hyperparameters in machine learning.
