  Hyperparameter tuning, particularly the selection of an appropriate learning
rate in adaptive gradient training methods, remains a challenge. To tackle this
challenge, in this paper, we propose a novel parameter-free optimizer,
\textsc{AdamG} (Adam with the golden step size), designed to automatically
adapt to diverse optimization problems without manual tuning. The core
technique underlying \textsc{AdamG} is our golden step size derived for the
AdaGrad-Norm algorithm, which is expected to help AdaGrad-Norm preserve the
tuning-free convergence and approximate the optimal step size in expectation
w.r.t. various optimization scenarios. To better evaluate tuning-free
performance, we propose a novel evaluation criterion, \textit{reliability}, to
comprehensively assess the efficacy of parameter-free optimizers in addition to
classical performance criteria. Empirical results demonstrate that compared
with other parameter-free baselines, \textsc{AdamG} achieves superior
performance, which is consistently on par with Adam using a manually tuned
learning rate across various optimization tasks.
