  Optimizers are essential components for successfully training deep neural
network models. In order to achieve the best performance from such models,
designers need to carefully choose the optimizer hyperparameters. However, this
can be a computationally expensive and time-consuming process. Although it is
known that all optimizer hyperparameters must be tuned for maximum performance,
there is still a lack of clarity regarding the individual influence of minor
priority hyperparameters, including the safeguard factor $\epsilon$ and
momentum factor $\beta$, in leading adaptive optimizers (specifically, those
based on the Adam optimizers). In this manuscript, we introduce a new framework
based on gradient histograms to analyze and justify important attributes of
adaptive optimizers, such as their optimal performance and the relationships
and dependencies among hyperparameters. Furthermore, we propose a novel
gradient histogram-based algorithm that automatically estimates a reduced and
accurate search space for the safeguard hyperparameter $\epsilon$, where the
optimal value can be easily found.
