  In the recent years, various gradient descent algorithms including the
methods of gradient descent, gradient descent with momentum, adaptive gradient
(AdaGrad), root-mean-square propagation (RMSProp) and adaptive moment
estimation (Adam) have been applied to the parameter optimization of several
deep learning models with higher accuracies or lower errors. These optimization
algorithms may need to set the values of several hyperparameters which include
a learning rate, momentum coefficients, etc. Furthermore, the convergence speed
and solution accuracy may be influenced by the values of hyperparameters.
Therefore, this study proposes an analytical framework to use mathematical
models for analyzing the mean error of each objective function based on various
gradient descent algorithms. Moreover, the suitable value of each
hyperparameter could be determined by minimizing the mean error. The principles
of hyperparameter value setting have been generalized based on analysis results
for model optimization. The experimental results show that higher efficiency
convergences and lower errors can be obtained by the proposed method.
