  Artificial neural networks (ANN) have been used in many applications such
like handwriting recognition and speech recognition. It is well-known that
learning rate is a crucial value in the training procedure for artificial
neural networks. It is shown that the initial value of learning rate can
confoundedly affect the final result and this value is always set manually in
practice. A new parameter called beta stabilizer has been introduced to reduce
the sensitivity of the initial learning rate. But this method has only been
proposed for deep neural network (DNN) with sigmoid activation function. In
this paper we extended beta stabilizer to long short-term memory (LSTM) and
investigated the effects of beta stabilizer parameters on different models,
including LSTM and DNN with relu activation function. It is concluded that beta
stabilizer parameters can reduce the sensitivity of learning rate with almost
the same performance on DNN with relu activation function and LSTM. However, it
is shown that the effects of beta stabilizer on DNN with relu activation
function and LSTM are fewer than the effects on DNN with sigmoid activation
function.
