  Gate functions in recurrent models, such as an LSTM and GRU, play a central
role in learning various time scales in modeling time series data by using a
bounded activation function. However, it is difficult to train gates to capture
extremely long time scales due to gradient vanishing of the bounded function
for large inputs, which is known as the saturation problem. We closely analyze
the relation between saturation of the gate function and efficiency of the
training. We prove that the gradient vanishing of the gate function can be
mitigated by accelerating the convergence of the saturating function, i.e.,
making the output of the function converge to 0 or 1 faster. Based on the
analysis results, we propose a gate function called fast gate that has a doubly
exponential convergence rate with respect to inputs by simple function
composition. We empirically show that our method outperforms previous methods
in accuracy and computational efficiency on benchmark tasks involving extremely
long time scales.
