  In theoretical analysis of deep learning, discovering which features of deep
learning lead to good performance is an important task. In this paper, using
the framework for analyzing the generalization error developed in Suzuki
(2018), we derive a fast learning rate for deep neural networks with more
general activation functions. In Suzuki (2018), assuming the scale invariance
of activation functions, the tight generalization error bound of deep learning
was derived. They mention that the scale invariance of the activation function
is essential to derive tight error bounds. Whereas the rectified linear unit
(ReLU; Nair and Hinton, 2010) satisfies the scale invariance, the other famous
activation functions including the sigmoid and the hyperbolic tangent
functions, and the exponential linear unit (ELU; Clevert et al., 2016) does not
satisfy this condition. The existing analysis indicates a possibility that a
deep learning with the non scale invariant activations may have a slower
convergence rate of $O(1/\sqrt{n})$ when one with the scale invariant
activations can reach a rate faster than $O(1/\sqrt{n})$. In this paper,
without the scale invariance of activation functions, we derive the tight
generalization error bound which is essentially the same as that of Suzuki
(2018). From this result, at least in the framework of Suzuki (2018), it is
shown that the scale invariance of the activation functions is not essential to
get the fast rate of convergence. Simultaneously, it is also shown that the
theoretical framework proposed by Suzuki (2018) can be widely applied for
analysis of deep learning with general activation functions.
