  We develop a new theoretical framework to analyze the generalization error of
deep learning, and derive a new fast learning rate for two representative
algorithms: empirical risk minimization and Bayesian deep learning. The series
of theoretical analyses of deep learning has revealed its high expressive power
and universal approximation capability. Although these analyses are highly
nonparametric, existing generalization error analyses have been developed
mainly in a fixed dimensional parametric model. To compensate this gap, we
develop an infinite dimensional model that is based on an integral form as
performed in the analysis of the universal approximation capability. This
allows us to define a reproducing kernel Hilbert space corresponding to each
layer. Our point of view is to deal with the ordinary finite dimensional deep
neural network as a finite approximation of the infinite dimensional one. The
approximation error is evaluated by the degree of freedom of the reproducing
kernel Hilbert space in each layer. To estimate a good finite dimensional
model, we consider both of empirical risk minimization and Bayesian deep
learning. We derive its generalization error bound and it is shown that there
appears bias-variance trade-off in terms of the number of parameters of the
finite dimensional approximation. We show that the optimal width of the
internal layers can be determined through the degree of freedom and the
convergence rate can be faster than $O(1/\sqrt{n})$ rate which has been shown
in the existing studies.
