  We propose two new criteria to understand the advantage of deepening neural
networks. It is important to know the expressivity of functions computable by
deep neural networks in order to understand the advantage of deepening neural
networks. Unless deep neural networks have enough expressivity, they cannot
have good performance even though learning is successful. In this situation,
the proposed criteria contribute to understanding the advantage of deepening
neural networks since they can evaluate the expressivity independently from the
efficiency of learning. The first criterion shows the approximation accuracy of
deep neural networks to the target function. This criterion has the background
that the goal of deep learning is approximating the target function by deep
neural networks. The second criterion shows the property of linear regions of
functions computable by deep neural networks. This criterion has the background
that deep neural networks whose activation functions are piecewise linear are
also piecewise linear. Furthermore, by the two criteria, we show that to
increase layers is more effective than to increase units at each layer on
improving the expressivity of deep neural networks.
