  Convergence of deep neural networks as the depth of the networks tends to
infinity is fundamental in building the mathematical foundation for deep
learning. In a previous study, we investigated this question for deep ReLU
networks with a fixed width. This does not cover the important convolutional
neural networks where the widths are increasing from layer to layer. For this
reason, we first study convergence of general ReLU networks with increasing
widths and then apply the results obtained to deep convolutional neural
networks. It turns out the convergence reduces to convergence of infinite
products of matrices with increasing sizes, which has not been considered in
the literature. We establish sufficient conditions for convergence of such
infinite products of matrices. Based on the conditions, we present sufficient
conditions for piecewise convergence of general deep ReLU networks with
increasing widths, and as well as pointwise convergence of deep ReLU
convolutional neural networks.
