  This paper focuses on establishing $L^2$ approximation properties for deep
ReLU convolutional neural networks (CNNs) in two-dimensional space. The
analysis is based on a decomposition theorem for convolutional kernels with a
large spatial size and multi-channels. Given the decomposition result, the
property of the ReLU activation function, and a specific structure for
channels, a universal approximation theorem of deep ReLU CNNs with classic
structure is obtained by showing its connection with one-hidden-layer ReLU
neural networks (NNs). Furthermore, approximation properties are obtained for
one version of neural networks with ResNet, pre-act ResNet, and MgNet
architecture based on connections between these networks.
