  Modern neural networks often have great expressive power and can be trained
to overfit the training data, while still achieving a good test performance.
This phenomenon is referred to as "benign overfitting". Recently, there emerges
a line of works studying "benign overfitting" from the theoretical perspective.
However, they are limited to linear models or kernel/random feature models, and
there is still a lack of theoretical understanding about when and how benign
overfitting occurs in neural networks. In this paper, we study the benign
overfitting phenomenon in training a two-layer convolutional neural network
(CNN). We show that when the signal-to-noise ratio satisfies a certain
condition, a two-layer CNN trained by gradient descent can achieve arbitrarily
small training and test loss. On the other hand, when this condition does not
hold, overfitting becomes harmful and the obtained CNN can only achieve a
constant level test loss. These together demonstrate a sharp phase transition
between benign overfitting and harmful overfitting, driven by the
signal-to-noise ratio. To the best of our knowledge, this is the first work
that precisely characterizes the conditions under which benign overfitting can
occur in training convolutional neural networks.
