  Deep neural network (DNN) usually learns the target function from low to high
frequency, which is called frequency principle or spectral bias. This frequency
principle sheds light on a high-frequency curse of DNNs -- difficult to learn
high-frequency information. Inspired by the frequency principle, a series of
works are devoted to develop algorithms for overcoming the high-frequency
curse. A natural question arises: what is the upper limit of the decaying rate
w.r.t. frequency when one trains a DNN? In this work, our theory, confirmed by
numerical experiments, suggests that there is a critical decaying rate w.r.t.
frequency in DNN training. Below the upper limit of the decaying rate, the DNN
interpolates the training data by a function with a certain regularity.
However, above the upper limit, the DNN interpolates the training data by a
trivial function, i.e., a function is only non-zero at training data points.
Our results indicate a better way to overcome the high-frequency curse is to
design a proper pre-condition approach to shift high-frequency information to
low-frequency one, which coincides with several previous developed algorithms
for fast learning high-frequency information. More importantly, this work
rigorously proves that the high-frequency curse is an intrinsic difficulty of
DNNs.
