  In this paper, we study the compression of a target two-layer neural network
with N nodes into a compressed network with M<N nodes. More precisely, we
consider the setting in which the weights of the target network are i.i.d.
sub-Gaussian, and we minimize the population L_2 loss between the outputs of
the target and of the compressed network, under the assumption of Gaussian
inputs. By using tools from high-dimensional probability, we show that this
non-convex problem can be simplified when the target network is sufficiently
over-parameterized, and provide the error rate of this approximation as a
function of the input dimension and N. In this mean-field limit, the simplified
objective, as well as the optimal weights of the compressed network, does not
depend on the realization of the target network, but only on expected scaling
factors. Furthermore, for networks with ReLU activation, we conjecture that the
optimum of the simplified optimization problem is achieved by taking weights on
the Equiangular Tight Frame (ETF), while the scaling of the weights and the
orientation of the ETF depend on the parameters of the target network.
Numerical evidence is provided to support this conjecture.
