  It has been observed in practical applications and in theoretical analysis
that over-parametrization helps to find good minima in neural network training.
Similarly, in this article we study widening and deepening neural networks by a
relaxation argument so that the enlarged networks are rich enough to run $r$
copies of parts of the original network in parallel, without necessarily
achieving zero training error as in over-parametrized scenarios. The partial
copies can be combined in $r^\theta$ possible ways for layer width $\theta$.
Therefore, the enlarged networks can potentially achieve the best training
error of $r^\theta$ random initializations, but it is not immediately clear if
this can be realized via gradient descent or similar training methods.
  The same construction can be applied to other optimization problems by
introducing a similar layered structure. We apply this idea to non-convex
compressed sensing, where we show that in some scenarios we can realize the
$r^\theta$ times increased chance to obtain a global optimum by solving a
convex optimization problem of dimension $r\theta$.
