  Training a neural network (NN) typically relies on some type of
curve-following method, such as gradient descent (GD) (and stochastic gradient
descent (SGD)), ADADELTA, ADAM or limited memory algorithms. Convergence for
these algorithms usually relies on having access to a large quantity of
observations in order to achieve a high level of accuracy and, with certain
classes of functions, these algorithms could take multiple epochs of data
points to catch on. Herein, a different technique with the potential of
achieving dramatically better speeds of convergence, especially for shallow
networks, is explored: it does not curve-follow but rather relies on
'decoupling' hidden layers and on updating their weighted connections through
bootstrapping, resampling and linear regression. By utilizing resampled
observations, the convergence of this process is empirically shown to be
remarkably fast and to require a lower amount of data points: in particular,
our experiments show that one needs a fraction of the observations that are
required with traditional neural network training methods to approximate
various classes of functions.
