  This paper presents a new method for pre-training neural networks that can
decrease the total training time for a neural network while maintaining the
final performance, which motivates its use on deep neural networks. By
partitioning the training task in multiple training subtasks with sub-models,
which can be performed independently and in parallel, it is shown that the size
of the sub-models reduces almost quadratically with the number of subtasks
created, quickly scaling down the sub-models used for the pre-training. The
sub-models are then merged to provide a pre-trained initial set of weights for
the original model. The proposed method is independent of the other aspects of
the training, such as architecture of the neural network, training method, and
objective, making it compatible with a wide range of existing approaches. The
speedup without loss of performance is validated experimentally on MNIST and on
CIFAR10 data sets, also showing that even performing the subtasks sequentially
can decrease the training time. Moreover, we show that larger models may
present higher speedups and conjecture about the benefits of the method in
distributed learning systems.
