  Although deep neural networks (DNN) are able to scale with direct advances in
computational power (e.g., memory and processing speed), they are not well
suited to exploit the recent trends for parallel architectures. In particular,
gradient descent is a sequential process and the resulting serial dependencies
mean that DNN training cannot be parallelized effectively. Here, we show that a
DNN may be replicated over a massive parallel architecture and used to provide
a cumulative sampling of local solution space which results in rapid and robust
learning. We introduce a complimentary convolutional bootstrapping approach
that enhances performance of the parallel architecture further. Our
parallelized convolutional bootstrapping DNN out-performs an identical
fully-trained traditional DNN after only a single iteration of training.
