  Convolutional networks (ConvNets) have become a popular approach to computer
vision. It is important to accelerate ConvNet training, which is
computationally costly. We propose a novel parallel algorithm based on
decomposition into a set of tasks, most of which are convolutions or FFTs.
Applying Brent's theorem to the task dependency graph implies that linear
speedup with the number of processors is attainable within the PRAM model of
parallel computation, for wide network architectures. To attain such
performance on real shared-memory machines, our algorithm computes convolutions
converging on the same node of the network with temporal locality to reduce
cache misses, and sums the convergent convolution outputs via an almost
wait-free concurrent method to reduce time spent in critical sections. We
implement the algorithm with a publicly available software package called ZNN.
Benchmarking with multi-core CPUs shows that ZNN can attain speedup roughly
equal to the number of physical cores. We also show that ZNN can attain over
90x speedup on a many-core CPU (Xeon Phi Knights Corner). These speedups are
achieved for network architectures with widths that are in common use. The task
parallelism of the ZNN algorithm is suited to CPUs, while the SIMD parallelism
of previous algorithms is compatible with GPUs. Through examples, we show that
ZNN can be either faster or slower than certain GPU implementations depending
on specifics of the network architecture, kernel sizes, and density and size of
the output patch. ZNN may be less costly to develop and maintain, due to the
relative ease of general-purpose CPU programming.
