  Neuromorphic hardware tends to pose limits on the connectivity of deep
networks that one can run on them. But also generic hardware and software
implementations of deep learning run more efficiently for sparse networks.
Several methods exist for pruning connections of a neural network after it was
trained without connectivity constraints. We present an algorithm, DEEP R, that
enables us to train directly a sparsely connected neural network. DEEP R
automatically rewires the network during supervised training so that
connections are there where they are most needed for the task, while its total
number is all the time strictly bounded. We demonstrate that DEEP R can be used
to train very sparse feedforward and recurrent neural networks on standard
benchmark tasks with just a minor loss in performance. DEEP R is based on a
rigorous theoretical foundation that views rewiring as stochastic sampling of
network configurations from a posterior.
