  We introduce Divnet, a flexible technique for learning networks with diverse
neurons. Divnet models neuronal diversity by placing a Determinantal Point
Process (DPP) over neurons in a given layer. It uses this DPP to select a
subset of diverse neurons and subsequently fuses the redundant neurons into the
selected ones. Compared with previous approaches, Divnet offers a more
principled, flexible technique for capturing neuronal diversity and thus
implicitly enforcing regularization. This enables effective auto-tuning of
network architecture and leads to smaller network sizes without hurting
performance. Moreover, through its focus on diversity and neuron fusing, Divnet
remains compatible with other procedures that seek to reduce memory footprints
of networks. We present experimental results to corroborate our claims: for
pruning neural networks, Divnet is seen to be notably superior to competing
approaches.
