  Fully-connected layers in deep neural networks (DNN) are often the throughput
and power bottleneck during training. This is due to their large size and low
data reuse. Pruning dense layers can significantly reduce the size of these
networks, but this approach can only be applied after training. In this work we
propose a novel fully-connected layer that reduces the memory requirements of
DNNs without sacrificing accuracy. We replace a dense matrix with products of
sparse matrices whose topologies we pick in advance. This allows us to: (1)
train significantly smaller networks without a loss in accuracy, and (2) store
the network weights without having to store connection indices. We therefore
achieve significant training speedups due to the smaller network size, and a
reduced amount of computation per epoch. We tested several sparse layer
topologies and found that Clos networks perform well due to their high path
diversity, shallowness, and high model accuracy. With the ClosNets, we are able
to reduce dense layer sizes by as much as an order of magnitude without hurting
model accuracy.
