  We explore the redundancy of parameters in deep neural networks by replacing
the conventional linear projection in fully-connected layers with the circulant
projection. The circulant structure substantially reduces memory footprint and
enables the use of the Fast Fourier Transform to speed up the computation.
Considering a fully-connected neural network layer with d input nodes, and d
output nodes, this method improves the time complexity from O(d^2) to O(dlogd)
and space complexity from O(d^2) to O(d). The space savings are particularly
important for modern deep convolutional neural network architectures, where
fully-connected layers typically contain more than 90% of the network
parameters. We further show that the gradient computation and optimization of
the circulant projections can be performed very efficiently. Our experiments on
three standard datasets show that the proposed approach achieves this
significant gain in storage and efficiency with minimal increase in error rate
compared to neural networks with unstructured projections.
