  Recently deep neural networks have received considerable attention due to
their ability to extract and represent high-level abstractions in data sets.
Deep neural networks such as fully-connected and convolutional neural networks
have shown excellent performance on a wide range of recognition and
classification tasks. However, their hardware implementations currently suffer
from large silicon area and high power consumption due to the their high degree
of complexity. The power/energy consumption of neural networks is dominated by
memory accesses, the majority of which occur in fully-connected networks. In
fact, they contain most of the deep neural network parameters. In this paper,
we propose sparsely-connected networks, by showing that the number of
connections in fully-connected networks can be reduced by up to 90% while
improving the accuracy performance on three popular datasets (MNIST, CIFAR10
and SVHN). We then propose an efficient hardware architecture based on
linear-feedback shift registers to reduce the memory requirements of the
proposed sparsely-connected networks. The proposed architecture can save up to
90% of memory compared to the conventional implementations of fully-connected
neural networks. Moreover, implementation results show up to 84% reduction in
the energy consumption of a single neuron of the proposed sparsely-connected
networks compared to a single neuron of fully-connected neural networks.
