  The energy consumed by running large deep neural networks (DNNs) on hardware
accelerators is dominated by the need for lots of fast memory to store both
states and weights. This large required memory is currently only economically
viable through DRAM. Although DRAM is high-throughput and low-cost memory
(costing 20X less than SRAM), its long random access latency is bad for the
unpredictable access patterns in spiking neural networks (SNNs). In addition,
accessing data from DRAM costs orders of magnitude more energy than doing
arithmetic with that data. SNNs are energy-efficient if local memory is
available and few spikes are generated. This paper reports on our developments
over the last 5 years of convolutional and recurrent deep neural network
hardware accelerators that exploit either spatial or temporal sparsity similar
to SNNs but achieve SOA throughput, power efficiency and latency even with the
use of DRAM for the required storage of the weights and states of large DNNs.
