  We propose to execute deep neural networks (DNNs) with dynamic and sparse
graph (DSG) structure for compressive memory and accelerative execution during
both training and inference. The great success of DNNs motivates the pursuing
of lightweight models for the deployment onto embedded devices. However, most
of the previous studies optimize for inference while neglect training or even
complicate it. Training is far more intractable, since (i) the neurons dominate
the memory cost rather than the weights in inference; (ii) the dynamic
activation makes previous sparse acceleration via one-off optimization on fixed
weight invalid; (iii) batch normalization (BN) is critical for maintaining
accuracy while its activation reorganization damages the sparsity. To address
these issues, DSG activates only a small amount of neurons with high
selectivity at each iteration via a dimension-reduction search (DRS) and
obtains the BN compatibility via a double-mask selection (DMS). Experiments
show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x)
with little accuracy loss on various benchmarks.
