  Sparse deep neural networks(DNNs) are efficient in both memory and compute
when compared to dense DNNs. But due to irregularity in computation of sparse
DNNs, their efficiencies are much lower than that of dense DNNs on regular
parallel hardware such as TPU. This inefficiency leads to poor/no performance
benefits for sparse DNNs. Performance issue for sparse DNNs can be alleviated
by bringing structure to the sparsity and leveraging it for improving runtime
efficiency. But such structural constraints often lead to suboptimal
accuracies. In this work, we jointly address both accuracy and performance of
sparse DNNs using our proposed class of sparse neural networks called HBsNN
(Hierarchical Block sparse Neural Networks). For a given sparsity, HBsNN models
achieve better runtime performance than unstructured sparse models and better
accuracy than highly structured sparse models.
