  Due to complex interactions among various deep neural network (DNN)
optimization techniques, modern DNNs can have weights and activations that are
dense or sparse with diverse sparsity degrees. To offer a good trade-off
between accuracy and hardware performance, an ideal DNN accelerator should have
high flexibility to efficiently translate DNN sparsity into reductions in
energy and/or latency without incurring significant complexity overhead.
  This paper introduces hierarchical structured sparsity (HSS), with the key
insight that we can systematically represent diverse sparsity degrees by having
them hierarchically composed from multiple simple sparsity patterns. As a
result, HSS simplifies the underlying hardware since it only needs to support
simple sparsity patterns; this significantly reduces the sparsity acceleration
overhead, which improves efficiency. Motivated by such opportunities, we
propose a simultaneously efficient and flexible accelerator, named HighLight,
to accelerate DNNs that have diverse sparsity degrees (including dense). Due to
the flexibility of HSS, different HSS patterns can be introduced to DNNs to
meet different applications' accuracy requirements. Compared to existing works,
HighLight achieves a geomean of up to 6.4x better energy-delay product (EDP)
across workloads with diverse sparsity degrees, and always sits on the
EDP-accuracy Pareto frontier for representative DNNs
