  General Matrix Multiply (GEMM) hardware, employing large arrays of
multiply-accumulate (MAC) units, perform bulk of the computation in deep
learning (DL). Recent trends have established 8-bit integer (INT8) as the most
widely used precision for DL inference. This paper proposes a novel MAC design
capable of dynamically exploiting bit sparsity (i.e., number of `0' bits within
a binary value) in input data to achieve significant improvements on area,
power and energy. The proposed architecture, called OzMAC (Omit-zero-MAC),
skips over zeros within a binary input value and performs simple
shift-and-add-based compute in place of expensive multipliers. We implement
OzMAC in SystemVerilog and present post-synthesis performance-power-area (PPA)
results using commercial TSMC N5 (5nm) process node. Using eight pretrained
INT8 deep neural networks (DNNs) as benchmarks, we demonstrate the existence of
high bit sparsity in real DNN workloads and show that 8-bit OzMAC improves all
three metrics of area, power, and energy significantly by 21%, 70%, and 28%,
respectively. Similar improvements are achieved when scaling data precisions
(4, 8, 16 bits) and clock frequencies (0.5 GHz, 1 GHz, 1.5 GHz). For the 8-bit
OzMAC, scaling its frequency to normalize the throughput relative to
conventional MAC, it still achieves 30% improvement on both power and energy.
