  Weight pruning has been widely acknowledged as a straightforward and
effective method to eliminate redundancy in Deep Neural Networks (DNN), thereby
achieving acceleration on various platforms. However, most of the pruning
techniques are essentially trade-offs between model accuracy and regularity
which lead to impaired inference accuracy and limited on-device acceleration
performance. To solve the problem, we introduce a new sparsity dimension,
namely pattern-based sparsity that comprises pattern and connectivity sparsity,
and becoming both highly accurate and hardware friendly. With carefully
designed patterns, the proposed pruning unprecedentedly and consistently
achieves accuracy enhancement and better feature extraction ability on
different DNN structures and datasets, and our pattern-aware pruning framework
also achieves pattern library extraction, pattern selection, pattern and
connectivity pruning and weight training simultaneously. Our approach on the
new pattern-based sparsity naturally fits into compiler optimization for highly
efficient DNN execution on mobile platforms. To the best of our knowledge, it
is the first time that mobile devices achieve real-time inference for the
large-scale DNN models thanks to the unique spatial property of pattern-based
sparsity and the help of the code generation capability of compilers.
