  Natural Language Processing (NLP) has recently achieved success by using huge
pre-trained Transformer networks. However, these models often contain hundreds
of millions or even billions of parameters, bringing challenges to online
deployment due to latency constraints. Recently, hardware manufacturers have
introduced dedicated hardware for NxM sparsity to provide the flexibility of
unstructured pruning with the runtime efficiency of structured approaches. NxM
sparsity permits arbitrarily selecting M parameters to retain from a contiguous
group of N in the dense representation. However, due to the extremely high
complexity of pre-trained models, the standard sparse fine-tuning techniques
often fail to generalize well on downstream tasks, which have limited data
resources. To address such an issue in a principled manner, we introduce a new
learning framework, called NxMTransformer, to induce NxM semi-structured
sparsity on pretrained language models for natural language understanding to
obtain better performance. In particular, we propose to formulate the NxM
sparsity as a constrained optimization problem and use Alternating Direction
Method of Multipliers (ADMM) to optimize the downstream tasks while taking the
underlying hardware constraints into consideration. ADMM decomposes the NxM
sparsification problem into two sub-problems that can be solved sequentially,
generating sparsified Transformer networks that achieve high accuracy while
being able to effectively execute on newly released hardware. We apply our
approach to a wide range of NLP tasks, and our proposed method is able to
achieve 1.7 points higher accuracy in GLUE score than current practices.
Moreover, we perform detailed analysis on our approach and shed light on how
ADMM affects fine-tuning accuracy for downstream tasks. Finally, we illustrate
how NxMTransformer achieves performance improvement with knowledge
distillation.
