  We study model pruning methods applied to Transformer-based neural network
language models for automatic speech recognition. We explore three aspects of
the pruning frame work, namely criterion, method and scheduler, analyzing their
contribution in terms of accuracy and inference speed. To the best of our
knowledge, such in-depth analyses on large-scale recognition systems has not
been reported in the literature. In addition, we propose a variant of low-rank
approximation suitable for incrementally compressing models, and delivering
multiple models with varied target sizes. Among other results, we show that a)
data-driven pruning outperforms magnitude-driven in several scenarios; b)
incremental pruning achieves higher accuracy compared to one-shot pruning,
especially when targeting smaller sizes; and c) low-rank approximation presents
the best trade-off between size reduction and inference speed-up for moderate
compression.
