  As language models increase in size by the day, methods for efficient
inference are critical to leveraging their capabilities for various
applications. Prior work has investigated techniques like model pruning,
knowledge distillation, and data multiplexing to increase model throughput
without sacrificing accuracy. In this paper, we combine two such methods --
structured pruning and data multiplexing -- to compound the speedup gains
obtained by either method. Our approach, PruMUX, obtains up to 7.5-29.5X
throughput improvement over BERT-base model with accuracy threshold from 80% to
74%. We further study various combinations of parameters (such as sparsity and
multiplexing factor) in the two techniques to provide a comprehensive analysis
of the tradeoff between accuracy and throughput in the resulting models. We
then propose Auto-PruMUX, a meta-level model that can predict the
high-performance parameters for pruning and multiplexing given a desired
accuracy loss budget, providing a practical method to leverage the combination
effectively.
