  Quantization, knowledge distillation, and magnitude pruning are among the
most popular methods for neural network compression in NLP. Independently,
these methods reduce model size and can accelerate inference, but their
relative benefit and combinatorial interactions have not been rigorously
studied. For each of the eight possible subsets of these techniques, we compare
accuracy vs. model size tradeoffs across six BERT architecture sizes and eight
GLUE tasks. We find that quantization and distillation consistently provide
greater benefit than pruning. Surprisingly, except for the pair of pruning and
quantization, using multiple methods together rarely yields diminishing
returns. Instead, we observe complementary and super-multiplicative reductions
to model size. Our work quantitatively demonstrates that combining compression
methods can synergistically reduce model size, and that practitioners should
prioritize (1) quantization, (2) knowledge distillation, and (3) pruning to
maximize accuracy vs. model size tradeoffs.
