  Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech
audio waveforms into latent representations. The largest version of wav2vec 2.0
contains 317 million parameters. Hence, the inference latency of wav2vec 2.0
will be a bottleneck in production, leading to high costs and a significant
environmental footprint. To improve wav2vec's applicability to a production
setting, we explore multiple model compression methods borrowed from the domain
of large language models. Using a teacher-student approach, we distilled the
knowledge from the original wav2vec 2.0 model into a student model, which is 2
times faster and 4.8 times smaller than the original model. This increase in
performance is accomplished with only a 7% degradation in word error rate
(WER). Our quantized model is 3.6 times smaller than the original model, with
only a 0.1% degradation in WER. To the best of our knowledge, this is the first
work that compresses wav2vec 2.0.
