  Self-supervised speech representation learning has shown promising results in
various speech processing tasks. However, the pre-trained models, e.g., HuBERT,
are storage-intensive Transformers, limiting their scope of applications under
low-resource settings. To this end, we propose LightHuBERT, a once-for-all
Transformer compression framework, to find the desired architectures
automatically by pruning structured parameters. More precisely, we create a
Transformer-based supernet that is nested with thousands of weight-sharing
subnets and design a two-stage distillation strategy to leverage the
contextualized latent representations from HuBERT. Experiments on automatic
speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT
enables over $10^9$ architectures concerning the embedding dimension, attention
dimension, head number, feed-forward network ratio, and network depth.
LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with
the HuBERT size, achieves comparable performance to the teacher model in most
tasks with a reduction of 29% parameters, and obtains a $3.5\times$ compression
ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword
spotting, and intent classification, with a slight accuracy loss. The code and
pre-trained models are available at
https://github.com/mechanicalsea/lighthubert.
