  Transformer-based self-supervised models are trained as feature extractors
and have empowered many downstream speech tasks to achieve state-of-the-art
performance. However, both the training and inference process of these models
may encounter prohibitively high computational cost and large parameter budget.
Although Parameter Sharing Strategy (PSS) proposed in ALBERT paves the way for
parameter reduction, the computation required remains the same. Interestingly,
we found in experiments that distributions of feature embeddings from different
Transformer layers are similar when PSS is integrated: a property termed as
Layer Consistency (LC) in this paper. Given this similarity of feature
distributions, we assume that feature embeddings from different layers would
have similar representing power. In this work, Layer Consistency enables us to
adopt Transformer-based models in a more efficient manner: the number of
Conformer layers in each training iteration could be uniformly sampled and
Shallow Layer Inference (SLI) could be applied to reduce the number of layers
in inference stage. In experiments, our models are trained with LibriSpeech
dataset and then evaluated on both phone classification and Speech Recognition
tasks. We experimentally achieve 7.8X parameter reduction, 41.9% training
speedup and 37.7% inference speedup while maintaining comparable performance
with conventional BERT-like self-supervised methods.
