  Self-supervised learning has shown great success in Speech Recognition.
However, it has been observed that finetuning all layers of the learned model
leads to lower performance compared to resetting top layers. This phenomenon is
attributed to the ''autoencoder'' behavior: top layers contain information
closer to the input and are less suitable for tasks that require linguistic
information, such as Speech Recognition.To better our understanding of this
behavior, we propose to study the evolution of high-level information within
the model during pretraining. We focus on the HuBERT model, which exhibits a
less pronounced ''autoencoder'' behavior. By experimentally exploring various
factors that may have an impact, we aim to improve the training procedure and
enhance the top layers of HuBERT for high-level tasks.Furthermore, our
experiments demonstrate that these improvements in the training procedure
result in faster convergence and competitive performance on downstream tasks.
