  Many self-supervised speech models, varying in their pre-training objective,
input modality, and pre-training data, have been proposed in the last few
years. Despite impressive successes on downstream tasks, we still have a
limited understanding of the properties encoded by the models and the
differences across models. In this work, we examine the intermediate
representations for a variety of recent models. Specifically, we measure
acoustic, phonetic, and word-level properties encoded in individual layers,
using a lightweight analysis tool based on canonical correlation analysis
(CCA). We find that these properties evolve across layers differently depending
on the model, and the variations relate to the choice of pre-training
objective. We further investigate the utility of our analyses for downstream
tasks by comparing the property trends with performance on speech recognition
and spoken language understanding tasks. We discover that CCA trends provide
reliable guidance to choose layers of interest for downstream tasks and that
single-layer performance often matches or improves upon using all layers,
suggesting implications for more efficient use of pre-trained models.
