  Self-supervised features are typically used in place of filter-bank features
in speaker verification models. However, these models were originally designed
to ingest filter-bank features as inputs, and thus, training them on top of
self-supervised features assumes that both feature types require the same
amount of learning for the task. In this work, we observe that pre-trained
self-supervised speech features inherently include information required for
downstream speaker verification task, and therefore, we can simplify the
downstream model without sacrificing performance. To this end, we revisit the
design of the downstream model for speaker verification using self-supervised
features. We show that we can simplify the model to use 97.51% fewer parameters
while achieving a 29.93% average improvement in performance on SUPERB.
Consequently, we show that the simplified downstream model is more data
efficient compared to baseline--it achieves better performance with only 60% of
the training data.
