  Large pre-trained speech models are widely used as the de-facto paradigm,
especially in scenarios when there is a limited amount of labeled data
available. However, finetuning all parameters from the self-supervised learned
model can be computationally expensive, and becomes infeasiable as the size of
the model and the number of downstream tasks scales. In this paper, we propose
a novel approach called Two Parallel Adapter (TPA) that is inserted into the
conformer-based model pre-trained model instead. TPA is based on systematic
studies of the residual adapter, a popular approach for finetuning a subset of
parameters. We evaluate TPA on various public benchmarks and experiment results
demonstrates its superior performance, which is close to the full finetuning on
different datasets and speech tasks. These results show that TPA is an
effective and efficient approach for serving large pre-trained speech models.
Ablation studies show that TPA can also be pruned, especially for lower blocks.
