  Parameter-Efficient Fine-Tuning (PEFT) is increasingly recognized as an
effective method in speech processing. However, the optimal approach and the
placement of PEFT methods remain inconclusive. Our study conducts extensive
experiments to compare different PEFT methods and their layer-wise placement
adapting Differentiable Architecture Search (DARTS). We also explore the use of
ensemble learning to leverage diverse PEFT strategies. The results reveal that
DARTS does not outperform the baseline approach, which involves inserting the
same PEFT method into all layers of a Self-Supervised Learning (SSL) model. In
contrast, an ensemble learning approach, particularly one employing majority
voting, demonstrates superior performance. Our statistical evidence indicates
that different PEFT methods learn in varied ways. This variation might explain
why the synergistic integration of various PEFT methods through ensemble
learning can harness their unique learning capabilities more effectively
compared to individual layer-wise optimization.
