  Parameter-efficient transfer learning (PETL) aims to adapt large pre-trained
models using limited parameters. While most PETL approaches update the added
parameters and freeze pre-trained weights during training, the minimal impact
of task-specific deep layers on cross-domain data poses a challenge as PETL
cannot modify them, resulting in redundant model structures. Structural pruning
effectively reduces model redundancy; however, common pruning methods often
lead to an excessive increase in stored parameters due to varying pruning
structures based on pruning rates and data. Recognizing the storage parameter
volume issue, we propose a Straightforward layer-wise pruning method, called
SLS, for pruning PETL-transferred models. By evaluating parameters from a
feature perspective of each layer and utilizing clustering metrics to assess
current parameters based on clustering phenomena in low-dimensional space
obtained through t-SNE, SLS facilitates informed pruning decisions. Our study
reveals that layer-wise pruning, with a focus on storing pruning indices,
addresses storage volume concerns. Notably, mainstream Layer-wise pruning
methods may not be suitable for assessing layer importance in PETL-transferred
models, where the majority of parameters are pre-trained and have limited
relevance to downstream datasets. Comparative analysis against state-of-the-art
PETL methods demonstrates that the pruned model achieved a notable balance
between model throughput and accuracy. Moreover, SLS effectively reduces
storage overhead arising from varying pruned structures while enhancing the
accuracy and speed of pruned models compared to conventional pruning methods.
