  While transferring a pretrained language model, common approaches
conventionally attach their task-specific classifiers to the top layer and
adapt all the pretrained layers. We investigate whether one could make a
task-specific selection on which subset of the layers to adapt and where to
place the classifier. The goal is to reduce the computation cost of transfer
learning methods (e.g. fine-tuning or adapter-tuning) without sacrificing its
performance.
  We propose to select layers based on the variability of their hidden states
given a task-specific corpus. We say a layer is already "well-specialized" in a
task if the within-class variability of its hidden states is low relative to
the between-class variability. Our variability metric is cheap to compute and
doesn't need any training or hyperparameter tuning. It is robust to data
imbalance and data scarcity. Extensive experiments on the GLUE benchmark
demonstrate that selecting layers based on our metric can yield significantly
stronger performance than using the same number of top layers and often match
the performance of fine-tuning or adapter-tuning the entire language model.
