  As foundation models continue to exponentially scale in size, efficient
methods of adaptation become increasingly critical. Parameter-efficient
fine-tuning (PEFT), a recent class of techniques that require only modifying a
small percentage of the model parameters, is currently the most popular method
for adapting large language models (LLMs). Several PEFT techniques have
recently been proposed with varying tradeoffs. We provide a comprehensive and
uniform benchmark of various PEFT techniques across a representative LLM, the
FLAN-T5 model, and evaluate model performance across different data scales of
classification and generation datasets. Based on this, we provide a framework
for choosing the optimal fine-tuning techniques given the task type and data
availability. Contrary to popular belief, we also empirically prove that PEFT
techniques converge slower than full tuning in low data scenarios, and posit
the amount of data required for PEFT methods to both perform well and converge
efficiently. Lastly, we further optimize these PEFT techniques by selectively
choosing which parts of the model to train, and find that these techniques can
be applied with significantly fewer parameters while maintaining and even
improving performance.
