  Pre-trained language models (PLMs) have ignited a surge in demand for
effective fine-tuning techniques, particularly in low-resource domains and
languages. Active learning (AL), a set of algorithms designed to decrease
labeling costs by minimizing label complexity, has shown promise in confronting
the labeling bottleneck. In parallel, adapter modules designed for
parameter-efficient fine-tuning (PEFT) have demonstrated notable potential in
low-resource settings. However, the interplay between AL and adapter-based PEFT
remains unexplored. We present an empirical study of PEFT behavior with AL in
low-resource settings for text classification tasks. Our findings affirm the
superiority of PEFT over full-fine tuning (FFT) in low-resource settings and
demonstrate that this advantage persists in AL setups. We further examine the
properties of PEFT and FFT through the lens of forgetting dynamics and
instance-level representations, where we find that PEFT yields more stable
representations of early and middle layers compared to FFT. Our research
underscores the synergistic potential of AL and PEFT in low-resource settings,
paving the way for advancements in efficient and effective fine-tuning.
