  Recent parameter-efficient finetuning (PEFT) techniques aim to improve over
the considerable cost of fully finetuning large pretrained language models
(PLM). As different PEFT techniques proliferate, it is becoming difficult to
compare them, in particular in terms of (i) the structure and functionality
they add to the PLM, (ii) the different types and degrees of efficiency
improvements achieved, (iii) performance at different downstream tasks, and
(iv) how differences in structure and functionality relate to efficiency and
task performance. To facilitate such comparisons, this paper presents a
reference architecture which standardises aspects shared by different PEFT
techniques, while isolating differences to specific locations and interactions
with the standard components. Through this process of standardising and
isolating differences, a modular view of PEFT techniques emerges, supporting
not only direct comparison of different techniques and their efficiency and
task performance, but also systematic exploration of reusability and
composability of the different types of finetuned modules. We demonstrate how
the reference architecture can be applied to understand properties and relative
advantages of PEFT techniques, hence to inform selection of techniques for
specific tasks, and design choices for new PEFT techniques.
