  A primary function of back-propagation is to compute both the gradient of
hidden representations and parameters for optimization with gradient descent.
Training large models requires high computational costs due to their vast
parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to
train smaller auxiliary models to save computational space, they still present
computational overheads, especially in Fine-Tuning as a Service (FTaaS) for
numerous users. We introduce Collaborative Adaptation (ColA) with Gradient
Learning (GL), a parameter-free, model-agnostic fine-tuning approach that
decouples the computation of the gradient of hidden representations and
parameters. In comparison to PEFT methods, ColA facilitates more cost-effective
FTaaS by offloading the computation of the gradient to low-cost devices. We
also provide a theoretical analysis of ColA and experimentally demonstrate that
ColA can perform on par or better than existing PEFT methods on various
benchmarks.
