  Model training requires significantly more memory, compared with inference.
Parameter efficient fine-tuning (PEFT) methods provide a means of adapting
large models to downstream tasks using less memory. However, existing methods
such as adapters, prompt tuning or low-rank adaptation (LoRA) either introduce
latency overhead at inference time or achieve subpar downstream performance
compared with full fine-tuning. In this work we propose Random Subspace
Adaptation (ROSA), a method that outperforms previous PEFT methods by a
significant margin, while maintaining a zero latency overhead during inference
time. In contrast to previous methods, ROSA is able to adapt subspaces of
arbitrarily large dimension, better approximating full-finetuning. We
demonstrate both theoretically and experimentally that this makes ROSA strictly
more expressive than LoRA, without consuming additional memory during runtime.
As PEFT methods are especially useful in the natural language processing
domain, where models operate on scales that make full fine-tuning very
expensive, we evaluate ROSA in two common NLP scenarios: natural language
generation (NLG) and natural language understanding (NLU) with GPT-2 and
RoBERTa, respectively. We show that on almost every GLUE task ROSA outperforms
LoRA by a significant margin, while also outperforming LoRA on NLG tasks. Our
code is available at https://github.com/rosa-paper/rosa
