  The machine learning community has witnessed impressive advancements since
large language models (LLMs) first appeared. Yet, their massive memory
consumption has become a significant roadblock to large-scale training. For
instance, a 7B model typically requires at least 60 GB of GPU memory with full
parameter training, which presents challenges for researchers without access to
high-resource environments. Parameter Efficient Fine-Tuning techniques such as
Low-Rank Adaptation (LoRA) have been proposed to alleviate this problem.
However, in most large-scale fine-tuning settings, their performance does not
reach the level of full parameter training because they confine the parameter
search to a low-rank subspace. Attempting to complement this deficiency, we
investigate the layerwise properties of LoRA on fine-tuning tasks and observe
an unexpected but consistent skewness of weight norms across different layers.
Utilizing this key observation, a surprisingly simple training strategy is
discovered, which outperforms both LoRA and full parameter training in a wide
range of settings with memory costs as low as LoRA. We name it Layerwise
Importance Sampled AdamW (LISA), a promising alternative for LoRA, which
applies the idea of importance sampling to different layers in LLMs and
randomly freezes most middle layers during optimization. Experimental results
show that with similar or less GPU memory consumption, LISA surpasses LoRA or
even full parameter tuning in downstream fine-tuning tasks, where LISA
consistently outperforms LoRA by over 10%-35% in terms of MT-Bench score while
achieving on-par or better performance in MMLU, AGIEval and WinoGrande. On
large models, specifically LLaMA-2-70B, LISA surpasses LoRA on MT-Bench, GSM8K,
and PubMedQA, demonstrating its effectiveness across different domains.
