  LoRA is a technique that reduces the number of trainable parameters in a
neural network by introducing low-rank adapters to linear layers. This
technique is used both for fine-tuning and full training of large language
models. This paper presents the RunLoRA framework for efficient implementations
of LoRA that significantly improves the speed of neural network training and
fine-tuning using low-rank adapters. The proposed implementation optimizes the
computation of LoRA operations based on dimensions of corresponding linear
layer, layer input dimensions and lora rank by choosing best forward and
backward computation graph based on FLOPs and time estimations, resulting in
faster training without sacrificing accuracy. The experimental results show up
to 28\% speedup on language modeling networks.
