# LoRA for Neuron

LoRA (Low-Rank Adaptation) implementation optimized for distributed training on AWS Trainium devices. This module provides efficient parameter-efficient fine-tuning with tensor parallelism and sequence parallelism support.

## PEFT Model Classes

### NeuronPeftModel[[optimum.neuron.peft.NeuronPeftModel]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.NeuronPeftModel</name><anchor>optimum.neuron.peft.NeuronPeftModel</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/peft_model.py#L82</source><parameters>[{"name": "model", "val": ": PreTrainedModel"}, {"name": "peft_config", "val": ": PeftConfig"}, {"name": "adapter_name", "val": ": str = 'default'"}, {"name": "autocast_adapter_dtype", "val": ": bool = True"}, {"name": "**kwargs", "val": ": Any"}]</parameters></docstring>


</div>

### NeuronPeftModelForCausalLM[[optimum.neuron.peft.NeuronPeftModelForCausalLM]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.NeuronPeftModelForCausalLM</name><anchor>optimum.neuron.peft.NeuronPeftModelForCausalLM</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/peft_model.py#L463</source><parameters>[{"name": "model", "val": ": PreTrainedModel"}, {"name": "peft_config", "val": ": PeftConfig"}, {"name": "adapter_name", "val": ": str = 'default'"}, {"name": "autocast_adapter_dtype", "val": ": bool = True"}, {"name": "**kwargs", "val": ": Any"}]</parameters></docstring>


</div>

## LoRA Layer Implementations

### Base LoRA Layer[[optimum.neuron.peft.tuners.lora.layer.NeuronLoraLayer]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.tuners.lora.layer.NeuronLoraLayer</name><anchor>optimum.neuron.peft.tuners.lora.layer.NeuronLoraLayer</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/tuners/lora/layer.py#L73</source><parameters>[{"name": "base_layer", "val": ": Module"}, {"name": "ephemeral_gpu_offload", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

### Parallel Linear LoRA[[optimum.neuron.peft.tuners.lora.layer.ParallelLinear]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.tuners.lora.layer.ParallelLinear</name><anchor>optimum.neuron.peft.tuners.lora.layer.ParallelLinear</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/tuners/lora/layer.py#L224</source><parameters>[{"name": "base_layer", "val": ""}, {"name": "adapter_name", "val": ": str"}, {"name": "r", "val": ": int = 0"}, {"name": "lora_alpha", "val": ": int = 1"}, {"name": "lora_dropout", "val": ": float = 0.0"}, {"name": "fan_in_fan_out", "val": ": bool = False"}, {"name": "is_target_conv_1d_layer", "val": ": bool = False"}, {"name": "init_lora_weights", "val": ": bool | str = True"}, {"name": "use_rslora", "val": ": bool = False"}, {"name": "use_dora", "val": ": bool = False"}, {"name": "lora_bias", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

### GQA QKV Column Parallel LoRA[[optimum.neuron.peft.tuners.lora.layer.GQAQKVColumnParallelLinear]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.tuners.lora.layer.GQAQKVColumnParallelLinear</name><anchor>optimum.neuron.peft.tuners.lora.layer.GQAQKVColumnParallelLinear</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/tuners/lora/layer.py#L315</source><parameters>[{"name": "base_layer", "val": ""}, {"name": "adapter_name", "val": ": str"}, {"name": "r", "val": ": int = 0"}, {"name": "lora_alpha", "val": ": int = 1"}, {"name": "lora_dropout", "val": ": float = 0.0"}, {"name": "fan_in_fan_out", "val": ": bool = False"}, {"name": "is_target_conv_1d_layer", "val": ": bool = False"}, {"name": "init_lora_weights", "val": ": bool | str = True"}, {"name": "use_rslora", "val": ": bool = False"}, {"name": "use_dora", "val": ": bool = False"}, {"name": "lora_bias", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

### Parallel Embedding LoRA[[optimum.neuron.peft.tuners.lora.layer.ParallelEmbedding]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.tuners.lora.layer.ParallelEmbedding</name><anchor>optimum.neuron.peft.tuners.lora.layer.ParallelEmbedding</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/tuners/lora/layer.py#L488</source><parameters>[{"name": "base_layer", "val": ": Module"}, {"name": "adapter_name", "val": ": str"}, {"name": "r", "val": ": int = 0"}, {"name": "lora_alpha", "val": ": int = 1"}, {"name": "lora_dropout", "val": ": float = 0.0"}, {"name": "fan_in_fan_out", "val": ": bool = False"}, {"name": "init_lora_weights", "val": ": bool | str = True"}, {"name": "use_rslora", "val": ": bool = False"}, {"name": "use_dora", "val": ": bool = False"}, {"name": "lora_bias", "val": ": bool = False"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## LoRA Model

### NeuronLoraModel[[optimum.neuron.peft.tuners.NeuronLoraModel]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.neuron.peft.tuners.NeuronLoraModel</name><anchor>optimum.neuron.peft.tuners.NeuronLoraModel</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/tuners/lora/model.py#L29</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ""}, {"name": "adapter_name", "val": ""}, {"name": "low_cpu_mem_usage", "val": ": bool = False"}]</parameters></docstring>


</div>

## Utility Functions

### get_peft_model[[optimum.neuron.peft.get_peft_model]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>optimum.neuron.peft.get_peft_model</name><anchor>optimum.neuron.peft.get_peft_model</anchor><source>https://github.com/huggingface/optimum-neuron/blob/main/optimum/neuron/peft/mapping_func.py#L43</source><parameters>[{"name": "model", "val": ": PreTrainedModel"}, {"name": "peft_config", "val": ": PeftConfig"}, {"name": "adapter_name", "val": ": str = 'default'"}, {"name": "mixed", "val": ": bool = False"}, {"name": "autocast_adapter_dtype", "val": ": bool = True"}, {"name": "revision", "val": ": str | None = None"}, {"name": "low_cpu_mem_usage", "val": ": bool = False"}]</parameters></docstring>


</div>

## Architecture Support

The Neuron LoRA implementation supports the following parallel layer types:

- **ColumnParallelLinear**: For layers that split weights along the output dimension
- **RowParallelLinear**: For layers that split weights along the input dimension  
- **ParallelEmbedding**: For embedding layers distributed across ranks
- **GQAQKVColumnParallelLinear**: For grouped query attention projections with challenging tensor parallel configurations

Each layer type has a corresponding LoRA implementation that maintains the parallelization strategy while adding low-rank adaptation capabilities.

## Key Features

- **Distributed Training**: Full support for tensor parallelism and sequence parallelism
- **Checkpoint Consolidation**: Automatic conversion between sharded and consolidated checkpoints
- **Weight Transformation**: Seamless integration with model weight transformation specs
- **Compatibility**: Works with all supported custom modeling architectures in Optimum Neuron