# Tuners

## Tuners

Each tuner (or PEFT method) has a configuration and model.

### LoRA

For finetuning a model with LoRA.

#### class peft.LoraConfig

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/config.py#L24)

( peft\_type: typing.Union\[str, peft.utils.peft\_types.PeftType] = Noneauto\_mapping: typing.Optional\[dict] = Nonebase\_model\_name\_or\_path: str = Nonerevision: str = Nonetask\_type: typing.Union\[str, peft.utils.peft\_types.TaskType] = Noneinference\_mode: bool = Falser: int = 8target\_modules: typing.Union\[str, typing.List\[str], NoneType] = Nonelora\_alpha: int = 8lora\_dropout: float = 0.0fan\_in\_fan\_out: bool = Falsebias: str = 'none'modules\_to\_save: typing.Optional\[typing.List\[str]] = Noneinit\_lora\_weights: bool = Truelayers\_to\_transform: typing.Union\[int, typing.List\[int], NoneType] = Nonelayers\_pattern: typing.Union\[str, typing.List\[str], NoneType] = Nonerank\_pattern: typing.Optional\[dict] = \<factory>alpha\_pattern: typing.Optional\[dict] = \<factory> )

Parameters

* **r** (`int`) — Lora attention dimension.
* **target\_modules** (`Union[List[str],str]`) — The names of the modules to apply Lora to.
* **lora\_alpha** (`int`) — The alpha parameter for Lora scaling.
* **lora\_dropout** (`float`) — The dropout probability for Lora layers.
* **fan\_in\_fan\_out** (`bool`) — Set this to True if the layer to replace stores weight like (fan\_in, fan\_out). For example, gpt-2 uses `Conv1D` which stores weights like (fan\_in, fan\_out) and hence this should be set to `True`.
* **bias** (`str`) — Bias type for Lora. Can be ‘none’, ‘all’ or ‘lora\_only’. If ‘all’ or ‘lora\_only’, the corresponding biases will be updated during training. Be aware that this means that, even when disabling the adapters, the model will not produce the same output as the base model would have without adaptation.
* **modules\_to\_save** (`List[str]`) —List of modules apart from LoRA layers to be set as trainable and saved in the final checkpoint.
* **layers\_to\_transform** (`Union[List[int],int]`) — The layer indexes to transform, if this argument is specified, it will apply the LoRA transformations on the layer indexes that are specified in this list. If a single integer is passed, it will apply the LoRA transformations on the layer at this index.
* **layers\_pattern** (`str`) — The layer pattern name, used only if `layers_to_transform` is different from `None` and if the layer pattern is not in the common layers pattern.
* **rank\_pattern** (`dict`) — The mapping from layer names or regexp expression to ranks which are different from the default rank specified by `r`.
* **alpha\_pattern** (`dict`) — The mapping from layer names or regexp expression to alphas which are different from the default alpha specified by `lora_alpha`.

This is the configuration class to store the configuration of a [LoraModel](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.LoraModel).

#### class peft.LoraModel

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py#L53)

( modelconfigadapter\_name ) → `torch.nn.Module`

Parameters

* **model** ([PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel)) — The model to be adapted.
* **config** ([LoraConfig](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.LoraConfig)) — The configuration of the Lora model.
* **adapter\_name** (`str`) — The name of the adapter, defaults to `"default"`.

Returns

`torch.nn.Module`

The Lora model.

Creates Low Rank Adapter (Lora) model from a pretrained transformers model.

Example:

Copied

```
>>> from transformers import AutoModelForSeq2SeqLM
>>> from peft import LoraModel, LoraConfig

>>> config = LoraConfig(
...     task_type="SEQ_2_SEQ_LM",
...     r=8,
...     lora_alpha=32,
...     target_modules=["q", "v"],
...     lora_dropout=0.01,
... )

>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> lora_model = LoraModel(model, config, "default")
```

Copied

```
>>> import transformers
>>> from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_int8_training

>>> target_modules = ["q_proj", "k_proj", "v_proj", "out_proj", "fc_in", "fc_out", "wte"]
>>> config = LoraConfig(
...     r=4, lora_alpha=16, target_modules=target_modules, lora_dropout=0.1, bias="none", task_type="CAUSAL_LM"
... )

>>> model = transformers.GPTJForCausalLM.from_pretrained(
...     "kakaobrain/kogpt",
...     revision="KoGPT6B-ryan1.5b-float16",  # or float32 version: revision=KoGPT6B-ryan1.5b
...     pad_token_id=tokenizer.eos_token_id,
...     use_cache=False,
...     device_map={"": rank},
...     torch_dtype=torch.float16,
...     load_in_8bit=True,
... )
>>> model = prepare_model_for_int8_training(model)
>>> lora_model = get_peft_model(model, config)
```

**Attributes**:

* **model** ([PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel)) — The model to be adapted.
* **peft\_config** ([LoraConfig](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.LoraConfig)): The configuration of the Lora model.

**add\_weighted\_adapter**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py#L442)

( adaptersweightsadapter\_namecombination\_type = 'svd'svd\_rank = Nonesvd\_clamp = Nonesvd\_full\_matrices = Truesvd\_driver = None )

Parameters

* **adapters** (`list`) — List of adapter names to be merged.
* **weights** (`list`) — List of weights for each adapter.
* **adapter\_name** (`str`) — Name of the new adapter.
* **combination\_type** (`str`) — Type of merging. Can be one of \[`svd`, `linear`, `cat`]. When using the `cat` combination\_type you should be aware that rank of the resulting adapter will be equal to the sum of all adapters ranks. So it’s possible that the mixed adapter may become too big and result in OOM errors.
* **svd\_rank** (`int`, *optional*) — Rank of output adapter for svd. If None provided, will use max rank of merging adapters.
* **svd\_clamp** (`float`, *optional*) — A quantile threshold for clamping SVD decomposition output. If None is provided, do not perform clamping. Defaults to None.
* **svd\_full\_matrices** (`bool`, *optional*) — Controls whether to compute the full or reduced SVD, and consequently, the shape of the returned tensors U and Vh. Defaults to True.
* **svd\_driver** (`str`, *optional*) — Name of the cuSOLVER method to be used. This keyword argument only works when merging on CUDA. Can be one of \[None, `gesvd`, `gesvdj`, `gesvda`]. For more info please refer to `torch.linalg.svd` documentation. Defaults to None.

This method adds a new adapter by merging the given adapters with the given weights.

When using the `cat` combination\_type you should be aware that rank of the resulting adapter will be equal to the sum of all adapters ranks. So it’s possible that the mixed adapter may become too big and result in OOM errors.

**delete\_adapter**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py#L652)

( adapter\_name: str )

Parameters

* **adapter\_name** (str) — Name of the adapter to be deleted.

Deletes an existing adapter.

**merge\_and\_unload**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py#L688)

( progressbar: bool = Falsesafe\_merge: bool = False )

Parameters

* **progressbar** (`bool`) — whether to show a progressbar indicating the unload and merge process
* **safe\_merge** (`bool`) — whether to activate the safe merging check to check if there is any potential Nan in the adapter weights

This method merges the LoRa layers into the base model. This is needed if someone wants to use the base model as a standalone model.

Example:

Copied

```
>>> from transformers import AutoModelForCausalLM
>>> from peft import PeftModel

>>> base_model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b")
>>> peft_model_id = "smangrul/falcon-40B-int4-peft-lora-sfttrainer-sample"
>>> model = PeftModel.from_pretrained(base_model, peft_model_id)
>>> merged_model = model.merge_and_unload()
```

**unload**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/model.py#L714)

( )

Gets back the base model by removing all the lora modules without merging. This gives back the original base model.

#### class peft.tuners.lora.LoraLayer

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py#L28)

( in\_features: intout\_features: int\*\*kwargs )

#### class peft.tuners.lora.Linear

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py#L189)

( adapter\_name: strin\_features: intout\_features: intr: int = 0lora\_alpha: int = 1lora\_dropout: float = 0.0fan\_in\_fan\_out: bool = Falseis\_target\_conv\_1d\_layer: bool = False\*\*kwargs )

**get\_delta\_weight**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py#L261)

( adapter )

Parameters

* **adapter** (str) — The name of the adapter for which the delta weight should be computed.

Compute the delta weight for the given adapter.

**merge**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora/layer.py#L219)

( safe\_merge: bool = False )

Parameters

* **safe\_merge** (`bool`, *optional*) — If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`.

Merge the active adapter weights into the base weights

### P-tuning

#### class peft.PromptEncoderConfig

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/p_tuning/config.py#L30)

( peft\_type: typing.Union\[str, peft.utils.peft\_types.PeftType] = Noneauto\_mapping: typing.Optional\[dict] = Nonebase\_model\_name\_or\_path: str = Nonerevision: str = Nonetask\_type: typing.Union\[str, peft.utils.peft\_types.TaskType] = Noneinference\_mode: bool = Falsenum\_virtual\_tokens: int = Nonetoken\_dim: int = Nonenum\_transformer\_submodules: typing.Optional\[int] = Nonenum\_attention\_heads: typing.Optional\[int] = Nonenum\_layers: typing.Optional\[int] = Noneencoder\_reparameterization\_type: typing.Union\[str, peft.tuners.p\_tuning.config.PromptEncoderReparameterizationType] = \<PromptEncoderReparameterizationType.MLP: 'MLP'>encoder\_hidden\_size: int = Noneencoder\_num\_layers: int = 2encoder\_dropout: float = 0.0 )

Parameters

* **encoder\_reparameterization\_type** (Union\[`PromptEncoderReparameterizationType`, `str`]) — The type of reparameterization to use.
* **encoder\_hidden\_size** (`int`) — The hidden size of the prompt encoder.
* **encoder\_num\_layers** (`int`) — The number of layers of the prompt encoder.
* **encoder\_dropout** (`float`) — The dropout probability of the prompt encoder.

This is the configuration class to store the configuration of a [PromptEncoder](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.PromptEncoder).

#### class peft.PromptEncoder

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/p_tuning/model.py#L25)

( config )

Parameters

* **config** ([PromptEncoderConfig](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.PromptEncoderConfig)) — The configuration of the prompt encoder.

The prompt encoder network that is used to generate the virtual token embeddings for p-tuning.

Example:

Copied

```
>>> from peft import PromptEncoder, PromptEncoderConfig

>>> config = PromptEncoderConfig(
...     peft_type="P_TUNING",
...     task_type="SEQ_2_SEQ_LM",
...     num_virtual_tokens=20,
...     token_dim=768,
...     num_transformer_submodules=1,
...     num_attention_heads=12,
...     num_layers=12,
...     encoder_reparameterization_type="MLP",
...     encoder_hidden_size=768,
... )

>>> prompt_encoder = PromptEncoder(config)
```

**Attributes**:

* **embedding** (`torch.nn.Embedding`) — The embedding layer of the prompt encoder.
* **mlp\_head** (`torch.nn.Sequential`) — The MLP head of the prompt encoder if `inference_mode=False`.
* **lstm\_head** (`torch.nn.LSTM`) — The LSTM head of the prompt encoder if `inference_mode=False` and `encoder_reparameterization_type="LSTM"`.
* **token\_dim** (`int`) — The hidden embedding dimension of the base transformer model.
* **input\_size** (`int`) — The input size of the prompt encoder.
* **output\_size** (`int`) — The output size of the prompt encoder.
* **hidden\_size** (`int`) — The hidden size of the prompt encoder.
* **total\_virtual\_tokens** (`int`): The total number of virtual tokens of the prompt encoder.
* **encoder\_type** (Union\[`PromptEncoderReparameterizationType`, `str`]): The encoder type of the prompt encoder.

Input shape: (`batch_size`, `total_virtual_tokens`)

Output shape: (`batch_size`, `total_virtual_tokens`, `token_dim`)

### Prefix tuning

#### class peft.PrefixTuningConfig

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning/config.py#L23)

( peft\_type: typing.Union\[str, peft.utils.peft\_types.PeftType] = Noneauto\_mapping: typing.Optional\[dict] = Nonebase\_model\_name\_or\_path: str = Nonerevision: str = Nonetask\_type: typing.Union\[str, peft.utils.peft\_types.TaskType] = Noneinference\_mode: bool = Falsenum\_virtual\_tokens: int = Nonetoken\_dim: int = Nonenum\_transformer\_submodules: typing.Optional\[int] = Nonenum\_attention\_heads: typing.Optional\[int] = Nonenum\_layers: typing.Optional\[int] = Noneencoder\_hidden\_size: int = Noneprefix\_projection: bool = False )

Parameters

* **encoder\_hidden\_size** (`int`) — The hidden size of the prompt encoder.
* **prefix\_projection** (`bool`) — Whether to project the prefix embeddings.

This is the configuration class to store the configuration of a [PrefixEncoder](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.PrefixEncoder).

#### class peft.PrefixEncoder

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning/model.py#L21)

( config )

Parameters

* **config** ([PrefixTuningConfig](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.PrefixTuningConfig)) — The configuration of the prefix encoder.

The `torch.nn` model to encode the prefix.

Example:

Copied

```
>>> from peft import PrefixEncoder, PrefixTuningConfig

>>> config = PrefixTuningConfig(
...     peft_type="PREFIX_TUNING",
...     task_type="SEQ_2_SEQ_LM",
...     num_virtual_tokens=20,
...     token_dim=768,
...     num_transformer_submodules=1,
...     num_attention_heads=12,
...     num_layers=12,
...     encoder_hidden_size=768,
... )
>>> prefix_encoder = PrefixEncoder(config)
```

**Attributes**:

* **embedding** (`torch.nn.Embedding`) — The embedding layer of the prefix encoder.
* **transform** (`torch.nn.Sequential`) — The two-layer MLP to transform the prefix embeddings if `prefix_projection` is `True`.
* **prefix\_projection** (`bool`) — Whether to project the prefix embeddings.

Input shape: (`batch_size`, `num_virtual_tokens`)

Output shape: (`batch_size`, `num_virtual_tokens`, `2*layers*hidden`)

### Prompt tuning

#### class peft.PromptTuningConfig

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning/config.py#L30)

( peft\_type: typing.Union\[str, peft.utils.peft\_types.PeftType] = Noneauto\_mapping: typing.Optional\[dict] = Nonebase\_model\_name\_or\_path: str = Nonerevision: str = Nonetask\_type: typing.Union\[str, peft.utils.peft\_types.TaskType] = Noneinference\_mode: bool = Falsenum\_virtual\_tokens: int = Nonetoken\_dim: int = Nonenum\_transformer\_submodules: typing.Optional\[int] = Nonenum\_attention\_heads: typing.Optional\[int] = Nonenum\_layers: typing.Optional\[int] = Noneprompt\_tuning\_init: typing.Union\[peft.tuners.prompt\_tuning.config.PromptTuningInit, str] = \<PromptTuningInit.RANDOM: 'RANDOM'>prompt\_tuning\_init\_text: typing.Optional\[str] = Nonetokenizer\_name\_or\_path: typing.Optional\[str] = None )

Parameters

* **prompt\_tuning\_init** (Union\[`PromptTuningInit`, `str`]) — The initialization of the prompt embedding.
* **prompt\_tuning\_init\_text** (`str`, *optional*) — The text to initialize the prompt embedding. Only used if `prompt_tuning_init` is `TEXT`.
* **tokenizer\_name\_or\_path** (`str`, *optional*) — The name or path of the tokenizer. Only used if `prompt_tuning_init` is `TEXT`.

This is the configuration class to store the configuration of a [PromptEmbedding](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.PromptEmbedding).

#### class peft.PromptEmbedding

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning/model.py#L23)

( configword\_embeddings )

Parameters

* **config** ([PromptTuningConfig](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.PromptTuningConfig)) — The configuration of the prompt embedding.
* **word\_embeddings** (`torch.nn.Module`) — The word embeddings of the base transformer model.

The model to encode virtual tokens into prompt embeddings.

**Attributes**:

* **embedding** (`torch.nn.Embedding`) — The embedding layer of the prompt embedding.

Example:

Copied

```
>>> from peft import PromptEmbedding, PromptTuningConfig

>>> config = PromptTuningConfig(
...     peft_type="PROMPT_TUNING",
...     task_type="SEQ_2_SEQ_LM",
...     num_virtual_tokens=20,
...     token_dim=768,
...     num_transformer_submodules=1,
...     num_attention_heads=12,
...     num_layers=12,
...     prompt_tuning_init="TEXT",
...     prompt_tuning_init_text="Predict if sentiment of this review is positive, negative or neutral",
...     tokenizer_name_or_path="t5-base",
... )

>>> # t5_model.shared is the word embeddings of the base model
>>> prompt_embedding = PromptEmbedding(config, t5_model.shared)
```

Input Shape: (`batch_size`, `total_virtual_tokens`)

Output Shape: (`batch_size`, `total_virtual_tokens`, `token_dim`)

### IA3

#### class peft.IA3Config

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/config.py#L24)

( peft\_type: typing.Union\[str, peft.utils.peft\_types.PeftType] = Noneauto\_mapping: typing.Optional\[dict] = Nonebase\_model\_name\_or\_path: str = Nonerevision: str = Nonetask\_type: typing.Union\[str, peft.utils.peft\_types.TaskType] = Noneinference\_mode: bool = Falsetarget\_modules: typing.Union\[str, typing.List\[str], NoneType] = Nonefeedforward\_modules: typing.Union\[str, typing.List\[str], NoneType] = Nonefan\_in\_fan\_out: bool = Falsemodules\_to\_save: typing.Optional\[typing.List\[str]] = Noneinit\_ia3\_weights: bool = True )

Parameters

* **target\_modules** (`Union[List[str],str]`) — The names of the modules to apply (IA)^3 to.
* **feedforward\_modules** (`Union[List[str],str]`) — The names of the modules to be treated as feedforward modules, as in the original paper.
* **fan\_in\_fan\_out** (`bool`) — Set this to True if the layer to replace stores weight like (fan\_in, fan\_out). For example, gpt-2 uses `Conv1D` which stores weights like (fan\_in, fan\_out) and hence this should be set to `True`.
* **modules\_to\_save** (`List[str]`) — List of modules apart from (IA)^3 layers to be set as trainable and saved in the final checkpoint.
* **init\_ia3\_weights** (`bool`) — Whether to initialize the vectors in the (IA)^3 layers, defaults to `True`.

This is the configuration class to store the configuration of a [IA3Model](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.IA3Model).

#### class peft.IA3Model

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/model.py#L45)

( modelconfigadapter\_name ) → `torch.nn.Module`

Parameters

* **model** ([PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel)) — The model to be adapted.
* **config** ([IA3Config](https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.IA3Config)) — The configuration of the (IA)^3 model.
* **adapter\_name** (`str`) — The name of the adapter, defaults to `"default"`.

Returns

`torch.nn.Module`

The (IA)^3 model.

Creates a Infused Adapter by Inhibiting and Amplifying Inner Activations ((IA)^3) model from a pretrained transformers model. The method is described in detail in <https://arxiv.org/abs/2205.05638>

Example:

Copied

```
>>> from transformers import AutoModelForSeq2SeqLM, ia3Config
>>> from peft import IA3Model, IA3Config

>>> config = IA3Config(
...     peft_type="IA3",
...     task_type="SEQ_2_SEQ_LM",
...     target_modules=["k", "v", "w0"],
...     feedforward_modules=["w0"],
... )

>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> ia3_model = IA3Model(config, model)
```

**Attributes**:

* **model** ([PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel)) — The model to be adapted.
* **peft\_config** (`ia3Config`): The configuration of the (IA)^3 model.

**merge\_and\_unload**

[\<source>](https://github.com/huggingface/peft/blob/main/src/peft/tuners/ia3/model.py#L300)

( safe\_merge: bool = False )

Parameters

* **safe\_merge** (`bool`, `optional`, defaults to `False`) — If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`.

This method merges the (IA)^3 layers into the base model. This is needed if someone wants to use the base model as a standalone model.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://boinc-ai.gitbook.io/peft/reference/tuners.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.