# Attention Processor ## Attention Processor An attention processor is a class for applying different types of attention mechanisms. ### AttnProcessor #### class diffusers.models.attention\_processor.AttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L541) ( ) Default processor for performing attention-related computations. ### AttnProcessor2\_0 #### class diffusers.models.attention\_processor.AttnProcessor2\_0 [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L977) ( ) Processor for implementing scaled dot-product attention (enabled by default if you’re using PyTorch 2.0). ### LoRAAttnProcessor #### class diffusers.models.attention\_processor.LoRAAttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1361) ( hidden\_sizecross\_attention\_dim = Nonerank = 4network\_alpha = None\*\*kwargs ) Parameters * **hidden\_size** (`int`, *optional*) — The hidden size of the attention layer. * **cross\_attention\_dim** (`int`, *optional*) — The number of channels in the `encoder_hidden_states`. * **rank** (`int`, defaults to 4) — The dimension of the LoRA update matrices. * **network\_alpha** (`int`, *optional*) — Equivalent to `alpha` but it’s usage is specific to Kohya (A1111) style LoRAs. Processor for implementing the LoRA attention mechanism. ### LoRAAttnProcessor2\_0 #### class diffusers.models.attention\_processor.LoRAAttnProcessor2\_0 [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1424) ( hidden\_sizecross\_attention\_dim = Nonerank = 4network\_alpha = None\*\*kwargs ) Parameters * **hidden\_size** (`int`) — The hidden size of the attention layer. * **cross\_attention\_dim** (`int`, *optional*) — The number of channels in the `encoder_hidden_states`. * **rank** (`int`, defaults to 4) — The dimension of the LoRA update matrices. * **network\_alpha** (`int`, *optional*) — Equivalent to `alpha` but it’s usage is specific to Kohya (A1111) style LoRAs. Processor for implementing the LoRA attention mechanism using PyTorch 2.0’s memory-efficient scaled dot-product attention. ### CustomDiffusionAttnProcessor #### class diffusers.models.attention\_processor.CustomDiffusionAttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L608) ( train\_kv = Truetrain\_q\_out = Truehidden\_size = Nonecross\_attention\_dim = Noneout\_bias = Truedropout = 0.0 ) Parameters * **train\_kv** (`bool`, defaults to `True`) — Whether to newly train the key and value matrices corresponding to the text features. * **train\_q\_out** (`bool`, defaults to `True`) — Whether to newly train query matrices corresponding to the latent image features. * **hidden\_size** (`int`, *optional*, defaults to `None`) — The hidden size of the attention layer. * **cross\_attention\_dim** (`int`, *optional*, defaults to `None`) — The number of channels in the `encoder_hidden_states`. * **out\_bias** (`bool`, defaults to `True`) — Whether to include the bias parameter in `train_q_out`. * **dropout** (`float`, *optional*, defaults to 0.0) — The dropout probability to use. Processor for implementing attention for the Custom Diffusion method. ### AttnAddedKVProcessor #### class diffusers.models.attention\_processor.AttnAddedKVProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L706) ( ) Processor for performing attention-related computations with extra learnable key and value matrices for the text encoder. ### AttnAddedKVProcessor2\_0 #### class diffusers.models.attention\_processor.AttnAddedKVProcessor2\_0 [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L760) ( ) Processor for performing scaled dot-product attention (enabled by default if you’re using PyTorch 2.0), with extra learnable key and value matrices for the text encoder. ### LoRAAttnAddedKVProcessor #### class diffusers.models.attention\_processor.LoRAAttnAddedKVProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1568) ( hidden\_sizecross\_attention\_dim = Nonerank = 4network\_alpha = None ) Parameters * **hidden\_size** (`int`, *optional*) — The hidden size of the attention layer. * **cross\_attention\_dim** (`int`, *optional*, defaults to `None`) — The number of channels in the `encoder_hidden_states`. * **rank** (`int`, defaults to 4) — The dimension of the LoRA update matrices. Processor for implementing the LoRA attention mechanism with extra learnable key and value matrices for the text encoder. ### XFormersAttnProcessor #### class diffusers.models.attention\_processor.XFormersAttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L888) ( attention\_op: typing.Optional\[typing.Callable] = None ) Parameters * **attention\_op** (`Callable`, *optional*, defaults to `None`) — The base [operator](https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.AttentionOpBase) to use as the attention operator. It is recommended to set to `None`, and allow xFormers to choose the best operator. Processor for implementing memory efficient attention using xFormers. ### LoRAXFormersAttnProcessor #### class diffusers.models.attention\_processor.LoRAXFormersAttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1490) ( hidden\_sizecross\_attention\_dimrank = 4attention\_op: typing.Optional\[typing.Callable] = Nonenetwork\_alpha = None\*\*kwargs ) Parameters * **hidden\_size** (`int`, *optional*) — The hidden size of the attention layer. * **cross\_attention\_dim** (`int`, *optional*) — The number of channels in the `encoder_hidden_states`. * **rank** (`int`, defaults to 4) — The dimension of the LoRA update matrices. * **attention\_op** (`Callable`, *optional*, defaults to `None`) — The base [operator](https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.AttentionOpBase) to use as the attention operator. It is recommended to set to `None`, and allow xFormers to choose the best operator. * **network\_alpha** (`int`, *optional*) — Equivalent to `alpha` but it’s usage is specific to Kohya (A1111) style LoRAs. Processor for implementing the LoRA attention mechanism with memory efficient attention using xFormers. ### CustomDiffusionXFormersAttnProcessor #### class diffusers.models.attention\_processor.CustomDiffusionXFormersAttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1062) ( train\_kv = Truetrain\_q\_out = Falsehidden\_size = Nonecross\_attention\_dim = Noneout\_bias = Truedropout = 0.0attention\_op: typing.Optional\[typing.Callable] = None ) Parameters * **train\_kv** (`bool`, defaults to `True`) — Whether to newly train the key and value matrices corresponding to the text features. * **train\_q\_out** (`bool`, defaults to `True`) — Whether to newly train query matrices corresponding to the latent image features. * **hidden\_size** (`int`, *optional*, defaults to `None`) — The hidden size of the attention layer. * **cross\_attention\_dim** (`int`, *optional*, defaults to `None`) — The number of channels in the `encoder_hidden_states`. * **out\_bias** (`bool`, defaults to `True`) — Whether to include the bias parameter in `train_q_out`. * **dropout** (`float`, *optional*, defaults to 0.0) — The dropout probability to use. * **attention\_op** (`Callable`, *optional*, defaults to `None`) — The base [operator](https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.AttentionOpBase) to use as the attention operator. It is recommended to set to `None`, and allow xFormers to choose the best operator. Processor for implementing memory efficient attention using xFormers for the Custom Diffusion method. ### SlicedAttnProcessor #### class diffusers.models.attention\_processor.SlicedAttnProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1171) ( slice\_size ) Parameters * **slice\_size** (`int`, *optional*) — The number of steps to compute attention. Uses as many slices as `attention_head_dim // slice_size`, and `attention_head_dim` must be a multiple of the `slice_size`. Processor for implementing sliced attention. ### SlicedAttnAddedKVProcessor #### class diffusers.models.attention\_processor.SlicedAttnAddedKVProcessor [\](https://github.com/huggingface/diffusers/blob/v0.21.0/src/diffusers/models/attention_processor.py#L1252) ( slice\_size ) Parameters * **slice\_size** (`int`, *optional*) — The number of steps to compute attention. Uses as many slices as `attention_head_dim // slice_size`, and `attention_head_dim` must be a multiple of the `slice_size`. Processor for implementing sliced attention with extra learnable key and value matrices for the text encoder. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://boinc-ai.gitbook.io/diffusers/api/main-classes/attention-processor.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.