Overview
Last updated
Last updated
🌍 Optimum provides an API called BetterTransformer, a fast path of standard PyTorch Transformer APIs to benefit from interesting speedups on CPU & GPU through sparsity and fused kernels as Flash Attention. For now, BetterTransformer supports the fastpath from the native as well as Flash Attention and Memory-Efficient Attention from .
Since its 1.13 version, the stable version of a fast path for its standard Transformer APIs that provides out of the box performance improvements for transformer-based models. You can benefit from interesting speedup on most consumer-type devices, including CPUs, older and newer versions of NIVIDIA GPUs. You can now use this feature in🌍Optimum together with Transformers and use it for major models in the BOINC AI ecosystem.
In the 2.0 version, PyTorch includes a native scaled dot-product attention operator (SDPA) as part of torch.nn.functional
. This function encompasses several implementations that can be applied depending on the inputs and the hardware in use. See the for more information, and for benchmarks.
We provide an integration with these optimizations out of the box in 🌍 Optimum, so that you can convert any supported 🌍 Transformers model so as to use the optimized paths & scaled_dot_product_attention
function when relevant.
The PyTorch-native `scaled_dot_product_attention` operator can only dispatch to Flash Attention if no `attention_mask` is provided.
Thus, by default in training mode, the BetterTransformer integration drops the mask support and can only be used for training that do not require a padding mask for batched training. This is the case for example for masked language modeling or causal language modeling. BetterTransformer is not suited for the fine-tuning of models on tasks that requires a padding mask.
In inference mode, the padding mask is kept for correctness and thus speedups should be expected only in the batch size = 1 case.
The list of supported model below:
(SantaCoder, StarCoder)
In order to use the BetterTransformer
API just run the following commands:
Copied
You can leave keep_original_model=False
in case you want to overwrite the current model with its BetterTransformer
version.
Let us know by opening an issue in 🌍 Optimum if you want more models to be supported, or check out the if you want to add it by yourself!
More details on tutorials
section to deeply understand how to use it, or check the !