# AutoTokenizer

### AutoTokenizer

#### class transformers.AutoTokenizer

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/tokenization_auto.py#L560)

( )

This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [AutoTokenizer.from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) class method.

This class cannot be instantiated directly using `__init__()` (throws an error).

**from\_pretrained**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/tokenization_auto.py#L574)

( pretrained\_model\_name\_or\_path\*inputs\*\*kwargs )

Parameters

* **pretrained\_model\_name\_or\_path** (`str` or `os.PathLike`) — Can be either:
  * A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`.
  * A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [save\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.save_pretrained) method, e.g., `./my_model_directory/`.
  * A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes)
* **inputs** (additional positional arguments, *optional*) — Will be passed along to the Tokenizer `__init__()` method.
* **config** ([PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig), *optional*) — The configuration object used to determine the tokenizer class to instantiate.
* **cache\_dir** (`str` or `os.PathLike`, *optional*) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
* **force\_download** (`bool`, *optional*, defaults to `False`) — Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist.
* **resume\_download** (`bool`, *optional*, defaults to `False`) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
* **proxies** (`Dict[str, str]`, *optional*) — A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
* **revision** (`str`, *optional*, defaults to `"main"`) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git.
* **subfolder** (`str`, *optional*) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here.
* **use\_fast** (`bool`, *optional*, defaults to `True`) — Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead.
* **tokenizer\_type** (`str`, *optional*) — Tokenizer type to be loaded.
* **trust\_remote\_code** (`bool`, *optional*, defaults to `False`) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
* **kwargs** (additional keyword arguments, *optional*) — Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details.

Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary.

The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it’s missing, by falling back to using pattern matching on `pretrained_model_name_or_path`:

* **albert** — [AlbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizer) or [AlbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizerFast) (ALBERT model)
* **align** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (ALIGN model)
* **bark** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Bark model)
* **bart** — [BartTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartTokenizer) or [BartTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartTokenizerFast) (BART model)
* **barthez** — [BarthezTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/barthez#transformers.BarthezTokenizer) or [BarthezTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/barthez#transformers.BarthezTokenizerFast) (BARThez model)
* **bartpho** — [BartphoTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bartpho#transformers.BartphoTokenizer) (BARTpho model)
* **bert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (BERT model)
* **bert-generation** — [BertGenerationTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-generation#transformers.BertGenerationTokenizer) (Bert Generation model)
* **bert-japanese** — [BertJapaneseTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-japanese#transformers.BertJapaneseTokenizer) (BertJapanese model)
* **bertweet** — [BertweetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bertweet#transformers.BertweetTokenizer) (BERTweet model)
* **big\_bird** — [BigBirdTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdTokenizer) or [BigBirdTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdTokenizerFast) (BigBird model)
* **bigbird\_pegasus** — [PegasusTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizer) or [PegasusTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizerFast) (BigBird-Pegasus model)
* **biogpt** — [BioGptTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/biogpt#transformers.BioGptTokenizer) (BioGpt model)
* **blenderbot** — [BlenderbotTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotTokenizer) or [BlenderbotTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotTokenizerFast) (Blenderbot model)
* **blenderbot-small** — [BlenderbotSmallTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot-small#transformers.BlenderbotSmallTokenizer) (BlenderbotSmall model)
* **blip** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (BLIP model)
* **blip-2** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (BLIP-2 model)
* **bloom** — [BloomTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bloom#transformers.BloomTokenizerFast) (BLOOM model)
* **bridgetower** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (BridgeTower model)
* **bros** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (BROS model)
* **byt5** — [ByT5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/byt5#transformers.ByT5Tokenizer) (ByT5 model)
* **camembert** — [CamembertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertTokenizer) or [CamembertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertTokenizerFast) (CamemBERT model)
* **canine** — [CanineTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/canine#transformers.CanineTokenizer) (CANINE model)
* **chinese\_clip** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Chinese-CLIP model)
* **clap** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (CLAP model)
* **clip** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (CLIP model)
* **clipseg** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (CLIPSeg model)
* **code\_llama** — [CodeLlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/code_llama#transformers.CodeLlamaTokenizer) or [CodeLlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/code_llama#transformers.CodeLlamaTokenizerFast) (CodeLlama model)
* **codegen** — [CodeGenTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenTokenizer) or [CodeGenTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenTokenizerFast) (CodeGen model)
* **convbert** — [ConvBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/convbert#transformers.ConvBertTokenizer) or [ConvBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/convbert#transformers.ConvBertTokenizerFast) (ConvBERT model)
* **cpm** — [CpmTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpm#transformers.CpmTokenizer) or [CpmTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpm#transformers.CpmTokenizerFast) (CPM model)
* **cpmant** — [CpmAntTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpmant#transformers.CpmAntTokenizer) (CPM-Ant model)
* **ctrl** — [CTRLTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ctrl#transformers.CTRLTokenizer) (CTRL model)
* **data2vec-audio** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Data2VecAudio model)
* **data2vec-text** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (Data2VecText model)
* **deberta** — [DebertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta#transformers.DebertaTokenizer) or [DebertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta#transformers.DebertaTokenizerFast) (DeBERTa model)
* **deberta-v2** — [DebertaV2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta-v2#transformers.DebertaV2Tokenizer) or [DebertaV2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta-v2#transformers.DebertaV2TokenizerFast) (DeBERTa-v2 model)
* **distilbert** — [DistilBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/distilbert#transformers.DistilBertTokenizer) or [DistilBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/distilbert#transformers.DistilBertTokenizerFast) (DistilBERT model)
* **dpr** — [DPRQuestionEncoderTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/dpr#transformers.DPRQuestionEncoderTokenizer) or [DPRQuestionEncoderTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/dpr#transformers.DPRQuestionEncoderTokenizerFast) (DPR model)
* **electra** — [ElectraTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraTokenizer) or [ElectraTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraTokenizerFast) (ELECTRA model)
* **ernie** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (ERNIE model)
* **ernie\_m** — [ErnieMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ernie_m#transformers.ErnieMTokenizer) (ErnieM model)
* **esm** — [EsmTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/esm#transformers.EsmTokenizer) (ESM model)
* **flaubert** — [FlaubertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/flaubert#transformers.FlaubertTokenizer) (FlauBERT model)
* **fnet** — [FNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/fnet#transformers.FNetTokenizer) or [FNetTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/fnet#transformers.FNetTokenizerFast) (FNet model)
* **fsmt** — [FSMTTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/fsmt#transformers.FSMTTokenizer) (FairSeq Machine-Translation model)
* **funnel** — [FunnelTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/funnel#transformers.FunnelTokenizer) or [FunnelTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/funnel#transformers.FunnelTokenizerFast) (Funnel Transformer model)
* **git** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (GIT model)
* **gpt-sw3** — [GPTSw3Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt-sw3#transformers.GPTSw3Tokenizer) (GPT-Sw3 model)
* **gpt2** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (OpenAI GPT-2 model)
* **gpt\_bigcode** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (GPTBigCode model)
* **gpt\_neo** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (GPT Neo model)
* **gpt\_neox** — [GPTNeoXTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXTokenizerFast) (GPT NeoX model)
* **gpt\_neox\_japanese** — [GPTNeoXJapaneseTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox_japanese#transformers.GPTNeoXJapaneseTokenizer) (GPT NeoX Japanese model)
* **gptj** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (GPT-J model)
* **gptsan-japanese** — [GPTSanJapaneseTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gptsan-japanese#transformers.GPTSanJapaneseTokenizer) (GPTSAN-japanese model)
* **groupvit** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (GroupViT model)
* **herbert** — [HerbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/herbert#transformers.HerbertTokenizer) or [HerbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/herbert#transformers.HerbertTokenizerFast) (HerBERT model)
* **hubert** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Hubert model)
* **ibert** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (I-BERT model)
* **idefics** — [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (IDEFICS model)
* **instructblip** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (InstructBLIP model)
* **jukebox** — [JukeboxTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/jukebox#transformers.JukeboxTokenizer) (Jukebox model)
* **layoutlm** — [LayoutLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlm#transformers.LayoutLMTokenizer) or [LayoutLMTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlm#transformers.LayoutLMTokenizerFast) (LayoutLM model)
* **layoutlmv2** — [LayoutLMv2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv2#transformers.LayoutLMv2Tokenizer) or [LayoutLMv2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv2#transformers.LayoutLMv2TokenizerFast) (LayoutLMv2 model)
* **layoutlmv3** — [LayoutLMv3Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3Tokenizer) or [LayoutLMv3TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3TokenizerFast) (LayoutLMv3 model)
* **layoutxlm** — [LayoutXLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer) or [LayoutXLMTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizerFast) (LayoutXLM model)
* **led** — [LEDTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/led#transformers.LEDTokenizer) or [LEDTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/led#transformers.LEDTokenizerFast) (LED model)
* **lilt** — [LayoutLMv3Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3Tokenizer) or [LayoutLMv3TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3TokenizerFast) (LiLT model)
* **llama** — [LlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizer) or [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (LLaMA model)
* **longformer** — [LongformerTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/longformer#transformers.LongformerTokenizer) or [LongformerTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/longformer#transformers.LongformerTokenizerFast) (Longformer model)
* **longt5** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (LongT5 model)
* **luke** — [LukeTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/luke#transformers.LukeTokenizer) (LUKE model)
* **lxmert** — [LxmertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/lxmert#transformers.LxmertTokenizer) or [LxmertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/lxmert#transformers.LxmertTokenizerFast) (LXMERT model)
* **m2m\_100** — [M2M100Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/m2m_100#transformers.M2M100Tokenizer) (M2M100 model)
* **marian** — [MarianTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/marian#transformers.MarianTokenizer) (Marian model)
* **mbart** — [MBartTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartTokenizer) or [MBartTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartTokenizerFast) (mBART model)
* **mbart50** — [MBart50Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBart50Tokenizer) or [MBart50TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBart50TokenizerFast) (mBART-50 model)
* **mega** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (MEGA model)
* **megatron-bert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Megatron-BERT model)
* **mgp-str** — [MgpstrTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mgp-str#transformers.MgpstrTokenizer) (MGP-STR model)
* **mistral** — [LlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizer) or [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (Mistral model)
* **mluke** — [MLukeTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mluke#transformers.MLukeTokenizer) (mLUKE model)
* **mobilebert** — [MobileBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mobilebert#transformers.MobileBertTokenizer) or [MobileBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mobilebert#transformers.MobileBertTokenizerFast) (MobileBERT model)
* **mpnet** — [MPNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpnet#transformers.MPNetTokenizer) or [MPNetTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpnet#transformers.MPNetTokenizerFast) (MPNet model)
* **mpt** — [GPTNeoXTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXTokenizerFast) (MPT model)
* **mra** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (MRA model)
* **mt5** — [MT5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [MT5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (MT5 model)
* **musicgen** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (MusicGen model)
* **mvp** — [MvpTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpTokenizer) or [MvpTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpTokenizerFast) (MVP model)
* **nezha** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Nezha model)
* **nllb** — [NllbTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizer) or [NllbTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizerFast) (NLLB model)
* **nllb-moe** — [NllbTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizer) or [NllbTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizerFast) (NLLB-MOE model)
* **nystromformer** — [AlbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizer) or [AlbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizerFast) (Nyströmformer model)
* **oneformer** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (OneFormer model)
* **openai-gpt** — [OpenAIGPTTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTTokenizer) or [OpenAIGPTTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTTokenizerFast) (OpenAI GPT model)
* **opt** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (OPT model)
* **owlvit** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (OWL-ViT model)
* **pegasus** — [PegasusTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizer) or [PegasusTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizerFast) (Pegasus model)
* **pegasus\_x** — [PegasusTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizer) or [PegasusTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizerFast) (PEGASUS-X model)
* **perceiver** — [PerceiverTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/perceiver#transformers.PerceiverTokenizer) (Perceiver model)
* **persimmon** — [LlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizer) or [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (Persimmon model)
* **phobert** — [PhobertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/phobert#transformers.PhobertTokenizer) (PhoBERT model)
* **pix2struct** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (Pix2Struct model)
* **plbart** — [PLBartTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/plbart#transformers.PLBartTokenizer) (PLBart model)
* **prophetnet** — [ProphetNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/prophetnet#transformers.ProphetNetTokenizer) (ProphetNet model)
* **qdqbert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (QDQBert model)
* **rag** — [RagTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rag#transformers.RagTokenizer) (RAG model)
* **realm** — [RealmTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/realm#transformers.RealmTokenizer) or [RealmTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/realm#transformers.RealmTokenizerFast) (REALM model)
* **reformer** — [ReformerTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerTokenizer) or [ReformerTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerTokenizerFast) (Reformer model)
* **rembert** — [RemBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertTokenizer) or [RemBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertTokenizerFast) (RemBERT model)
* **retribert** — [RetriBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/retribert#transformers.RetriBertTokenizer) or [RetriBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/retribert#transformers.RetriBertTokenizerFast) (RetriBERT model)
* **roberta** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (RoBERTa model)
* **roberta-prelayernorm** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (RoBERTa-PreLayerNorm model)
* **roc\_bert** — [RoCBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roc_bert#transformers.RoCBertTokenizer) (RoCBert model)
* **roformer** — [RoFormerTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerTokenizer) or [RoFormerTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerTokenizerFast) (RoFormer model)
* **rwkv** — [GPTNeoXTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXTokenizerFast) (RWKV model)
* **speech\_to\_text** — [Speech2TextTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text#transformers.Speech2TextTokenizer) (Speech2Text model)
* **speech\_to\_text\_2** — [Speech2Text2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text_2#transformers.Speech2Text2Tokenizer) (Speech2Text2 model)
* **speecht5** — [SpeechT5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speecht5#transformers.SpeechT5Tokenizer) (SpeechT5 model)
* **splinter** — [SplinterTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/splinter#transformers.SplinterTokenizer) or [SplinterTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/splinter#transformers.SplinterTokenizerFast) (Splinter model)
* **squeezebert** — [SqueezeBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/squeezebert#transformers.SqueezeBertTokenizer) or [SqueezeBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/squeezebert#transformers.SqueezeBertTokenizerFast) (SqueezeBERT model)
* **switch\_transformers** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (SwitchTransformers model)
* **t5** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (T5 model)
* **tapas** — [TapasTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/tapas#transformers.TapasTokenizer) (TAPAS model)
* **tapex** — [TapexTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/tapex#transformers.TapexTokenizer) (TAPEX model)
* **transfo-xl** — [TransfoXLTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/transfo-xl#transformers.TransfoXLTokenizer) (Transformer-XL model)
* **umt5** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (UMT5 model)
* **vilt** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (ViLT model)
* **visual\_bert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (VisualBERT model)
* **vits** — [VitsTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/vits#transformers.VitsTokenizer) (VITS model)
* **wav2vec2** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Wav2Vec2 model)
* **wav2vec2-conformer** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Wav2Vec2-Conformer model)
* **wav2vec2\_phoneme** — [Wav2Vec2PhonemeCTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2_phoneme#transformers.Wav2Vec2PhonemeCTCTokenizer) (Wav2Vec2Phoneme model)
* **whisper** — [WhisperTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/whisper#transformers.WhisperTokenizer) or [WhisperTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/whisper#transformers.WhisperTokenizerFast) (Whisper model)
* **xclip** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (X-CLIP model)
* **xglm** — [XGLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMTokenizer) or [XGLMTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMTokenizerFast) (XGLM model)
* **xlm** — [XLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm#transformers.XLMTokenizer) (XLM model)
* **xlm-prophetnet** — [XLMProphetNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-prophetnet#transformers.XLMProphetNetTokenizer) (XLM-ProphetNet model)
* **xlm-roberta** — [XLMRobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) or [XLMRobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast) (XLM-RoBERTa model)
* **xlm-roberta-xl** — [XLMRobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) or [XLMRobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast) (XLM-RoBERTa-XL model)
* **xlnet** — [XLNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetTokenizer) or [XLNetTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetTokenizerFast) (XLNet model)
* **xmod** — [XLMRobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) or [XLMRobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast) (X-MOD model)
* **yoso** — [AlbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizer) or [AlbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizerFast) (YOSO model)

Examples:

Copied

```
>>> from transformers import AutoTokenizer

>>> # Download vocabulary from huggingface.co and cache.
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

>>> # Download vocabulary from huggingface.co (user-uploaded) and cache.
>>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")

>>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*)
>>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/")

>>> # Download vocabulary from huggingface.co and define model-specific arguments
>>> tokenizer = AutoTokenizer.from_pretrained("roberta-base", add_prefix_space=True)
```

**register**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/tokenization_auto.py#L784)

( config\_classslow\_tokenizer\_class = Nonefast\_tokenizer\_class = Noneexist\_ok = False )

Parameters

* **config\_class** ([PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig)) — The configuration corresponding to the model to register.
* **slow\_tokenizer\_class** (`PretrainedTokenizer`, *optional*) — The slow tokenizer to register.
* **fast\_tokenizer\_class** (`PretrainedTokenizerFast`, *optional*) — The fast tokenizer to register.

Register a new tokenizer in this mapping.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://boinc-ai.gitbook.io/transformers/api/main-classes/auto-classes/autotokenizer.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.