> For the complete documentation index, see [llms.txt](https://boinc-ai.gitbook.io/transformers/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://boinc-ai.gitbook.io/transformers/api/main-classes/auto-classes/autotokenizer.md).

# AutoTokenizer

### AutoTokenizer

#### class transformers.AutoTokenizer

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/tokenization_auto.py#L560)

( )

This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [AutoTokenizer.from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) class method.

This class cannot be instantiated directly using `__init__()` (throws an error).

**from\_pretrained**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/tokenization_auto.py#L574)

( pretrained\_model\_name\_or\_path\*inputs\*\*kwargs )

Parameters

* **pretrained\_model\_name\_or\_path** (`str` or `os.PathLike`) — Can be either:
  * A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`.
  * A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [save\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.save_pretrained) method, e.g., `./my_model_directory/`.
  * A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes)
* **inputs** (additional positional arguments, *optional*) — Will be passed along to the Tokenizer `__init__()` method.
* **config** ([PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig), *optional*) — The configuration object used to determine the tokenizer class to instantiate.
* **cache\_dir** (`str` or `os.PathLike`, *optional*) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
* **force\_download** (`bool`, *optional*, defaults to `False`) — Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist.
* **resume\_download** (`bool`, *optional*, defaults to `False`) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
* **proxies** (`Dict[str, str]`, *optional*) — A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
* **revision** (`str`, *optional*, defaults to `"main"`) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git.
* **subfolder** (`str`, *optional*) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here.
* **use\_fast** (`bool`, *optional*, defaults to `True`) — Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead.
* **tokenizer\_type** (`str`, *optional*) — Tokenizer type to be loaded.
* **trust\_remote\_code** (`bool`, *optional*, defaults to `False`) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine.
* **kwargs** (additional keyword arguments, *optional*) — Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details.

Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary.

The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it’s missing, by falling back to using pattern matching on `pretrained_model_name_or_path`:

* **albert** — [AlbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizer) or [AlbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizerFast) (ALBERT model)
* **align** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (ALIGN model)
* **bark** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Bark model)
* **bart** — [BartTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartTokenizer) or [BartTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartTokenizerFast) (BART model)
* **barthez** — [BarthezTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/barthez#transformers.BarthezTokenizer) or [BarthezTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/barthez#transformers.BarthezTokenizerFast) (BARThez model)
* **bartpho** — [BartphoTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bartpho#transformers.BartphoTokenizer) (BARTpho model)
* **bert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (BERT model)
* **bert-generation** — [BertGenerationTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-generation#transformers.BertGenerationTokenizer) (Bert Generation model)
* **bert-japanese** — [BertJapaneseTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-japanese#transformers.BertJapaneseTokenizer) (BertJapanese model)
* **bertweet** — [BertweetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bertweet#transformers.BertweetTokenizer) (BERTweet model)
* **big\_bird** — [BigBirdTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdTokenizer) or [BigBirdTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdTokenizerFast) (BigBird model)
* **bigbird\_pegasus** — [PegasusTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizer) or [PegasusTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizerFast) (BigBird-Pegasus model)
* **biogpt** — [BioGptTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/biogpt#transformers.BioGptTokenizer) (BioGpt model)
* **blenderbot** — [BlenderbotTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotTokenizer) or [BlenderbotTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotTokenizerFast) (Blenderbot model)
* **blenderbot-small** — [BlenderbotSmallTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot-small#transformers.BlenderbotSmallTokenizer) (BlenderbotSmall model)
* **blip** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (BLIP model)
* **blip-2** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (BLIP-2 model)
* **bloom** — [BloomTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bloom#transformers.BloomTokenizerFast) (BLOOM model)
* **bridgetower** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (BridgeTower model)
* **bros** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (BROS model)
* **byt5** — [ByT5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/byt5#transformers.ByT5Tokenizer) (ByT5 model)
* **camembert** — [CamembertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertTokenizer) or [CamembertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertTokenizerFast) (CamemBERT model)
* **canine** — [CanineTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/canine#transformers.CanineTokenizer) (CANINE model)
* **chinese\_clip** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Chinese-CLIP model)
* **clap** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (CLAP model)
* **clip** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (CLIP model)
* **clipseg** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (CLIPSeg model)
* **code\_llama** — [CodeLlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/code_llama#transformers.CodeLlamaTokenizer) or [CodeLlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/code_llama#transformers.CodeLlamaTokenizerFast) (CodeLlama model)
* **codegen** — [CodeGenTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenTokenizer) or [CodeGenTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenTokenizerFast) (CodeGen model)
* **convbert** — [ConvBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/convbert#transformers.ConvBertTokenizer) or [ConvBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/convbert#transformers.ConvBertTokenizerFast) (ConvBERT model)
* **cpm** — [CpmTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpm#transformers.CpmTokenizer) or [CpmTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpm#transformers.CpmTokenizerFast) (CPM model)
* **cpmant** — [CpmAntTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpmant#transformers.CpmAntTokenizer) (CPM-Ant model)
* **ctrl** — [CTRLTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ctrl#transformers.CTRLTokenizer) (CTRL model)
* **data2vec-audio** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Data2VecAudio model)
* **data2vec-text** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (Data2VecText model)
* **deberta** — [DebertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta#transformers.DebertaTokenizer) or [DebertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta#transformers.DebertaTokenizerFast) (DeBERTa model)
* **deberta-v2** — [DebertaV2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta-v2#transformers.DebertaV2Tokenizer) or [DebertaV2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/deberta-v2#transformers.DebertaV2TokenizerFast) (DeBERTa-v2 model)
* **distilbert** — [DistilBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/distilbert#transformers.DistilBertTokenizer) or [DistilBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/distilbert#transformers.DistilBertTokenizerFast) (DistilBERT model)
* **dpr** — [DPRQuestionEncoderTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/dpr#transformers.DPRQuestionEncoderTokenizer) or [DPRQuestionEncoderTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/dpr#transformers.DPRQuestionEncoderTokenizerFast) (DPR model)
* **electra** — [ElectraTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraTokenizer) or [ElectraTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraTokenizerFast) (ELECTRA model)
* **ernie** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (ERNIE model)
* **ernie\_m** — [ErnieMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ernie_m#transformers.ErnieMTokenizer) (ErnieM model)
* **esm** — [EsmTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/esm#transformers.EsmTokenizer) (ESM model)
* **flaubert** — [FlaubertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/flaubert#transformers.FlaubertTokenizer) (FlauBERT model)
* **fnet** — [FNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/fnet#transformers.FNetTokenizer) or [FNetTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/fnet#transformers.FNetTokenizerFast) (FNet model)
* **fsmt** — [FSMTTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/fsmt#transformers.FSMTTokenizer) (FairSeq Machine-Translation model)
* **funnel** — [FunnelTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/funnel#transformers.FunnelTokenizer) or [FunnelTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/funnel#transformers.FunnelTokenizerFast) (Funnel Transformer model)
* **git** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (GIT model)
* **gpt-sw3** — [GPTSw3Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt-sw3#transformers.GPTSw3Tokenizer) (GPT-Sw3 model)
* **gpt2** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (OpenAI GPT-2 model)
* **gpt\_bigcode** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (GPTBigCode model)
* **gpt\_neo** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (GPT Neo model)
* **gpt\_neox** — [GPTNeoXTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXTokenizerFast) (GPT NeoX model)
* **gpt\_neox\_japanese** — [GPTNeoXJapaneseTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox_japanese#transformers.GPTNeoXJapaneseTokenizer) (GPT NeoX Japanese model)
* **gptj** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (GPT-J model)
* **gptsan-japanese** — [GPTSanJapaneseTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gptsan-japanese#transformers.GPTSanJapaneseTokenizer) (GPTSAN-japanese model)
* **groupvit** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (GroupViT model)
* **herbert** — [HerbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/herbert#transformers.HerbertTokenizer) or [HerbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/herbert#transformers.HerbertTokenizerFast) (HerBERT model)
* **hubert** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Hubert model)
* **ibert** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (I-BERT model)
* **idefics** — [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (IDEFICS model)
* **instructblip** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (InstructBLIP model)
* **jukebox** — [JukeboxTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/jukebox#transformers.JukeboxTokenizer) (Jukebox model)
* **layoutlm** — [LayoutLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlm#transformers.LayoutLMTokenizer) or [LayoutLMTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlm#transformers.LayoutLMTokenizerFast) (LayoutLM model)
* **layoutlmv2** — [LayoutLMv2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv2#transformers.LayoutLMv2Tokenizer) or [LayoutLMv2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv2#transformers.LayoutLMv2TokenizerFast) (LayoutLMv2 model)
* **layoutlmv3** — [LayoutLMv3Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3Tokenizer) or [LayoutLMv3TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3TokenizerFast) (LayoutLMv3 model)
* **layoutxlm** — [LayoutXLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer) or [LayoutXLMTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizerFast) (LayoutXLM model)
* **led** — [LEDTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/led#transformers.LEDTokenizer) or [LEDTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/led#transformers.LEDTokenizerFast) (LED model)
* **lilt** — [LayoutLMv3Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3Tokenizer) or [LayoutLMv3TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/layoutlmv3#transformers.LayoutLMv3TokenizerFast) (LiLT model)
* **llama** — [LlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizer) or [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (LLaMA model)
* **longformer** — [LongformerTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/longformer#transformers.LongformerTokenizer) or [LongformerTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/longformer#transformers.LongformerTokenizerFast) (Longformer model)
* **longt5** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (LongT5 model)
* **luke** — [LukeTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/luke#transformers.LukeTokenizer) (LUKE model)
* **lxmert** — [LxmertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/lxmert#transformers.LxmertTokenizer) or [LxmertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/lxmert#transformers.LxmertTokenizerFast) (LXMERT model)
* **m2m\_100** — [M2M100Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/m2m_100#transformers.M2M100Tokenizer) (M2M100 model)
* **marian** — [MarianTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/marian#transformers.MarianTokenizer) (Marian model)
* **mbart** — [MBartTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartTokenizer) or [MBartTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartTokenizerFast) (mBART model)
* **mbart50** — [MBart50Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBart50Tokenizer) or [MBart50TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBart50TokenizerFast) (mBART-50 model)
* **mega** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (MEGA model)
* **megatron-bert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Megatron-BERT model)
* **mgp-str** — [MgpstrTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mgp-str#transformers.MgpstrTokenizer) (MGP-STR model)
* **mistral** — [LlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizer) or [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (Mistral model)
* **mluke** — [MLukeTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mluke#transformers.MLukeTokenizer) (mLUKE model)
* **mobilebert** — [MobileBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mobilebert#transformers.MobileBertTokenizer) or [MobileBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mobilebert#transformers.MobileBertTokenizerFast) (MobileBERT model)
* **mpnet** — [MPNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpnet#transformers.MPNetTokenizer) or [MPNetTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpnet#transformers.MPNetTokenizerFast) (MPNet model)
* **mpt** — [GPTNeoXTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXTokenizerFast) (MPT model)
* **mra** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (MRA model)
* **mt5** — [MT5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [MT5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (MT5 model)
* **musicgen** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (MusicGen model)
* **mvp** — [MvpTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpTokenizer) or [MvpTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpTokenizerFast) (MVP model)
* **nezha** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (Nezha model)
* **nllb** — [NllbTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizer) or [NllbTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizerFast) (NLLB model)
* **nllb-moe** — [NllbTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizer) or [NllbTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/nllb#transformers.NllbTokenizerFast) (NLLB-MOE model)
* **nystromformer** — [AlbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizer) or [AlbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizerFast) (Nyströmformer model)
* **oneformer** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (OneFormer model)
* **openai-gpt** — [OpenAIGPTTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTTokenizer) or [OpenAIGPTTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTTokenizerFast) (OpenAI GPT model)
* **opt** — [GPT2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Tokenizer) or [GPT2TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2TokenizerFast) (OPT model)
* **owlvit** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (OWL-ViT model)
* **pegasus** — [PegasusTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizer) or [PegasusTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizerFast) (Pegasus model)
* **pegasus\_x** — [PegasusTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizer) or [PegasusTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusTokenizerFast) (PEGASUS-X model)
* **perceiver** — [PerceiverTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/perceiver#transformers.PerceiverTokenizer) (Perceiver model)
* **persimmon** — [LlamaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizer) or [LlamaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaTokenizerFast) (Persimmon model)
* **phobert** — [PhobertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/phobert#transformers.PhobertTokenizer) (PhoBERT model)
* **pix2struct** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (Pix2Struct model)
* **plbart** — [PLBartTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/plbart#transformers.PLBartTokenizer) (PLBart model)
* **prophetnet** — [ProphetNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/prophetnet#transformers.ProphetNetTokenizer) (ProphetNet model)
* **qdqbert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (QDQBert model)
* **rag** — [RagTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rag#transformers.RagTokenizer) (RAG model)
* **realm** — [RealmTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/realm#transformers.RealmTokenizer) or [RealmTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/realm#transformers.RealmTokenizerFast) (REALM model)
* **reformer** — [ReformerTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerTokenizer) or [ReformerTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerTokenizerFast) (Reformer model)
* **rembert** — [RemBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertTokenizer) or [RemBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertTokenizerFast) (RemBERT model)
* **retribert** — [RetriBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/retribert#transformers.RetriBertTokenizer) or [RetriBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/retribert#transformers.RetriBertTokenizerFast) (RetriBERT model)
* **roberta** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (RoBERTa model)
* **roberta-prelayernorm** — [RobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizer) or [RobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaTokenizerFast) (RoBERTa-PreLayerNorm model)
* **roc\_bert** — [RoCBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roc_bert#transformers.RoCBertTokenizer) (RoCBert model)
* **roformer** — [RoFormerTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerTokenizer) or [RoFormerTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerTokenizerFast) (RoFormer model)
* **rwkv** — [GPTNeoXTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXTokenizerFast) (RWKV model)
* **speech\_to\_text** — [Speech2TextTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text#transformers.Speech2TextTokenizer) (Speech2Text model)
* **speech\_to\_text\_2** — [Speech2Text2Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text_2#transformers.Speech2Text2Tokenizer) (Speech2Text2 model)
* **speecht5** — [SpeechT5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speecht5#transformers.SpeechT5Tokenizer) (SpeechT5 model)
* **splinter** — [SplinterTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/splinter#transformers.SplinterTokenizer) or [SplinterTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/splinter#transformers.SplinterTokenizerFast) (Splinter model)
* **squeezebert** — [SqueezeBertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/squeezebert#transformers.SqueezeBertTokenizer) or [SqueezeBertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/squeezebert#transformers.SqueezeBertTokenizerFast) (SqueezeBERT model)
* **switch\_transformers** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (SwitchTransformers model)
* **t5** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (T5 model)
* **tapas** — [TapasTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/tapas#transformers.TapasTokenizer) (TAPAS model)
* **tapex** — [TapexTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/tapex#transformers.TapexTokenizer) (TAPEX model)
* **transfo-xl** — [TransfoXLTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/transfo-xl#transformers.TransfoXLTokenizer) (Transformer-XL model)
* **umt5** — [T5Tokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5Tokenizer) or [T5TokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mt5#transformers.T5TokenizerFast) (UMT5 model)
* **vilt** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (ViLT model)
* **visual\_bert** — [BertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizer) or [BertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertTokenizerFast) (VisualBERT model)
* **vits** — [VitsTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/vits#transformers.VitsTokenizer) (VITS model)
* **wav2vec2** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Wav2Vec2 model)
* **wav2vec2-conformer** — [Wav2Vec2CTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2#transformers.Wav2Vec2CTCTokenizer) (Wav2Vec2-Conformer model)
* **wav2vec2\_phoneme** — [Wav2Vec2PhonemeCTCTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/wav2vec2_phoneme#transformers.Wav2Vec2PhonemeCTCTokenizer) (Wav2Vec2Phoneme model)
* **whisper** — [WhisperTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/whisper#transformers.WhisperTokenizer) or [WhisperTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/whisper#transformers.WhisperTokenizerFast) (Whisper model)
* **xclip** — [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizer) or [CLIPTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/clip#transformers.CLIPTokenizerFast) (X-CLIP model)
* **xglm** — [XGLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMTokenizer) or [XGLMTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMTokenizerFast) (XGLM model)
* **xlm** — [XLMTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm#transformers.XLMTokenizer) (XLM model)
* **xlm-prophetnet** — [XLMProphetNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-prophetnet#transformers.XLMProphetNetTokenizer) (XLM-ProphetNet model)
* **xlm-roberta** — [XLMRobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) or [XLMRobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast) (XLM-RoBERTa model)
* **xlm-roberta-xl** — [XLMRobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) or [XLMRobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast) (XLM-RoBERTa-XL model)
* **xlnet** — [XLNetTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetTokenizer) or [XLNetTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetTokenizerFast) (XLNet model)
* **xmod** — [XLMRobertaTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizer) or [XLMRobertaTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaTokenizerFast) (X-MOD model)
* **yoso** — [AlbertTokenizer](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizer) or [AlbertTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/albert#transformers.AlbertTokenizerFast) (YOSO model)

Examples:

Copied

```
>>> from transformers import AutoTokenizer

>>> # Download vocabulary from huggingface.co and cache.
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

>>> # Download vocabulary from huggingface.co (user-uploaded) and cache.
>>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased")

>>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*)
>>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/")

>>> # Download vocabulary from huggingface.co and define model-specific arguments
>>> tokenizer = AutoTokenizer.from_pretrained("roberta-base", add_prefix_space=True)
```

**register**

[\<source>](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/tokenization_auto.py#L784)

( config\_classslow\_tokenizer\_class = Nonefast\_tokenizer\_class = Noneexist\_ok = False )

Parameters

* **config\_class** ([PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig)) — The configuration corresponding to the model to register.
* **slow\_tokenizer\_class** (`PretrainedTokenizer`, *optional*) — The slow tokenizer to register.
* **fast\_tokenizer\_class** (`PretrainedTokenizerFast`, *optional*) — The fast tokenizer to register.

Register a new tokenizer in this mapping.