> For the complete documentation index, see [llms.txt](https://boinc-ai.gitbook.io/transformers/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://boinc-ai.gitbook.io/transformers/api/main-classes/auto-classes/natural-language-processing/automodelforcausallm.md). # AutoModelForCausalLM #### AutoModelForCausalLM #### class transformers.AutoModelForCausalLM [\](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/modeling_auto.py#L1246) ( \*args\*\*kwargs ) This is a generic model class that will be instantiated as one of the model classes of the library (with a causal language modeling head) when created with the [from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.FlaxAutoModelForVision2Seq.from_pretrained) class method or the [from\_config()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.FlaxAutoModelForVision2Seq.from_config) class method. This class cannot be instantiated directly using `__init__()` (throws an error). **from\_config** [\](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/auto_factory.py#L417) ( \*\*kwargs ) Parameters * **config** ([PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig)) — The model class to instantiate is selected based on the configuration class: * [BartConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartConfig) configuration class: [BartForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartForCausalLM) (BART model) * [BertConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertConfig) configuration class: [BertLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertLMHeadModel) (BERT model) * [BertGenerationConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-generation#transformers.BertGenerationConfig) configuration class: [BertGenerationDecoder](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-generation#transformers.BertGenerationDecoder) (Bert Generation model) * [BigBirdConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdConfig) configuration class: [BigBirdForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdForCausalLM) (BigBird model) * [BigBirdPegasusConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bigbird_pegasus#transformers.BigBirdPegasusConfig) configuration class: [BigBirdPegasusForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bigbird_pegasus#transformers.BigBirdPegasusForCausalLM) (BigBird-Pegasus model) * [BioGptConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/biogpt#transformers.BioGptConfig) configuration class: [BioGptForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/biogpt#transformers.BioGptForCausalLM) (BioGpt model) * [BlenderbotConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotConfig) configuration class: [BlenderbotForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotForCausalLM) (Blenderbot model) * [BlenderbotSmallConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot-small#transformers.BlenderbotSmallConfig) configuration class: [BlenderbotSmallForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot-small#transformers.BlenderbotSmallForCausalLM) (BlenderbotSmall model) * [BloomConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bloom#transformers.BloomConfig) configuration class: [BloomForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bloom#transformers.BloomForCausalLM) (BLOOM model) * [CTRLConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ctrl#transformers.CTRLConfig) configuration class: [CTRLLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ctrl#transformers.CTRLLMHeadModel) (CTRL model) * [CamembertConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertConfig) configuration class: [CamembertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertForCausalLM) (CamemBERT model) * [CodeGenConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenConfig) configuration class: [CodeGenForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenForCausalLM) (CodeGen model) * [CpmAntConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpmant#transformers.CpmAntConfig) configuration class: [CpmAntForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpmant#transformers.CpmAntForCausalLM) (CPM-Ant model) * [Data2VecTextConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/data2vec#transformers.Data2VecTextConfig) configuration class: [Data2VecTextForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/data2vec#transformers.Data2VecTextForCausalLM) (Data2VecText model) * [ElectraConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraConfig) configuration class: [ElectraForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraForCausalLM) (ELECTRA model) * [ErnieConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ernie#transformers.ErnieConfig) configuration class: [ErnieForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ernie#transformers.ErnieForCausalLM) (ERNIE model) * [FalconConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/falcon#transformers.FalconConfig) configuration class: [FalconForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/falcon#transformers.FalconForCausalLM) (Falcon model) * [GPT2Config](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2Config) configuration class: [GPT2LMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2LMHeadModel) (OpenAI GPT-2 model) * [GPTBigCodeConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_bigcode#transformers.GPTBigCodeConfig) configuration class: [GPTBigCodeForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_bigcode#transformers.GPTBigCodeForCausalLM) (GPTBigCode model) * [GPTJConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gptj#transformers.GPTJConfig) configuration class: [GPTJForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gptj#transformers.GPTJForCausalLM) (GPT-J model) * [GPTNeoConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neo#transformers.GPTNeoConfig) configuration class: [GPTNeoForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neo#transformers.GPTNeoForCausalLM) (GPT Neo model) * [GPTNeoXConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXConfig) configuration class: [GPTNeoXForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXForCausalLM) (GPT NeoX model) * [GPTNeoXJapaneseConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox_japanese#transformers.GPTNeoXJapaneseConfig) configuration class: [GPTNeoXJapaneseForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox_japanese#transformers.GPTNeoXJapaneseForCausalLM) (GPT NeoX Japanese model) * [GitConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/git#transformers.GitConfig) configuration class: [GitForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/git#transformers.GitForCausalLM) (GIT model) * [LlamaConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaConfig) configuration class: [LlamaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaForCausalLM) (LLaMA model) * [MBartConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartConfig) configuration class: [MBartForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartForCausalLM) (mBART model) * [MarianConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/marian#transformers.MarianConfig) configuration class: [MarianForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/marian#transformers.MarianForCausalLM) (Marian model) * [MegaConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mega#transformers.MegaConfig) configuration class: [MegaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mega#transformers.MegaForCausalLM) (MEGA model) * [MegatronBertConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/megatron-bert#transformers.MegatronBertConfig) configuration class: [MegatronBertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/megatron-bert#transformers.MegatronBertForCausalLM) (Megatron-BERT model) * [MistralConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mistral#transformers.MistralConfig) configuration class: [MistralForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mistral#transformers.MistralForCausalLM) (Mistral model) * [MptConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpt#transformers.MptConfig) configuration class: [MptForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpt#transformers.MptForCausalLM) (MPT model) * [MusicgenConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/musicgen#transformers.MusicgenConfig) configuration class: [MusicgenForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/musicgen#transformers.MusicgenForCausalLM) (MusicGen model) * [MvpConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpConfig) configuration class: [MvpForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpForCausalLM) (MVP model) * [OPTConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/opt#transformers.OPTConfig) configuration class: [OPTForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/opt#transformers.OPTForCausalLM) (OPT model) * [OpenAIGPTConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTConfig) configuration class: [OpenAIGPTLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTLMHeadModel) (OpenAI GPT model) * [OpenLlamaConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/open-llama#transformers.OpenLlamaConfig) configuration class: [OpenLlamaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/open-llama#transformers.OpenLlamaForCausalLM) (OpenLlama model) * [PLBartConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/plbart#transformers.PLBartConfig) configuration class: [PLBartForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/plbart#transformers.PLBartForCausalLM) (PLBart model) * [PegasusConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusConfig) configuration class: [PegasusForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusForCausalLM) (Pegasus model) * [PersimmonConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/persimmon#transformers.PersimmonConfig) configuration class: [PersimmonForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/persimmon#transformers.PersimmonForCausalLM) (Persimmon model) * [ProphetNetConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/prophetnet#transformers.ProphetNetConfig) configuration class: [ProphetNetForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/prophetnet#transformers.ProphetNetForCausalLM) (ProphetNet model) * [QDQBertConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/qdqbert#transformers.QDQBertConfig) configuration class: [QDQBertLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/qdqbert#transformers.QDQBertLMHeadModel) (QDQBert model) * [ReformerConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerConfig) configuration class: [ReformerModelWithLMHead](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerModelWithLMHead) (Reformer model) * [RemBertConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertConfig) configuration class: [RemBertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertForCausalLM) (RemBERT model) * [RoCBertConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roc_bert#transformers.RoCBertConfig) configuration class: [RoCBertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roc_bert#transformers.RoCBertForCausalLM) (RoCBert model) * [RoFormerConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerConfig) configuration class: [RoFormerForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerForCausalLM) (RoFormer model) * [RobertaConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaConfig) configuration class: [RobertaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaForCausalLM) (RoBERTa model) * [RobertaPreLayerNormConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta-prelayernorm#transformers.RobertaPreLayerNormConfig) configuration class: [RobertaPreLayerNormForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta-prelayernorm#transformers.RobertaPreLayerNormForCausalLM) (RoBERTa-PreLayerNorm model) * [RwkvConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rwkv#transformers.RwkvConfig) configuration class: [RwkvForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rwkv#transformers.RwkvForCausalLM) (RWKV model) * [Speech2Text2Config](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text_2#transformers.Speech2Text2Config) configuration class: [Speech2Text2ForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text_2#transformers.Speech2Text2ForCausalLM) (Speech2Text2 model) * [TrOCRConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/trocr#transformers.TrOCRConfig) configuration class: [TrOCRForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/trocr#transformers.TrOCRForCausalLM) (TrOCR model) * [TransfoXLConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/transfo-xl#transformers.TransfoXLConfig) configuration class: [TransfoXLLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/transfo-xl#transformers.TransfoXLLMHeadModel) (Transformer-XL model) * [XGLMConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMConfig) configuration class: [XGLMForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMForCausalLM) (XGLM model) * [XLMConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm#transformers.XLMConfig) configuration class: [XLMWithLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm#transformers.XLMWithLMHeadModel) (XLM model) * [XLMProphetNetConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-prophetnet#transformers.XLMProphetNetConfig) configuration class: [XLMProphetNetForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-prophetnet#transformers.XLMProphetNetForCausalLM) (XLM-ProphetNet model) * [XLMRobertaConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaConfig) configuration class: [XLMRobertaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaForCausalLM) (XLM-RoBERTa model) * [XLMRobertaXLConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta-xl#transformers.XLMRobertaXLConfig) configuration class: [XLMRobertaXLForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta-xl#transformers.XLMRobertaXLForCausalLM) (XLM-RoBERTa-XL model) * [XLNetConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetConfig) configuration class: [XLNetLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetLMHeadModel) (XLNet model) * [XmodConfig](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xmod#transformers.XmodConfig) configuration class: [XmodForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xmod#transformers.XmodForCausalLM) (X-MOD model) Instantiates one of the model classes of the library (with a causal language modeling head) from a configuration. Note: Loading a model from its configuration file does **not** load the model weights. It only affects the model’s configuration. Use [from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/auto#transformers.FlaxAutoModelForVision2Seq.from_pretrained) to load the model weights. Examples: Copied ``` >>> from transformers import AutoConfig, AutoModelForCausalLM >>> # Download configuration from huggingface.co and cache. >>> config = AutoConfig.from_pretrained("bert-base-cased") >>> model = AutoModelForCausalLM.from_config(config) ``` **from\_pretrained** [\](https://github.com/huggingface/transformers/blob/v4.34.1/src/transformers/models/auto/auto_factory.py#L448) ( \*model\_args\*\*kwargs ) Parameters * **pretrained\_model\_name\_or\_path** (`str` or `os.PathLike`) — Can be either: * A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. * A path to a *directory* containing model weights saved using [save\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/model#transformers.PreTrainedModel.save_pretrained), e.g., `./my_model_directory/`. * A path or url to a *tensorflow index checkpoint file* (e.g, `./tf_model/model.ckpt.index`). In this case, `from_tf` should be set to `True` and a configuration object should be provided as `config` argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. * **model\_args** (additional positional arguments, *optional*) — Will be passed along to the underlying model `__init__()` method. * **config** ([PretrainedConfig](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig), *optional*) — Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: * The model is a model provided by the library (loaded with the *model id* string of a pretrained model). * The model was saved using [save\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/model#transformers.PreTrainedModel.save_pretrained) and is reloaded by supplying the save directory. * The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. * **state\_dict** (*Dict\[str, torch.Tensor]*, *optional*) — A state dictionary to use instead of a state dictionary loaded from saved weights file. This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [save\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/model#transformers.PreTrainedModel.save_pretrained) and [from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) is not a simpler option. * **cache\_dir** (`str` or `os.PathLike`, *optional*) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. * **from\_tf** (`bool`, *optional*, defaults to `False`) — Load the model weights from a TensorFlow checkpoint save file (see docstring of `pretrained_model_name_or_path` argument). * **force\_download** (`bool`, *optional*, defaults to `False`) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. * **resume\_download** (`bool`, *optional*, defaults to `False`) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. * **proxies** (`Dict[str, str]`, *optional*) — A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. * **output\_loading\_info(`bool`,** *optional*, defaults to `False`) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. * **local\_files\_only(`bool`,** *optional*, defaults to `False`) — Whether or not to only look at local files (e.g., not try downloading the model). * **revision** (`str`, *optional*, defaults to `"main"`) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. * **trust\_remote\_code** (`bool`, *optional*, defaults to `False`) — Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. * **code\_revision** (`str`, *optional*, defaults to `"main"`) — The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. * **kwargs** (additional keyword arguments, *optional*) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: * If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model’s `__init__` method (we assume all relevant updates to the configuration have already been done) * If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([from\_pretrained()](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained)). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s `__init__` function. Instantiate one of the model classes of the library (with a causal language modeling head) from a pretrained model. The model class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it’s missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: * **bart** — [BartForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bart#transformers.BartForCausalLM) (BART model) * **bert** — [BertLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert#transformers.BertLMHeadModel) (BERT model) * **bert-generation** — [BertGenerationDecoder](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bert-generation#transformers.BertGenerationDecoder) (Bert Generation model) * **big\_bird** — [BigBirdForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/big_bird#transformers.BigBirdForCausalLM) (BigBird model) * **bigbird\_pegasus** — [BigBirdPegasusForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bigbird_pegasus#transformers.BigBirdPegasusForCausalLM) (BigBird-Pegasus model) * **biogpt** — [BioGptForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/biogpt#transformers.BioGptForCausalLM) (BioGpt model) * **blenderbot** — [BlenderbotForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot#transformers.BlenderbotForCausalLM) (Blenderbot model) * **blenderbot-small** — [BlenderbotSmallForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/blenderbot-small#transformers.BlenderbotSmallForCausalLM) (BlenderbotSmall model) * **bloom** — [BloomForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/bloom#transformers.BloomForCausalLM) (BLOOM model) * **camembert** — [CamembertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/camembert#transformers.CamembertForCausalLM) (CamemBERT model) * **code\_llama** — [LlamaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaForCausalLM) (CodeLlama model) * **codegen** — [CodeGenForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/codegen#transformers.CodeGenForCausalLM) (CodeGen model) * **cpmant** — [CpmAntForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/cpmant#transformers.CpmAntForCausalLM) (CPM-Ant model) * **ctrl** — [CTRLLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ctrl#transformers.CTRLLMHeadModel) (CTRL model) * **data2vec-text** — [Data2VecTextForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/data2vec#transformers.Data2VecTextForCausalLM) (Data2VecText model) * **electra** — [ElectraForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/electra#transformers.ElectraForCausalLM) (ELECTRA model) * **ernie** — [ErnieForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/ernie#transformers.ErnieForCausalLM) (ERNIE model) * **falcon** — [FalconForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/falcon#transformers.FalconForCausalLM) (Falcon model) * **git** — [GitForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/git#transformers.GitForCausalLM) (GIT model) * **gpt-sw3** — [GPT2LMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2LMHeadModel) (GPT-Sw3 model) * **gpt2** — [GPT2LMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt2#transformers.GPT2LMHeadModel) (OpenAI GPT-2 model) * **gpt\_bigcode** — [GPTBigCodeForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_bigcode#transformers.GPTBigCodeForCausalLM) (GPTBigCode model) * **gpt\_neo** — [GPTNeoForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neo#transformers.GPTNeoForCausalLM) (GPT Neo model) * **gpt\_neox** — [GPTNeoXForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox#transformers.GPTNeoXForCausalLM) (GPT NeoX model) * **gpt\_neox\_japanese** — [GPTNeoXJapaneseForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gpt_neox_japanese#transformers.GPTNeoXJapaneseForCausalLM) (GPT NeoX Japanese model) * **gptj** — [GPTJForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/gptj#transformers.GPTJForCausalLM) (GPT-J model) * **llama** — [LlamaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/llama2#transformers.LlamaForCausalLM) (LLaMA model) * **marian** — [MarianForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/marian#transformers.MarianForCausalLM) (Marian model) * **mbart** — [MBartForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mbart#transformers.MBartForCausalLM) (mBART model) * **mega** — [MegaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mega#transformers.MegaForCausalLM) (MEGA model) * **megatron-bert** — [MegatronBertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/megatron-bert#transformers.MegatronBertForCausalLM) (Megatron-BERT model) * **mistral** — [MistralForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mistral#transformers.MistralForCausalLM) (Mistral model) * **mpt** — [MptForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mpt#transformers.MptForCausalLM) (MPT model) * **musicgen** — [MusicgenForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/musicgen#transformers.MusicgenForCausalLM) (MusicGen model) * **mvp** — [MvpForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/mvp#transformers.MvpForCausalLM) (MVP model) * **open-llama** — [OpenLlamaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/open-llama#transformers.OpenLlamaForCausalLM) (OpenLlama model) * **openai-gpt** — [OpenAIGPTLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/openai-gpt#transformers.OpenAIGPTLMHeadModel) (OpenAI GPT model) * **opt** — [OPTForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/opt#transformers.OPTForCausalLM) (OPT model) * **pegasus** — [PegasusForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/pegasus#transformers.PegasusForCausalLM) (Pegasus model) * **persimmon** — [PersimmonForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/persimmon#transformers.PersimmonForCausalLM) (Persimmon model) * **plbart** — [PLBartForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/plbart#transformers.PLBartForCausalLM) (PLBart model) * **prophetnet** — [ProphetNetForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/prophetnet#transformers.ProphetNetForCausalLM) (ProphetNet model) * **qdqbert** — [QDQBertLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/qdqbert#transformers.QDQBertLMHeadModel) (QDQBert model) * **reformer** — [ReformerModelWithLMHead](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/reformer#transformers.ReformerModelWithLMHead) (Reformer model) * **rembert** — [RemBertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rembert#transformers.RemBertForCausalLM) (RemBERT model) * **roberta** — [RobertaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta#transformers.RobertaForCausalLM) (RoBERTa model) * **roberta-prelayernorm** — [RobertaPreLayerNormForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roberta-prelayernorm#transformers.RobertaPreLayerNormForCausalLM) (RoBERTa-PreLayerNorm model) * **roc\_bert** — [RoCBertForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roc_bert#transformers.RoCBertForCausalLM) (RoCBert model) * **roformer** — [RoFormerForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/roformer#transformers.RoFormerForCausalLM) (RoFormer model) * **rwkv** — [RwkvForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/rwkv#transformers.RwkvForCausalLM) (RWKV model) * **speech\_to\_text\_2** — [Speech2Text2ForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/speech_to_text_2#transformers.Speech2Text2ForCausalLM) (Speech2Text2 model) * **transfo-xl** — [TransfoXLLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/transfo-xl#transformers.TransfoXLLMHeadModel) (Transformer-XL model) * **trocr** — [TrOCRForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/trocr#transformers.TrOCRForCausalLM) (TrOCR model) * **xglm** — [XGLMForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xglm#transformers.XGLMForCausalLM) (XGLM model) * **xlm** — [XLMWithLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm#transformers.XLMWithLMHeadModel) (XLM model) * **xlm-prophetnet** — [XLMProphetNetForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-prophetnet#transformers.XLMProphetNetForCausalLM) (XLM-ProphetNet model) * **xlm-roberta** — [XLMRobertaForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta#transformers.XLMRobertaForCausalLM) (XLM-RoBERTa model) * **xlm-roberta-xl** — [XLMRobertaXLForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlm-roberta-xl#transformers.XLMRobertaXLForCausalLM) (XLM-RoBERTa-XL model) * **xlnet** — [XLNetLMHeadModel](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xlnet#transformers.XLNetLMHeadModel) (XLNet model) * **xmod** — [XmodForCausalLM](https://huggingface.co/docs/transformers/v4.34.1/en/model_doc/xmod#transformers.XmodForCausalLM) (X-MOD model) The model is set in evaluation mode by default using `model.eval()` (so for instance, dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()` Examples: Copied ``` >>> from transformers import AutoConfig, AutoModelForCausalLM >>> # Download model and configuration from huggingface.co and cache. >>> model = AutoModelForCausalLM.from_pretrained("bert-base-cased") >>> # Update configuration during loading >>> model = AutoModelForCausalLM.from_pretrained("bert-base-cased", output_attentions=True) >>> model.config.output_attentions True >>> # Loading from a TF checkpoint file instead of a PyTorch model (slower) >>> config = AutoConfig.from_pretrained("./tf_model/bert_tf_model_config.json") >>> model = AutoModelForCausalLM.from_pretrained( ... "./tf_model/bert_tf_checkpoint.ckpt.index", from_tf=True, config=config ... ) ```