Utilities for Generation

This page lists all the utility functions used by generate(), greedy_search(), contrastive_search(), sample(), beam_search(), beam_sample(), group_beam_search(), and constrained_beam_search().

Most of those are only useful if you are studying the code of the generate methods in the library.

Generate Outputs

The output of generate() is an instance of a subclass of ModelOutput. This output is a data structure containing all the information returned by generate(), but that can also be used as tuple or dictionary.

Here’s an example:

Copied

from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

inputs = tokenizer("Hello, my dog is cute and ", return_tensors="pt")
generation_output = model.generate(**inputs, return_dict_in_generate=True, output_scores=True)

The generation_output object is a GreedySearchDecoderOnlyOutput, as we can see in the documentation of that class below, it means it has the following attributes:

sequences: the generated sequences of tokens
scores (optional): the prediction scores of the language modelling head, for each generation step
hidden_states (optional): the hidden states of the model, for each generation step
attentions (optional): the attention weights of the model, for each generation step

Here we have the scores since we passed along output_scores=True, but we don’t have hidden_states and attentions because we didn’t pass output_hidden_states=True or output_attentions=True.

You can access each attribute as you would usually do, and if that attribute has not been returned by the model, you will get None. Here for instance generation_output.scores are all the generated prediction scores of the language modeling head, and generation_output.attentions is None.

When using our generation_output object as a tuple, it only keeps the attributes that don’t have None values. Here, for instance, it has two elements, loss then logits, so

Copied

generation_output[:2]

will return the tuple (generation_output.sequences, generation_output.scores) for instance.

When using our generation_output object as a dictionary, it only keeps the attributes that don’t have None values. Here, for instance, it has two keys that are sequences and scores.

We document here all output types.

Utilities for Generation

Generate Outputs

PyTorch

class transformers.generation.GreedySearchEncoderDecoderOutput

class transformers.generation.GreedySearchDecoderOnlyOutput

class transformers.generation.SampleEncoderDecoderOutput

class transformers.generation.SampleDecoderOnlyOutput

class transformers.generation.BeamSearchEncoderDecoderOutput

class transformers.generation.BeamSearchDecoderOnlyOutput

class transformers.generation.BeamSampleEncoderDecoderOutput

class transformers.generation.BeamSampleDecoderOnlyOutput

class transformers.generation.ContrastiveSearchEncoderDecoderOutput

class transformers.generation.ContrastiveSearchDecoderOnlyOutput

TensorFlow

class transformers.generation.TFGreedySearchEncoderDecoderOutput

class transformers.generation.TFGreedySearchDecoderOnlyOutput

class transformers.generation.TFSampleEncoderDecoderOutput

class transformers.generation.TFSampleDecoderOnlyOutput

class transformers.generation.TFBeamSearchEncoderDecoderOutput

class transformers.generation.TFBeamSearchDecoderOnlyOutput

class transformers.generation.TFBeamSampleEncoderDecoderOutput

class transformers.generation.TFBeamSampleDecoderOnlyOutput

class transformers.generation.TFContrastiveSearchEncoderDecoderOutput

class transformers.generation.TFContrastiveSearchDecoderOnlyOutput

FLAX

class transformers.generation.FlaxSampleOutput

class transformers.generation.FlaxGreedySearchOutput

class transformers.generation.FlaxBeamSearchOutput

LogitsProcessor

PyTorch

class transformers.AlternatingCodebooksLogitsProcessor

class transformers.ClassifierFreeGuidanceLogitsProcessor

class transformers.EncoderNoRepeatNGramLogitsProcessor

class transformers.EncoderRepetitionPenaltyLogitsProcessor

class transformers.EpsilonLogitsWarper

class transformers.EtaLogitsWarper

class transformers.ExponentialDecayLengthPenalty

class transformers.ForcedBOSTokenLogitsProcessor

class transformers.ForcedEOSTokenLogitsProcessor

class transformers.ForceTokensLogitsProcessor

class transformers.HammingDiversityLogitsProcessor

class transformers.InfNanRemoveLogitsProcessor

class transformers.LogitNormalization

class transformers.LogitsProcessor

class transformers.LogitsProcessorList

class transformers.LogitsWarper

class transformers.MinLengthLogitsProcessor

class transformers.MinNewTokensLengthLogitsProcessor

class transformers.NoBadWordsLogitsProcessor

class transformers.NoRepeatNGramLogitsProcessor

class transformers.PrefixConstrainedLogitsProcessor

class transformers.RepetitionPenaltyLogitsProcessor

class transformers.SequenceBiasLogitsProcessor

class transformers.SuppressTokensAtBeginLogitsProcessor

class transformers.SuppressTokensLogitsProcessor

class transformers.TemperatureLogitsWarper

class transformers.TopKLogitsWarper

class transformers.TopPLogitsWarper

class transformers.TypicalLogitsWarper

class transformers.UnbatchedClassifierFreeGuidanceLogitsProcessor

class transformers.WhisperTimeStampLogitsProcessor

TensorFlow

class transformers.TFForcedBOSTokenLogitsProcessor

class transformers.TFForcedEOSTokenLogitsProcessor

class transformers.TFForceTokensLogitsProcessor

class transformers.TFLogitsProcessor

class transformers.TFLogitsProcessorList

class transformers.TFLogitsWarper

class transformers.TFMinLengthLogitsProcessor

class transformers.TFNoBadWordsLogitsProcessor

class transformers.TFNoRepeatNGramLogitsProcessor

class transformers.TFRepetitionPenaltyLogitsProcessor

class transformers.TFSuppressTokensAtBeginLogitsProcessor

class transformers.TFSuppressTokensLogitsProcessor

class transformers.TFTemperatureLogitsWarper

class transformers.TFTopKLogitsWarper

class transformers.TFTopPLogitsWarper

FLAX

class transformers.FlaxForcedBOSTokenLogitsProcessor

class transformers.FlaxForcedEOSTokenLogitsProcessor