Encode Inputs

Encode Inputs

PythonRustNode

These types represent all the different kinds of input that a Tokenizer accepts when using encode_batch().

TextEncodeInput

tokenizers.TextEncodeInput

Represents a textual input for encoding. Can be either:

• A single sequence: TextInputSequence

• A pair of sequences:

  • A Tuple of TextInputSequence

  • Or a List of TextInputSequence of size 2

alias of Union[str, Tuple[str, str], List[str]].

PreTokenizedEncodeInput

tokenizers.PreTokenizedEncodeInput

Represents a pre-tokenized input for encoding. Can be either:

• A single sequence: PreTokenizedInputSequence

• A pair of sequences:

  • A Tuple of PreTokenizedInputSequence

  • Or a List of PreTokenizedInputSequence of size 2

alias of Union[List[str], Tuple[str], Tuple[Union[List[str], Tuple[str]], Union[List[str], Tuple[str]]], List[Union[List[str], Tuple[str]]]].

EncodeInput

tokenizers.EncodeInput

Represents all the possible types of input for encoding. Can be:

• When is_pretokenized=False: TextEncodeInput

• When is_pretokenized=True: PreTokenizedEncodeInput

alias of Union[str, Tuple[str, str], List[str], Tuple[str], Tuple[Union[List[str], Tuple[str]], Union[List[str], Tuple[str]]], List[Union[List[str], Tuple[str]]]].