Visualizer

Visualizer

PythonRustNode

Annotation

class tokenizers.tools.Annotation

<source>

( start: intend: intlabel: str )

EncodingVisualizer

class tokenizers.tools.EncodingVisualizer

<source>

( tokenizer: Tokenizerdefault_to_notebook: bool = Trueannotation_converter: typing.Union[typing.Callable[[typing.Any], tokenizers.tools.visualizer.Annotation], NoneType] = None )

Parameters

• tokenizer (Tokenizer) — A tokenizer instance

• default_to_notebook (bool) — Whether to render html output in a notebook by default

• annotation_converter (Callable, optional) — An optional (lambda) function that takes an annotation in any format and returns an Annotation object

Build an EncodingVisualizer

__call__

<source>

( text: strannotations: typing.List[tokenizers.tools.visualizer.Annotation] = []default_to_notebook: typing.Optional[bool] = None )

Parameters

• text (str) — The text to tokenize

• annotations (List[Annotation], optional) — An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function

• default_to_notebook (bool, optional, defaults to False) — If True, will render the html in a notebook. Otherwise returns an html string.

Build a visualization of the given text