# Use fast tokenizers from BOINC AI Tokenizers

The [PreTrainedTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) depends on the 🌎[ Tokenizers](https://huggingface.co/docs/tokenizers) library. The tokenizers obtained from the 🌎 Tokenizers library can be loaded very simply into 🌎 Transformers.

Before getting in the specifics, let’s first start by creating a dummy tokenizer in a few lines:

Copied

```
>>> from tokenizers import Tokenizer
>>> from tokenizers.models import BPE
>>> from tokenizers.trainers import BpeTrainer
>>> from tokenizers.pre_tokenizers import Whitespace

>>> tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
>>> trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])

>>> tokenizer.pre_tokenizer = Whitespace()
>>> files = [...]
>>> tokenizer.train(files, trainer)
```

We now have a tokenizer trained on the files we defined. We can either continue using it in that runtime, or save it to a JSON file for future re-use.

### Loading directly from the tokenizer object

Let’s see how to leverage this tokenizer object in the 🌎 Transformers library. The [PreTrainedTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) class allows for easy instantiation, by accepting the instantiated *tokenizer* object as an argument:

Copied

```
>>> from transformers import PreTrainedTokenizerFast

>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
```

This object can now be used with all the methods shared by the 🌎 Transformers tokenizers! Head to [the tokenizer page](https://huggingface.co/docs/transformers/main_classes/tokenizer) for more information.

### Loading from a JSON file

In order to load a tokenizer from a JSON file, let’s first start by saving our tokenizer:

Copied

```
>>> tokenizer.save("tokenizer.json")
```

The path to which we saved this file can be passed to the [PreTrainedTokenizerFast](https://huggingface.co/docs/transformers/v4.34.1/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) initialization method using the `tokenizer_file` parameter:

Copied

```
>>> from transformers import PreTrainedTokenizerFast

>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
```

This object can now be used with all the methods shared by the 🌎 Transformers tokenizers! Head to [the tokenizer page](https://huggingface.co/docs/transformers/main_classes/tokenizer) for more information.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://boinc-ai.gitbook.io/transformers/developer-guides/use-fast-tokenizers-from-boinc-ai-tokenizers.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.