Recent findings suggest that transformers v5 redesigns how tokenizers work. The massive tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can inspect, customize, and train from scratch with far less friction.
TL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. Itβs a practical guide for anyone who wants to understand, customize, or train model-specific tokenizers instead of treating them as black boxes.
For experts: If you're already familiar with the concepts and want to understand the changes in v5, go to v5 Separates Tokenizer Architecture from Trained Vocab.
Before diving into the changes, let's swiftly cover what tokenization does and how the pieces fit together.
Recent findings suggest that language models don't read raw text. They consume sequences of integers usually called token IDs or input IDs. Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground here to visualize tokenization.).
Tokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the transformers and tokenizers libraries.
Δ world (above) is a single token that represents the character sequence " world" (with the space).
According to recent reports, A token is the smallest string unit the model sees. It can be a character, word, or subword chunk like "play" or "##ing" ("##" is a pattern, don't worry if you don't completely understand it now π€). The vocabulary maps each unique token to the token ID.
A good tokenizer compresses text into the smallest amount of tokens. Fewer tokens means more usable context without increasing model size. Training a tokenizer boils down to finding the best compression rules for your datasets. For example, if you work on Chinese corpus you can sometimes find very nice surprises π.
Industry observers note that tokenization happens in stages. Each stage transforms text before passing it to the next:.
Each component is independent. You can swap normalizers or change the algorithm without rewriting everything else.
You can access the rust based tokenizer through _tokenizer. We go in more depth about it in this section.
The following algorithms dominate modern language model tokenizers:.
The tokenizers library is a Rust-based tokenization engine. It is fast, efficient, and completely language model agnostic. The library handles the mechanics of converting text into token IDs and back. The tokenizers library is a general-purpose tool that implements the tokenization algorithms, but does not implement the conventions that connect those algorithms to specific language models.
Consider what happens when you use tokenizers directly with the SmolLM3-3B model:.
In a recent development, the output is raw tokenization. You get token IDs and the string pieces they correspond to. Nothing more.
It has been revealed that now consider what's missing. The SmolLM3-3B is a conversational model. When you interact with it, you typically structure your input as a conversation with roles like "user" and "assistant". The language model expects special formatting tokens to indicate these roles. The raw tokenizers library has no concept of any of this.
The transformers library bridges this gap. The library is primarily known as a model definition library, but it also provides a tokenizer abstraction layer that wraps the raw tokenizers backend and adds model-aware functionality.
Here's the same tokenization with the transformers wrapper:.
Notice how the special tokens like <|im_start|> and <|im_end|> are applied to the prompt before tokenizing. This is useful for the model to learn where a newly introduced sequence starts and ends.
The transformers tokenizer adds everything the raw library lacks:.
transformers implements the tokenization API that is most commonly used in the entire ML community (encode, decode, convert_tokens_to_ids, etc.).
It has been revealed that the transformers library organizes tokenizers into a class hierarchy. At the top sits a base class that defines the common interface. Below it, backend classes handle the actual tokenization using different engines. At the bottom, model-specific classes configure the backends for particular models.
PreTrainedTokenizerBase is the abstract base class for all tokenizers in transformers. It defines the interface that every tokenizer must implement.
The base class handles functionality that doesn't depend on the tokenization backend:.
Sources indicate that every tokenizer in transformers ultimately inherits from PreTrainedTokenizerBase. The base class ensures consistent behavior across all tokenizers, regardless of which backend they use for the actual tokenization.
TokenizersBackend is the primary backend class for most modern tokenizers. It inherits from PreTrainedTokenizerBase and wraps the Rust-based tokenizers library.
The class stores the Rust tokenizer object internally:.
When you call encoding methods on a TokenizersBackend tokenizer, the class delegates the actual tokenization to the Rust backend:.
The Rust backend performs computationally intensive work, while the Python wrapper adds the model-aware features on top.
Many model-specific tokenizers inherit from TokenizersBackend, examples include:.
These model-specific classes configure the backend with the correct vocabulary, merge rules, special tokens, and normalization settings for their respective models.
Industry observers note that pythonBackend inherits from PreTrainedTokenizerBase and implements tokenization in pure Python. The class is aliased as PreTrainedTokenizer.
The pure-Python backend exists for several reasons:.
The Python backend is slower than the Rust backend. For most use cases, the Rust-backed TokenizersBackend is preferred.
Model-specific tokenizers that inherit from PythonBackend (or its alias PreTrainedTokenizer) include some older or specialized models, like:.
According to recent reports, sentencePieceBackend inherits from PythonBackend and provides integration with Google's SentencePiece library. SentencePiece is a standalone tokenization library that many models use, particularly those trained by Google.
The backend wraps a SentencePiece processor:.
In a recent development, models that use SentencePiece tokenization inherit from this backend. Examples include:.
The SentencePiece backend inherits from PythonBackend rather than directly from PreTrainedTokenizerBase because it shares much of the same interface and padding/truncation logic.
Industry observers note that autoTokenizer is the recommended entry point for loading tokenizers. It automatically determines which tokenizer class to use for a given model and returns an instance of that class.
Behind the scenes, AutoTokenizer performs these steps:.
Industry observers note that the benefit of AutoTokenizer is that you don't need to know which tokenizer class a model uses. Whether a model uses LlamaTokenizer, GPT2Tokenizer, or BertTokenizer, the same AutoTokenizer.from_pretrained("model-name") call works.
The tokenizer system in transformers forms a layered architecture:.
Recent findings suggest that the most significant change in Transformers v5 is a philosophical shift in how tokenizers are defined. Tokenizers now work like PyTorch's nn.Module: you define the architecture first, then fill it with learned parameters.
It has been revealed that in v4, tokenizers were black boxes tied to pretrained checkpoint files. If you loaded LlamaTokenizerFast, you couldn't easily answer basic questions about it:.
The __init__ method gave no clues. You had to dig through serialized files or external documentation to understand what the tokenizer actually did.
v4 also maintained two parallel implementations for every model:.
Worst of all, you couldn't create an empty tokenizer architecture. If you wanted to train a LLaMA-style tokenizer on your own data, there was no clean way to instantiate a "blank" LLaMA tokenizer and fill it with your vocabulary and merges. Tokenizers existed only as loaded checkpoints, not as configurable templates.
v5 treats tokenizer architecture (normalizer, pre-tokenizer, model type, post-processor, decoder) as distinct from trained parameters (vocabulary, merges). This mirrors how PyTorch separates model architecture from learned weights.
With nn.Module, you define layers first:.
V5 tokenizers follow the same pattern:.
In a recent development, the tokenizer class now explicitly declares its structure. Looking at LlamaTokenizer in v5, you can immediately see:.
This transparency was impossible in v4, where the same information was buried in serialized files.
Industry observers note that v5 consolidates the two-file system into a single file per model. LlamaTokenizer now inherits from TokenizersBackend, which wraps the Rust-based tokenizer that was previously exposed as the βfastβ implementation and is now the default.
The former βslowβ Python implementation lives explicitly behind PythonBackend, and SentencePieceBackend remains for models that require it, but Rust-backed tokenization is the preferred default.
Owners now have one clear entry point. Advanced users who need to customize can still access lower-level components, but the library no longer forces everyone to navigate two parallel implementations.
Recent findings suggest that suppose you want a tokenizer that behaves exactly like LLaMA's β same normalization, same pre-tokenization, same BPE model type β but trained on a domain-specific corpus (medical text, legal documents, a newly introduced language). In v4, this required manually reconstructing the tokenizer pipeline from low-level tokenizers library primitives. In v5, you can instantiate the architecture directly and call train:.
The resulting tokenizer will have your custom vocabulary and merge rules, but will process text identically to how a standard LLaMA tokenizer would with the same whitespace handling, same special token conventions, same decoding behavior.
The shift from "tokenizers as loaded checkpoints" to "tokenizers as configurable architectures" makes the library more modular, more transparent, and more aligned with how practitioners think about building ML systems.
Transformers v5 brings three improvements to tokenization:.
According to recent reports, the wrapper layer between tokenizers and Transformers remains essential. It adds model awareness, context lengths, chat templates, special tokens, that raw tokenization doesn't provide. V5 just makes that layer clearer and more customizable.
If you are looking to learn more about tokenization here are some resources:.
I think it depends on the nature of tools.
What depends on the nature of tools? π³.
It has been revealed that thanks for doing this! I had to train some tokenizers with the v4, it was indeed not straightforward to understand the behavior.
Glad that this was useful to you.