Tokenization converts text into token IDs that a model can process. The tokenizer you choose determines vocabulary size, how many tokens a sentence becomes, and how well the system handles rare or new words. Most modern NLP pipelines use sub-word tokenization because it gives strong coverage without an enormous word list. If you are building projects in a generative AI course, tokenization is one of the first places where “small” design choices can change cost, latency, and output quality.
1) Why tokenization is a performance decision
A model stores an embedding vector for every token in its vocabulary. A bigger vocabulary increases parameters, memory, and often training time. At the same time, tokenization controls sequence length. If a tokenizer breaks text into many small pieces, the model sees more tokens per prompt, and attention computation increases. In transformers, attention cost grows quickly as sequences get longer, so token counts directly impact throughput.
Sub-word tokenization tackles the classic problem with word-level vocabularies: out-of-vocabulary words. Names, typos, new product terms, and domain jargon would be “unknown” if the tokenizer only knew full words. Sub-words let the model assemble unfamiliar terms from familiar pieces.
2) Sub-word tokenization in plain terms
Sub-word tokenizers build a vocabulary of frequent “pieces” that can be combined to represent any word. Common words become single tokens for efficiency. Rare words are split into multiple tokens. The goal is a workable compromise: enough vocabulary to keep sequences short, but not so much that the embedding and output layers become huge.
Two widely used methods are Byte Pair Encoding (BPE) and WordPiece. They start from small units (characters or bytes) and iteratively create larger units, but their merge rules differ.
3) BPE: frequency-based merging
BPE repeatedly merges the most frequent adjacent pair of symbols. Early merges create short fragments (like “ing” or “tion”). Later merges may create full words if those words are common. You control the final vocabulary by choosing how many merges to run, so BPE can be tuned for compactness or for shorter sequences.
What this means in practice:
- Strong coverage: any rare word can fall back to smaller pieces.
- Tunable vocabulary size: more merges usually reduce token counts but increase vocabulary.
- Robust variants: byte-level BPE begins from raw bytes, making it resilient to unusual characters and mixed-language text, though it may produce longer token sequences for certain scripts.
4) WordPiece: objective-based merging
WordPiece also builds sub-words, but it typically selects merges using a likelihood-based score rather than raw pair frequency. In effect, it prefers merges that improve how well the training text is “explained” by the current vocabulary. Many transformer encoder pipelines use WordPiece because it produces stable, reusable segments and handles morphology well.
Practical outcomes:
- Balanced splits for unfamiliar words (e.g., “bio”, “##chem”, “##istry” in some implementations).
- Strong generalisation across domains, because merges are guided by a scoring objective rather than frequency alone.
- Similar trade-offs to BPE: larger vocabularies shorten sequences, while smaller vocabularies increase sequence length.
5) Vocabulary size vs efficiency: the trade-off you must manage
Tokenization is a trade between memory and compute.
A larger vocabulary usually gives:
- Fewer tokens per sentence (shorter sequences).
- Less attention cost, which can reduce latency.
- More parameters in embeddings/output projections, raising memory use.
A smaller vocabulary usually gives:
- More tokens per sentence (longer sequences).
- Higher attention cost and potentially slower inference.
- More sharing of pieces across domains and languages, which can help robustness.
This is why token budgets matter in real workflows. In a generative AI course assignment using long documents, the tokenizer determines how much context fits into a fixed token window.
6) Practical ways to evaluate a tokenizer
Choose tokenization with measurements, not intuition. For teams applying lessons from a generative AI course, a quick evaluation avoids surprises:
- Compute average tokens per sentence on your target corpus (emails, code, support chats, product pages).
- Inspect long-tail inputs: URLs, IDs, names, and multilingual segments.
- Compare latency drivers: if you are attention-bound, reducing sequence length may matter more; if you are memory-bound, shrinking vocabulary may matter more.
- Watch for domain drift: if your terminology changes often, ensure the tokenizer handles new strings gracefully without exploding token counts.
Conclusion
Sub-word tokenization is a foundational design choice that shapes both model efficiency and vocabulary size. BPE leans on frequency-driven merges, while WordPiece uses an objective-driven score, but both aim to represent any text with a manageable vocabulary. Understanding these trade-offs helps you optimize cost, speed, and reliability—skills that transfer to production work and to any generative AI course you take.