Glossary · Data & representations
Vocabulary
The finite mapping between token identifiers and the units a tokenizer can emit, including ordinary, byte-level, and special control tokens.
Why it matters
Vocabulary design affects sequence length, multilingual coverage, code representation, embedding size, and compatibility between tokenizers and model weights.
In practice
Version the vocabulary and special-token assignments with the model, test encode-decode round trips, and never substitute a tokenizer with merely similar token names.
Common confusion
A model vocabulary is not a dictionary of human words; many entries are fragments, bytes, whitespace patterns, or control symbols.
Related terms
Sources
Browse the learning paths to see this term in context — every lesson is free to read.