Glossary · Data & representations
Token
An integer identifier produced by a model-specific tokenizer from text, bytes, images, audio, or another input representation. A token can be a whole word, part of a word, punctuation, whitespace, a byte sequence, or a special control symbol.
Common confusion
Character-to-token ratios vary by language, content, and tokenizer, so count with the target model's tokenizer or provider tools.
Related terms
Browse the learning paths to see this term in context — every lesson is free to read.