Concepts
Tokens
Tokens are the units a tokenizer represents for a model, often words, parts of words, punctuation or special symbols. The singular Token describes the same concept. Counts depend on the tokenizer and representation; a fixed words-to-tokens ratio is an estimate, not a universal measurement or billing rule.
Reviewed
Sources
Free complete lesson
This definition and the complete practical lesson are free. The interactive reader loads the lesson examples without a membership.