Concepts
Token
One unit in a model tokenizer representation. Token and Tokens are singular and plural names for the same concept, not different AI mechanisms. A token can correspond to a whole word, a fragment or another symbol; normalization and other tokenizer stages affect how original text becomes model input.
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.