Commands & Actions
Tokenize
Convert text into units and usually their numeric identifiers using a particular tokenizer. Units can represent words, word pieces, characters or bytes depending on the design. Tokenization is a representation step, not proof of understanding, and different tokenizers can divide the same text differently.
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.