AI Inference & Serving
Inter-Token Latency
The delay between successive generated-token outputs. Some serving metrics instead measure gaps between streamed events that can contain multiple tokens. Record the event definition before interpreting a reported average.
Also called: ITL
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.