AI Inference & Serving
Token Throughput
A token count divided by a declared time interval. Specify input versus output tokens and single-request versus aggregate system throughput; these measure different things.
Reviewed
Sources
Free complete lesson
This definition and the complete practical lesson are free. The interactive reader loads the lesson examples without a membership.