AI Inference & Serving
Continuous Batching
Serving multiple generation requests while allowing finished sequences to leave and new work to enter at scheduling boundaries. The active batch can change between generation iterations.
Also called: Iteration-Level Batching
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.