AI Inference & Serving
Dynamic Batching
Combining compatible incoming requests into execution batches according to a scheduling policy. Here the term means request-level batch formation, distinct from changing generation membership between token iterations.
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.