AI Inference & Serving
KV Cache Eviction
Removing selected stored key/value states to remain within a cache budget. The selection policy determines what is retained and can change attention computation or require recovery work. Eviction is distinct from moving the same states elsewhere, quantizing their representation or merely mapping them into different physical blocks.
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.