AI Inference & Serving
KV Cache Offloading
Keeping some key/value cache data in another memory tier, such as host memory, and transferring it when computation needs it. Offloading changes where states reside rather than intentionally removing them. It can reduce accelerator residency while adding transfer costs; implementation support and overlapping buffers determine the actual memory and timing behavior.
Reviewed
Sources
Member lesson
The definition and sources are public. The complete practical lesson is for members.