What Is Prefix Caching / Cache Reuse?
The Problem Without Caching
Section titled “The Problem Without Caching”Without caching, the same prompt prefix is reprocessed every time, even when it’s identical across consecutive requests. That costs time and tokens.
The Solution
Section titled “The Solution”With KV cache prefix caching, the operator keeps already-computed key-value states for recurring prompt prefixes in GPU VRAM/RAM. On a cache hit, the prefix doesn’t need to be recomputed.
Benefits
Section titled “Benefits”- Faster: lower latency, since the prefix isn’t reprocessed
- Cheaper: cached input tokens are billed at a reduced rate
Important Limitations
Section titled “Important Limitations”Suitable Use Cases
Section titled “Suitable Use Cases”- Fixed system prompts that are identical across every request
- RAG pipelines with the same context
- Multi-turn conversations sharing conversation history