Engineering Practice of Prompt Caching: Core Mechanisms, Boundaries, and Costs

Why This Round of Prompt Caching Is Worth Attention

In the second half of 2024, OpenAI, Anthropic, and Google will successively launch server-side Prompt Caching. It will no longer be a DIY "DIY method of 'setting up KV cache with Redis" but will become a low-cost API native feature. The core change is that cache hits no longer require manual management of expired and hashed caches; instead, they are automatically processed at the API layer, billed separately based on cache hit volume (usually 20%-50% of the full prompt price). For Agent loops or RAG pipelines that frequently call the same system prompt, latency can be reduced from 2-3 seconds to 200-400 milliseconds, cutting costs in half.

But the most easily overlooked aspect of this craze is: Cache is not a "free lunch". API providers have strict regulations on "what content can be cached, how long, and when it expires." Without understanding these boundaries, at best cache failure leads to zero acceleration, or at worst, unexpected context leaks.

What problems does it actually solve in real engineering processes?

Imagine a typical Agent workflow:

1. Users send messages saying, "Help me check tomorrow's weather in Shanghai."
2. Agent calls functions (get_weather) and attaches the current system prompt (including role definitions, tool lists, output format constraints).
3. The LLM returns a JSON instruction.
4. The Agent interprets the JSON and calls the weather API.
5. Stitch the results into the final reply.

In this process, if the system prompt between steps 2 and 3 is 2k tokens, reprocessing each loop is a huge waste. **The core problem solved by Prompt Caching is: for repeated prompt prefixes (usually system prompt + fixed few-shot), only one KV cache is computed, and subsequent inferences with the same prefix are directly reused. ** Scenarios like this are very common in agent loops, multi-turn conversations, and A/B testing.

Specific scenario: Caching strategies for customer service robots

A common customer service bot system prompt might include:

• Company Knowledge Base Index (1k tokens)
• Dialogue style instructions (0.5k tokens)
• Security filtering rules (0.5k tokens)

Assuming each user asks a question, the system must stitch these 2k tokens together, with 100,000 daily requests, and a KV cache hit rate of over 90%. Based on price cache hit, daily costs can drop from $20 to $4 (for example, OpenAI pricing).

The Most Prone Areas for Failure and Misunderstanding

Failure Point 1: Frequent changes in system prompts or context prefixes

If the system prompt contains custom variables (such as the user's current country), timestamps, random authorization codes, or other different content each time, then the prompt prefix will be contaminated. For example:

You are a helpful assistant. Today is 2025-04-07.

每次日期不同，缓存永远无法命中。解决方案是把易变字段放到前缀之后（比如用户消息里），并严格确保前 80% 的 token 保持稳定。

失败点 2：多用户共用一个 API Key 时的上下文泄漏

Prompt Caching 是按公共前缀匹配的，不是按会话隔离的。如果两个用户的对话 history 前几句相同（比如都是“你好”），则可能复用对方的缓存。第三方库 ]anthropic-python[ 的早期版本就曾因为自动拼接 assistant 前缀而意外触发跨会话缓存。解决方案是在每个用户请求中加入唯一标识 ]session_id[ 作为前缀的一部分（但不需要缓存它），例如在 prompt 最开头添加<session id="abc">，然后利用 API 提供的“前缀跳过”参数（如 Anthropic 的 `]prompt_caching' configuration) to avoid caching these metadata.

Misunderstanding: Caching definitely saves costs

In fact, if the cache hit rate is below 30%, the cost is actually higher than without a cache—because although the price per cache hit token is low, the cache write itself is usually uncharged, but it consumes server resources. Some APIs charge additional storage fees based on the number of cached tokens (e.g., Google Gemini's $0.10 / 1M tokens stored). For applications with extremely short conversation lengths (<100 tokens) or no fixed prefixes, the benefits from Prompt Caching are almost zero.

If you want to land now, what should you do first?

1. Audit Your Prompt: Identify all high-frequency API calls and record their common prefix length and change frequency. Prioritize caching for endpoints where the prefix is stable above 80%.
2. Check API documentation: OpenAI requires the prompt to match exactly, with length >=1024 tokens; Anthropic Only caches the first 1024 tokens; Google Supports any prefix but charges based on storage volume. Caching strategies vary greatly among providers.
3. From Single Endpoint Grayscale: Select a non-critical path (such as backend analysis report generation), enable caching first, and add monitoring metrics: cache hit rate, P50/P95 latency, and before-and-after cost comparison.
4. Set fallback switch: If the hit rate falls below 20%, or unexpected contextual confusion occurs (such as data from other users appearing in the reply), immediately downgrade to no-cache mode.

Below is the migration checklist (see images in the text):

• Determine the cacheable stable prefix (length > = 1024 tokens)
• Modify API call headers and add cache enable parameters (such as "anthropic-beta": "prompt-caching-2025-03-17")
• Move the variable field after the prefix or use the skip tag
• Running A/B tests in production (50% traffic with caching enabled)
• Monitor cache hit rates, error rates, and user feedback

Rollback Scheme for Failure

If prompt caching never gets meaningful hits (for example, platform unsupported, prefixes too short, frequent changes), you can revert to:

• Local KV cache: Cache LLM output using Redis or in-memory cache (Note: only applies to identical prompts and temperatures; output varies depending on temperature).
• Semantic Caching: Embedding prompts, searching Redis for historical requests with similarity >0.95, and reusing outputs (but requires hash verification to avoid logical bias).
• Request Merging: Repeated requests for the same prompt within a short period can be directly returned from memory (suitable for polling scenarios).

These solutions require you to handle expires, conflicts, and isolation yourself, which is less hassle-free than native API caching, but they offer complete control.

Next steps: From tool user to system designer

If you're already considering engineering details like cache hit rates, cross-session leaks, and cost models, it means you're no longer satisfied with just the "API tune" level. True system efficiency improvements come from a deep understanding of Provider features and composer design—such as combining prompt caching with Agent loop control flows, or using context reuse in MCP tool calls. This systematic knowledge cannot be closed in fragmented articles; it requires a systematic curriculum and engineering case studies to fill the gap.