Your LLM Agent's Prompt Cache Is Silently Broken — A 571-Project Study Found the Bill

You switched on prompt caching, watched your LLM bill drop, and moved on. The dashboard still shows cache hits. The answers still look right. So why does the invoice keep creeping up? A new study of real agent codebases has an uncomfortable answer: the most common caching bugs don't break anything visible. They just quietly stop saving you money — and in some cases, they lie to you about how much you saved.

What the researchers did

The paper is "Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents" by Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Yepang Liu, and Leo Zhang, submitted to arXiv on 26 October 2026 and accepted at ICSE 2027. It is the first empirical study of how prompt caching actually goes wrong in downstream applications — not in theory, but in shipped code. The team examined 120 prompt-caching-related pull requests from 10 popular open-source agent systems, labelling each defect by its root cause, its symptom, and its fix. Then they built a detector, CacheLint, and ran it on 571 real-world agent projects. The headline finding is the silence. Most of these mis-integrations fail without any observable runtime failure: no error, no crash, no wrong answer. Your agent keeps working. It just stops being cheap. The defects split into two families. On the request side (97 of the 120, about 81%), the application corrupts the outgoing request so the cache never matches — a prefix that should have been reused gets recomputed at full price. On the read side (23 of the 120, about 19%), the request is fine but the application misreads the provider's cache-usage numbers, so caching works while the reported token counts are wrong. You end up steering by a dashboard that disagrees with your bill. Underneath, the team grouped the defects into eight root causes — and found the fixes were largely predictable: assigning each root cause its most common fix correctly predicted 95 of the 120 fixes, or 79%. These are not exotic bugs. They are the same small mistakes, repeated. CacheLint, the detector, models each mis-integration as a caching call plus the preconditions required for it to work correctly. It locates caching calls with CodeQL, then discharges each precondition either against a dated record of provider capabilities or with an LLM guard check. On the 120 labelled defects it scored 0.71 precision and 0.78 recall — decent, not perfect. In the wild it did better: across 571 projects it flagged 32 previously unknown mis-integrations at 0.86 precision, earning 21 developer confirmations and 18 upstream fixes, including one in Anthropic's official SDK.

Why it matters

To see why this bites, you need the one mechanism prompt caching depends on: prefix matching. When a provider caches a prompt, it stores the processed prefix. A later request reuses that work only if it starts with the exact same prefix. Any modification, insertion, or reordering invalidates the match from that point on. Everything after the change is recomputed at full price. That is a brutal contract for agent code, because agents are constantly shuffling context: a retrieved document inserted near the top, a timestamp or session ID stamped into the system prompt, tool definitions reordered by a framework update. Each of those is a "harmless" edit that silently voids the cache. The paper's context for the stakes: enterprise-level LLM API costs were estimated at $8.4 billion by mid-2025. At that scale, a cache that quietly stops hitting is not a rounding error. The read-side failures are arguably worse, because they corrupt your decisions. If your accounting misparses cache usage, you can believe caching is saving 60% while the invoice says otherwise — or kill a caching setup that was actually working. Either way you are optimising against fiction. And the failure mode is structural, not sloppy. Providers expose different caching APIs, and their caching semantics keep evolving. Code that was correct against last quarter's API can be a mis-integration today, with no test going red — because the output never changes, only the price.

How YOU can use it

1. Put everything static first, everything dynamic last

Design your prompt as two zones. The stable prefix — system prompt, role definition, tool descriptions, stable few-shot examples — must be byte-identical on every call. The dynamic suffix — the user message, retrieved documents, timestamps, session IDs, tool results — goes after it, never before. One rule covers most request-side bugs: nothing that changes between calls may appear above the dynamic line. A timestamp in the system prompt is not a detail; it is a cache kill-switch.

2. Run the five-minute audit

Check four things, in order: Prefix stability — send the same task twice and diff the actual requests leaving your code. Is the prefix byte-identical? If your framework reorders tools or injects metadata, it isn't. Provider capability — does your setup still satisfy the provider's current rules (minimum cacheable tokens, supported models, time-to-live)? Check the current docs, not the blog post you read when you integrated. Accounting reconciliation — compare the provider's reported cached tokens against your actual spend for the same period. If they disagree, suspect a read-side miscount before you suspect the provider. Before/after every prompt edit — re-check the cache hit rate after any template change, however small. A hit-rate drop after a "harmless" edit is the signature symptom of a request-side break.

3. Sketch the split in code (illustrative, not paper code)

STABLE_PREFIX = system prompt + tool definitions (fixed text, fixed order); build_messages appends dynamic content ONLY after the stable prefix. If anything in STABLE_PREFIX can change between calls — a date, a user ID, a reordered tool list — move it below the line or accept that you are paying full price.

4. Alert on the two silent symptoms

Cache hit rate dropping after a deploy, and cached tokens reported without a matching drop in spend. Neither indicates a broken product. Both indicate a broken cache. The paper's central lesson is that nobody will page you for these — you have to watch for them deliberately.

Key takeaways

• Prompt caching fails silently: most mis-integrations in 120 real defects produced no error and no wrong answer — just a higher bill.
• Two failure families: request-side breaks (81%) where the cache never matches, and read-side miscounts (19%) where the reported savings are wrong.
• The root cause is prefix fragility: any insertion, edit, or reorder voids the cached prefix from that point on — timestamps and shuffled context are the usual culprits.
• Fixes are predictable: the most common fix per root cause predicted 79% of the 120 fixes, so a short checklist catches most of it.
• CacheLint proved the point in the wild: 32 previously unknown bugs across 571 projects at 0.86 precision, with 21 developer confirmations and 18 upstream fixes.
• Your move: static prompt first, dynamic content last, and reconcile reported cache savings against your actual bill after every template change.

Read the original paper: "Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents" (arXiv:2610.34817) by Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Yepang Liu, and Leo Zhang — https://arxiv.org/abs/2610.34817