Your LLM Isn't Slow — Your Capacity Math Is Wrong: A New Simulator Predicts the Latency Cliff Before Your Users Hit It

You ship an LLM feature, load-test it, and the dashboard says you can handle 100 requests a second. Then real users arrive — bursty, uneven, with long prompts and longer answers — and the experience falls apart at half that load. The first token takes forever, and nobody can explain why, because the number you planned around was never the right number. A new paper argues the problem is not your model or your GPUs: a single throughput number cannot describe a serving system at all — and it offers a simulator that predicts the full latency picture instead.

What the researchers did

The paper is “GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving” by Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis, Anjaly Parayil, Dhruv Deshmukh, and Victor Ruehle, submitted to arXiv on 27 October 2026. Inference is now an operational workload fleets must provision, monitor, and bill against latency service-level objectives (SLOs) — and one throughput number cannot answer the questions operators face. Generative serving couples two phases: prefill (the model reads the whole prompt at once, a compute-heavy burst) and decode (tokens drip out one at a time, a memory-bandwidth-bound crawl). Continuous batching reshuffles which requests share the GPU. The nasty result: user-visible tail latency degrades long before aggregate throughput saturates. GAUSS instead predicts distributions, not averages: full distributions of time-to-first-token (TTFT) and time-between-tokens (TBT), plus the sustainable request rate, for a given workload, engine, and accelerator. Its three design choices: (1) phase-specific calibrated latency surfaces for prefill and decode; (2) a Lindley-recurrence queueing model under stochastic arrivals, letting it reason about workloads, routing policies, and hardware never actually deployed instead of needing a recorded trace; (3) ensemble Monte-Carlo sampling of many possible futures, capturing the latency tails trace-driven simulators systematically under-sample. It supports operating-point selection, traffic segmentation, and shortest-remaining-processing-time (SRPT) scheduling capacity questions.

Why it matters

Most teams size LLM serving like web servers, but that playbook fails: averages hide the pain (users experience their own request, not the average; if p95 TTFT blows up while throughput looks healthy, the SLO is already breached); workloads are not uniform (short chat vs long documents are really two systems sharing one GPU pool); and guessing is expensive both ways (over-provision burns GPU budget, under-provision discovers the cliff in production at peak traffic). Caveat: GAUSS is a model, only as good as its calibration to your engine, accelerator, and workload mix — it tells you which measurements matter, it does not replace measuring.

How YOU can use it

(1) Write SLOs as distributions, not single numbers — e.g. “p95 TTFT under 1 second, p95 TBT under 50 milliseconds, at peak arrival rate.” (2) Measure prefill and decode separately: prefill latency vs prompt length, decode speed vs batch size and output length. (3) Find your sustainable rate and your cliff: ramp arrival rate in steps, plot p95 TTFT/TBT, the sustainable rate is the highest rate where SLOs hold, the cliff is where the tail bends sharply upward. (4) Segment traffic before sizing: short chat, long-context analysis, batch jobs, sized separately. (5) Test scheduling ideas on paper first: under a policy like SRPT, at this arrival rate, what happens to the tail? (6) The paper’s queueing core is a Lindley recurrence — an illustrative sketch only, NOT paper code: wait = max(0, wait + service_time - gap_since_last_arrival), run over many sampled arrival patterns to get a waiting-time distribution. Capacity template: workload, SLOs (p95 TTFT < X s, p95 TBT < Y ms), traffic classes, arrival pattern, hardware/engine — question: sustainable request rate per class, where does the tail bend upward, which scheduling policy keeps the tail inside the SLO?

Key takeaways

One throughput number cannot describe LLM serving; GAUSS predicts full TTFT/TBT distributions plus sustainable request rate; it combines phase-specific latency surfaces, Lindley-recurrence queueing, and ensemble Monte-Carlo sampling for workloads/routing/hardware never yet deployed; payoff is operating-point selection, traffic segmentation, and scheduling capacity questions answered before buying GPUs or signing SLOs; your move is to state SLOs as distributions, measure prefill and decode separately, and find your latency cliff on paper.

Read the original paper: https://arxiv.org/abs/2610.37989