To reduce LLM latency, measure where the time goes first, then work in this order: generate fewer output tokens, stream the response, stop calling a large model for steps that don’t need one, cache repeated prompt prefixes, and only then

tune the serving layer. Most teams get the biggest wins from the first three, and none of them need new hardware.

This guide is for engineers and technical founders who have a working LLM feature that feels slow. If you’re after low latency from an LLM that is already live, the sections below follow the order we’d use on a real codebase.

TL;DR

  • Output tokens dominate latency. Cutting them is the highest-impact change, and OpenAI’s latency guide gives a rough rule that halving output tokens roughly halves latency.
  • Streaming doesn’t make the model faster, but it makes the app feel fast by showing the first token in under a second.
  • Ask whether a step needs an LLM at all. Classification and formatting often run faster and cheaper in plain code or a small model.
  • Prompt caching and session-aware routing help most in high-traffic apps. Speculative decoding and batching tuning come last, and only if you host the model yourself.

Table of Contents

Measure before you optimize

Latency is not one number. Split it into at least three:

  • Time to first token (TTFT): how long from request to the first piece of output. This is what a user feels in a streaming chat. AWS describes it as the time from submitting a request until the beginning of a response arrives.
  • Inter-token latency: the gap between tokens once output starts. It sets how smooth the stream looks.
  • End-to-end latency: the full wait until the last token. This matters for anything a program consumes, such as JSON for a downstream step.

Microsoft’s latency guidebook lists four main drivers: the model, the number of prompt tokens, the number of generated tokens, and the load on the deployment. That list is a good diagnostic checklist. Log all four for every request, along with the three timings above, and look at p50 and p95 rather than averages. A single slow tail request is what users remember.

Benchmark with realistic traffic too. AWS guidance on SageMaker endpoints recommends testing against real token distributions and concurrency rather than a single clean prompt. A test with one user and a short prompt will flatter you.

Don’t optimize a prompt that doesn’t work yet

There’s a trap we see constantly: a team gets a demo working, panics about speed, and starts squeezing the prompt before it produces reliably good answers. Anthropic’s guidance on reducing latency makes the same point: get a prompt that performs well first, then apply latency techniques. Forcing brevity too early can degrade reasoning and send you back to undo it.

So set up a small evaluation set before you touch anything. Fifty to a hundred representative inputs, with a pass/fail check, is enough to tell you whether a faster variant is also a worse one. If you don’t have that yet, our guide to CI-gated LLM evaluation shows how to build it. Every change below should clear that gate.

Generate fewer output tokens

This is the first lever and usually the biggest. OpenAI’s latency guide says that generating fewer tokens is one of the highest-impact moves, with a general heuristic that cutting 50% of output tokens may cut latency by about 50%. Trimming input tokens helps too, but by comparison its effect is often small unless your context is very large.

Practical ways to shrink output:

  1. Ask for concise answers. Say what length and format you want. “Two sentences” beats “be brief.”
  2. Set a max token limit as a safety net, not as your main control. A truncated answer is a bug.
  3. Use few-shot examples that show the short style you want.
  4. Return structured fields, not prose. If your code only needs a label and a confidence score, don’t pay for a paragraph of explanation.
  5. Shorten field names and drop optional keys in JSON outputs. Every token generated is time spent.
  6. Avoid unnecessary reasoning output for simple tasks. Save extended reasoning for requests that need it.

Check the result against your evaluation set. If accuracy holds and the output is half as long, you’ve just bought a large latency gain for free.

Stream the response

Streaming is the cheapest way to improve how fast your app feels. The total generation time stays the same, but the user sees text begin in well under a second instead of staring at a spinner. OpenAI’s guide calls out streaming as a key technique for perceived responsiveness, and notes that if output needs post-processing such as moderation, you can chunk it so segments reach the interface progressively instead of waiting for the full response.

A few details that teams miss:

  • Make sure no proxy, gateway or serverless layer buffers the stream. A single buffering hop silently turns streaming back into a long wait.
  • Render partial output safely. Don’t parse half a JSON object.
  • Pair streaming with a loading state for the gap before the first token.

Streaming does nothing for machine-to-machine calls. Those need the real fixes below.

Ask whether the step needs an LLM

OpenAI’s guide suggests that before optimizing a step, ask whether an LLM is required at all. For deterministic work such as simple classification, routing on known keywords or reformatting, ordinary code or a specialized small model is often faster, more reliable and cheaper.

A quick audit of your pipeline usually finds:

Step Typical fix
Reformatting dates, currency, casing Plain code, no model
Intent routing with a handful of classes Small classifier or rules
Extracting fields from a fixed template Regex or a parser, with an LLM fallback
Open-ended drafting or reasoning Keep the LLM, tune it

The same logic applies to model size. Smaller models generate faster. Route easy requests to a small model and escalate only the hard ones, and measure the quality drop on your evaluation set before you commit. If your pipeline chains several calls, also look for calls that don’t depend on each other and run them in parallel.

Cache what repeats

If every request starts with the same long system prompt, tool definitions or reference document, you’re paying to process it again each time. Prompt caching lets the provider reuse that work. AWS describes prefix caching as avoiding redundant compute for repeated prompt segments, which lowers latency for users with overlapping interaction patterns.

The rule that makes caching work: put stable content first and variable content last. Instructions, tool schemas and shared documents go at the top. The user’s question goes at the bottom. A timestamp or user ID near the top of the prompt will break the shared prefix on every call.

If you self-host, routing matters just as much. The same AWS page notes that using round-robin routing instead of session-aware routing can force redundant recomputation and cancel out caching gains. Keep a conversation on the same instance when you can.

For the full playbook, including cache hit-rate monitoring and invalidation pitfalls, see our prompt caching guide for production LLM systems.

Also consider an application-level cache for exact or near-duplicate questions. An answer served from your own store returns in milliseconds, which no model tuning can match. Just be strict about what is safe to reuse: never share cached answers across users when the response depends on private data.

Cache what repeats — overview diagram

Trim and tighten the context

Input tokens matter less than output tokens, but long contexts still raise TTFT because the model must process everything before it starts writing. The guidance on time to first token points to the usual suspects: bloated chat history, repeated boilerplate in system text, and retrieval that stuffs in too many noisy chunks.

What we’d do:

  • Keep only the conversation turns that still matter, and summarize the rest.
  • Retrieve fewer, better chunks. Five relevant passages beat twenty loosely related ones, for speed and for accuracy.
  • Remove duplicate instructions that crept in over months of prompt edits.

For agent systems, memory and retrieval design drive a lot of delay. The AWS Well-Architected Agentic AI lens covers how short-term memory, knowledge stores and context windows affect responsiveness.

Tune the serving layer if you host the model

If you call a hosted API, your control stops around here. If you run models yourself, a second set of levers opens up.

Continuous batching. AWS notes that continuous batching groups requests dynamically to use GPU compute and memory bandwidth efficiently, keeping throughput steady under uneven traffic. Batch size is a trade-off: smaller batches favor low latency for interactive chat, while larger ones raise overall throughput.

Four model serving levers and tradeoffs

Speculative decoding. A small draft model proposes tokens and the large model verifies them. The vLLM documentation says it improves inter-token latency for memory-bound workloads at medium to low request rates. It is not a free win everywhere: it helps most when the system isn’t already saturated, so test it under your real load.

Quantization and parallelism. Lower-precision weights such as FP8 or INT8 cut memory pressure, and tensor parallelism spreads a large model across GPUs. Both change output quality slightly, so rerun your evaluation set.

Right-size the deployment. AWS warns that over-provisioning is a common pitfall and recommends benchmarking existing endpoints against real traffic to right-size them. Sometimes the fix for a slow endpoint is a differently shaped one, not a bigger one.

If you’re considering Kubernetes for this, our walkthrough of first moves to deploy LLMs on Kubernetes with vLLM covers the starting points. And if you’re weighing whether to host at all, the self-hosted LLM decision framework lays out the break-even thinking.

A sensible order of attack

  1. Instrument TTFT, inter-token latency, end-to-end time and token counts, at p50 and p95.
  2. Build a small evaluation set so you can tell fast from fast-and-wrong.
  3. Cut output tokens and tighten the format.
  4. Turn on streaming and check that nothing buffers it.
  5. Remove or downsize steps that don’t need a large model.
  6. Restructure prompts for caching, and add an application cache where safe.
  7. Trim context and retrieval.
  8. Only then touch batching, speculative decoding and hardware.

Skipping to step 8 is the expensive mistake. We’ve seen teams rent bigger GPUs to hide a prompt that asks for a thousand tokens when forty would do.

Our perspective

We get called when an AI feature works in a demo and falls apart under real use. Slowness is rarely a single bug. It’s usually a stack of small decisions: a verbose prompt, a model chosen because it was the default, a gateway that buffers, a cache that never hits because a timestamp sits at the top of the prompt.

Our view is that latency, cost and quality are one problem. Fewer tokens means faster responses and a smaller bill. A smaller model on the right step does the same. That is why our workflow is built around reducing token costs and complexity, not adding layers. The fastest request is the one you didn’t need to make.

We also think nontechnical founders deserve a straight answer. If an app built with AI tools feels sluggish, the cause is often visible in an afternoon of reading the code and the logs. You don’t need to rebuild. You need someone to tell you which three changes matter and which twelve don’t.

— Chad

Getting a second pair of eyes on a slow LLM app

If you’d rather not spend a week profiling, we can look at it with you. A few options, with published prices:

  • Vibe Check, $449 one-off: a review aimed at apps built with AI tools, useful when you don’t know why something is slow or fragile.
  • Infrastructure Assessment, $2,750 one-off: a look at how your services are deployed and where requests wait.
  • Automated Monitoring / Logging / Alerts, $2,750 one-off: the instrumentation from step 1, so latency regressions show up before users complain.
  • AI Assisted Engineering, from $9,950 one-off: hands-on engineering when you want the fixes built, not just listed.

For self-hosted or private deployments, see our services overview, which includes local AI model development.

FAQ

What is a good time to first token for a chat app?

There’s no universal number, because it depends on prompt size, model and load. As a working target, aim for the first token to appear in under a second, which is what streaming guidance describes as achievable. Then set your own p95 budget from user feedback and watch it over time.

Does a smaller model always mean lower latency?

Smaller models generally generate faster, but the saving only counts if quality holds. Test the smaller model on your evaluation set for the specific step. Often the best result is routing simple requests to a small model and hard ones to a larger one.

Will prompt caching reduce latency or just cost?

Both, when your prompts share a long stable prefix. Reused prefix work lowers the compute needed per request, which AWS notes lowers latency for overlapping usage patterns. It only works if variable content sits after the stable part of the prompt.

Is speculative decoding worth trying?

If you self-host, yes, but test it. The vLLM docs describe benefits for inter-token latency under medium to low request rates, and results vary with workload. If you use a hosted API, your provider decides this for you.

Sources