Reducing LLM cost and latency: caching, routing and shorter prompts

LLM apps are billed mostly by tokens (input plus output), and response time grows with output length and model size. A feature that is cheap in testing can become expensive at scale. The good news: a few habits often cut cost sharply with little loss of quality.

Step 1: measure first

You cannot optimise what you do not measure. Log for every request:

• Input tokens, output tokens, model, latency and estimated cost.
• Which feature or user triggered it.

Then look for the biggest spenders and the slowest paths. Often one feature produces most of the bill. Check current prices on your provider's pricing page, because they change.

Step 2: shorten what you send

Input tokens are paid on every call.

• Remove filler from prompts and repeated boilerplate.
• Replace long rule lists with a few good examples, or the other way round, whichever is shorter and works.
• In RAG, send fewer, better chunks rather than everything.
• Trim and summarise old conversation history rather than resending it all.
• Strip HTML, whitespace and irrelevant fields from the data you include.

Step 3: control the output

Output tokens are usually pricier and slower than input tokens.

• Ask for concise answers and set a sensible max tokens.
• Use structured output with short fields instead of long prose when code consumes the result.
• Do not ask the model to repeat the input back.
• Use extended step-by-step reasoning only where it measurably helps.

Step 4: pick the right model for each job

Not every task needs the biggest model.

• Model routing: use a small, cheap, fast model for easy tasks (classification, extraction, simple rewriting) and a larger one only for hard ones.
• A cascade: try the small model first and escalate if confidence is low or validation fails.
• Test the small model on your eval set. If it passes, you have saved a lot.
• Consider fine-tuning or distilling a small model for a high-volume narrow task (see the fine-tuning article).

Step 5: cache

• Response caching: if the same (or normalised-identical) request repeats, return the stored answer. Great for FAQs and deterministic tasks. Be careful never to serve one user's answer to another when the response depends on private data.
• Semantic caching: reuse answers for questions with the same meaning, using embeddings. Set a strict similarity threshold to avoid wrong matches.
• Prompt (prefix) caching: many providers discount or speed up repeated identical prompt prefixes. Put the stable parts (instructions, examples, documents) at the start and the changing user input at the end. Check your provider's rules and minimum sizes.

Step 6: make it feel faster

• Streaming: show the answer as it is generated. Total time is the same, but the user sees progress immediately.
• Run independent calls in parallel.
• Do cheap steps first and avoid unnecessary chains of calls.
• Use a model close to your users' region where options exist.
• Prepare results in advance for predictable requests.

Step 7: batch and defer

For work that is not urgent (summarising an archive, tagging content), use the provider's batch option if available, which is usually discounted in exchange for slower turnaround, or run jobs off-peak with a queue.

Step 8: protect against runaway spend

• Set rate limits and per-user quotas.
• Cap input size and max tokens per request.
• Limit agent steps and tool calls (see the agents article).
• Set budgets and alerts with the provider and in your own dashboards.
• Prevent loops where an LLM output triggers another LLM call endlessly.
• Protect public endpoints from abuse with authentication and bot protection, since attackers can drain your credit.

Common mistakes

• Using the largest model for everything.
• Resending the full history every turn.
• Putting the changing part at the start of the prompt, which breaks prefix caching.
• No usage logging, so the bill is a surprise.
• Optimising cost without re-running evals, and lowering quality unknowingly.

Takeaway: measure tokens and spend, shorten input and output, route easy work to cheaper models, cache what repeats, stream for perceived speed, and put hard limits and alerts in place. Re-run your evals after every change.

---

AI & LLM series, Part 2 (quality, agents and cost): 4 of 5. Previous: AI agents and tool use: function calling explained.