When an AI voice app grows, the inference bill grows faster. On TalkDrill, our in-house English-speaking practice app for Indian adults, the monthly AI spend per active user was climbing toward a number that would have broken the unit economics at scale. We cut inference cost per session by routing each task to the cheapest model that clears a quality bar, caching aggressively, and batching where latency allowed — without users noticing any drop in voice quality. This is the cost-engineering playbook, with the unit-economics math and the quality guardrails that kept it honest.
Figures here are illustrative composites for teaching the method, not live internal billing metrics.
How do you cut AI inference cost without hurting quality?
You stop using one expensive model for everything. Most AI apps route every request to a single frontier model, which is like sending every letter by courier. The fix is a routing layer that sends each task to the cheapest model that still clears a measured quality bar, plus caching for repeated work and batching for anything not on the live-latency path. Done with a hard quality floor, this typically cuts inference cost by half or more while keeping the user experience identical.
Why cost-engineering matters now (2025)
AI voice apps have a structural problem: every spoken turn can touch speech-to-text, a language model, and text-to-speech. At a few hundred users you ignore the bill; at thousands of daily-active users it becomes your largest variable cost and decides whether the business works. Model prices have fallen and the spread between cheap and frontier models has widened, which means routing is now a bigger lever than it was a year ago. We've written before about the latency side of this in how we hit a 740ms voice round-trip on Indian 4G; this post is the cost side of the same system.
The 4 levers we pulled
How to do it: the cost-engineering walkthrough
This is the sequence we ran, and it's the one we run for client AI systems too. Each step has a verification check.
If you want this done on your AI system, we run a fixed-scope cost-engineering engagement that starts with instrumentation and ends with a routing layer your team owns.
The unit-economics math
The number that matters is fully-loaded cost per active user per month, because that's what sits against your revenue per user. Here's the shape of the before-and-after on a per-session basis.
The big win was routing plus caching — that's where roughly 58% of the saving came from. Batching added a few more points but mostly on non-interactive work. The lesson for any AI product team: routing and caching are the heavy levers; chase those first before micro-optimising. For the broader business framing of why this matters, our piece on whether traditional SaaS is dying in the age of AI covers how inference economics are reshaping software margins.
Where cost-cutting goes wrong
Three more failure modes we've seen:
Caching dynamic content by mistake. Cache the static scaffolding of a prompt, not the personalised feedback. We once cached too aggressively and two users briefly saw feedback meant for a similar utterance. Version your cache keys and keep the personalised slice out of the cache.
Optimising the cheap tasks. Spending a week shaving cost off intent detection — which was already 3% of the bill — while the scoring calls dominated. Always optimise the biggest line first. Your instrumentation tells you which one that is.
No quality regression gate. Shipping a routing change without a CI check means a model provider's silent update can drop you below the floor and nobody notices for weeks. Automate the floor check. Make it block the build.
A real example beyond TalkDrill
We applied the identical playbook for a client running a customer-call-summarisation pipeline that was sending every transcript to a frontier model. We instrumented cost per summary, set a quality floor (a rubric-graded check that action items were captured), and found a mid-tier model cleared the floor on 80% of summaries — only long, multi-speaker calls needed the top model. Routing on transcript length cut their summarisation bill by just over half with no measurable drop in summary usefulness. The pattern travels: instrument, set the floor, route, cache, batch. This is bread-and-butter work for our AI and automation team, and the unit-economics discipline carries straight over from running our own products. For more on the model-routing decision itself, see our hands-on comparison in where GPT-5.5 beats Claude Opus 4.7 across 12 production tasks.
A cost-engineering checklist for AI products
- Every model call is tagged by task type, with tokens and cost logged
- Cost-per-session is broken down by task on a dashboard you actually look at
- Each task has a written quality floor with a recorded baseline
- No task is routed to a cheaper model without a recorded floor-pass
- The hardest, most user-visible task keeps the best model
- Repeated static work is cached with versioned keys; personalised slices are not
- Non-interactive work runs in off-peak batches
- A quality regression below the floor fails CI and blocks the build
Our take
Cost-engineering an AI product is mostly discipline, not cleverness. The clever part — model routing — takes a week. The discipline part — defining a quality floor, instrumenting cost per task, and never cutting below the floor — is what separates a sustainable margin from a slow-motion quality collapse. As Vivek, our CEO, frames it: "Anyone can make an AI app cheaper by making it worse. The job is making it cheaper while it stays exactly as good — and proving it." We crosschecked our routing thresholds against r/LocalLLaMA and r/MachineLearning threads on production inference cost, where task-based routing plus a held-out eval is the repeated community answer to the same problem.
Frequently asked questions
What is model routing and how does it cut AI costs?
Model routing sends each task to the cheapest model that still clears a quality bar instead of using one expensive model for everything. Simple tasks like intent detection go to a small fast model; only the hardest tasks touch a frontier model. Because cheaper models cost a fraction per token, routing often halves total inference spend.
How do you make sure quality doesn't drop when cutting costs?
Define a written quality floor per task with a held-out test set before changing anything, record the current model's baseline score, and only route a task to a cheaper model if it clears that floor. Then wire the floor check into CI so any regression fails the build. Cost cuts are gated, never blind.
Does caching work for AI voice apps?
Yes, for the repeated parts. Static scenario prompts, lesson scaffolding, and fixed audio phrases repeat constantly and can be cached so you don't pay to regenerate them. Don't cache personalised feedback — keep the dynamic slice out and version your cache keys so a content change busts the cache.
What's the difference between batching and real-time inference cost?
Real-time inference serves a waiting user and must be fast, so it's priced at a premium. Batched and off-peak processing of non-interactive work — nightly summaries, weekly reports, content pre-generation — is materially cheaper. Moving everything that isn't in front of a live user to batch is a clean saving with no UX cost.
Which AI task should never be routed to a cheaper model?
The one users came for. On a voice-practice app that's pronunciation-and-fluency scoring; on a summarisation tool it's the summaries themselves on hard inputs. Cut cost on invisible plumbing — intent detection, logging, background jobs — and keep the best model on the core, user-visible task.
How much can cost-engineering realistically save?
For an AI product using a single frontier model for everything, routing plus caching commonly cuts inference cost by half or more, with batching adding a few points on non-interactive work. The exact figure depends on your task mix and how much repeated work you have, but the heavy levers are routing and caching.
Is Your AI Product's Inference Bill Outpacing Revenue?
We run cost-engineering on AI systems for Indian product teams — instrumentation, model routing, caching, and a quality floor wired into CI so savings never break the experience. Typical engagement: 2–4 weeks, fixed scope, with a routing layer your team owns at the end. First call is technical, with the engineer who'd lead it.
Book a 20-min Call
