QA for AI Features: Testing Nondeterministic Outputs Without Losing Your Mind

The day your product shipped its first LLM feature, your QA strategy quietly broke. The old contract — same input, same output, pass or fail — no longer holds. Send the identical prompt to a model twice and you may get two different sentences, both correct, neither matching your expected string. As of May 2026, almost every Indian product team we work with has at least one AI feature in production, and most are testing it the way they test a login form: with assertions that flake on the first run and get disabled by the second sprint. This is a QA playbook for features whose outputs are nondeterministic — how to make them testable, regressable, and safe to ship.

Why exact-match testing fails for AI features

Traditional QA rests on one assumption: a function is a pure mapping from input to output. add(2, 2) is always 4. You assert equality and move on. LLM features violate this at three levels — temperature-driven sampling yields different tokens, providers silently update weights behind the same version string, and a tiny upstream prompt edit can ripple into totally different phrasing downstream. None of those changes are bugs. But every one breaks assertEquals(expected, actual).

The instinct is to set temperature to 0 and call it deterministic. It isn't. Even at temperature 0, floating-point non-associativity across GPU batches and provider-side load balancing between model revisions produce drift. We've watched a "deterministic" support-bot test pass 200 times in CI, then fail in production after the provider rotated to a new minor model build overnight. The lesson: stop testing for string identity. Start testing for behavioural properties that hold no matter which valid phrasing the model returns.

The five layers of an AI QA strategy

Think of AI testing as a pyramid, not a single test type. Each layer catches a different class of failure, and you need all five before you can ship AI changes with confidence.

Build a golden set before you write a single assertion

Everything downstream depends on a golden set — a curated collection of representative inputs paired with what "good" looks like for each. This is the single highest-leverage artifact in AI QA, and most teams skip it because it's unglamorous manual work. Don't.

A useful golden set for an Indian product has 50–200 cases spanning the realistic distribution: happy path, messy real-user phrasings, Hinglish and regional-language inputs, edge cases (empty, very long, contradictory), and adversarial probes. For a support bot, that means actual past tickets, not invented ones. For a writing-feedback feature like a client platform, it means real student submissions graded by a teacher — synthetic essays never reproduce the mistakes a real Class 9 student makes.

Semantic assertions: the three techniques that actually work

Once you have a golden set, you need a way to check a fresh output against "good" without exact match. Three techniques, in increasing order of cost and power.

1. Property and keyword assertions (cheap, fast, brittle-in-a-good-way)

The simplest semantic check is presence/absence. The answer to "What is the refund window?" must contain the number of days from your policy and must not contain a competitor's name. These are fast, free, and deterministic. Lead with them — a surprising share of AI bugs are caught by "the answer didn't mention the one fact it had to mention."

2. Embedding similarity (good for "is this roughly the right answer?")

Embed the candidate output and your reference answer, compute cosine similarity, and assert it clears a threshold you tune on the golden set. This tolerates rephrasing while catching answers that have drifted off-topic. The trap: similarity is blind to negation. "You are eligible" and "You are not eligible" sit very close in embedding space. Use similarity as a coarse filter, never as the final word on correctness.

3. LLM-as-judge (powerful, needs discipline)

Hand the input, the candidate output, and a strict rubric to a separate model and ask it to grade. This catches nuance nothing else does — but only if you treat the judge as a system under test in its own right.

Prompt regression: your most under-tested surface

Here is the failure we see most often. A developer tweaks one line of a prompt to fix a single bad case — and silently regresses ten cases that were working. With no prompt regression suite, nobody notices until a customer does. Prompts are code. They deserve a test suite and a CI gate exactly like code.

The pattern, which tools such as Promptfoo make routine, is straightforward: every prompt has a fixture file of golden cases with assertions; on every change you run the full set; the build fails if pass rate drops below threshold. We documented exactly this on our in-house English-fluency app TalkDrill, where a prompt-eval pipeline gates every change before it touches production — the full write-up lives in our prompt-eval pipeline case study. The discipline matters more than the specific tool: pick one, wire it into CI, and never merge a prompt change that drops the score.

Measuring flakiness instead of pretending it away

A nondeterministic feature has an inherent pass rate, and you should measure it. Run each golden case three to five times and record how often it passes. A case that passes 5/5 is stable; one that passes 3/5 is a genuine reliability problem you can now quantify and track — not a flaky test to disable. Set a per-feature target ("95% of golden cases must pass ≥4/5 runs") and treat regressions against that number as build failures. Instead of arguing whether a test is "flaky," you report that reliability moved from 96% to 89% and ask why.

Human-in-the-loop gates: where to put a person

Automation handles volume; humans handle judgement and calibration. The art is placing the human where the cost of being wrong is highest and the cost of review is affordable. Three placements earn their keep.

Pre-launch acceptance gate. Before a new AI feature ships, a person grades the full golden set by hand once, establishing the ground-truth pass rate your automated checks must reproduce. If suite and human disagree by more than a few points, your assertions are wrong, not the feature.

Continuous production sampling. Sample a small slice of live traffic — say 1–2% — and have a reviewer grade it against the same rubric weekly. This is your early-warning system for silent provider drift and the failure modes your golden set never imagined.

High-stakes inline review. For irreversible or regulated actions — anything touching money, health, legal, or a child's data — route the output to a human before it acts. Treat the model as a drafting assistant, not the final authority, until the data says otherwise.

A pragmatic rollout: from zero to a real AI test suite

You don't need all five layers on day one. Sequence it so each step pays for itself.

• Week 1 — write deterministic shell tests and JSON-schema structural assertions; these alone catch most "it broke" incidents
• Week 2 — build a 50-case golden set from real inputs, tagged by category, including Hinglish and regional-language cases
• Week 3 — add property/keyword and embedding-similarity assertions; record per-slice pass rates
• Week 4 — stand up an LLM-as-judge with a structured rubric and validate it against your own hand grades
• Week 5 — wire prompt regression into CI with a pass-rate gate; no prompt merges below threshold
• Ongoing — sample 1–2% of production weekly for human review; re-validate the judge every release

The teams that get this right stop fearing model updates. When a provider ships a new build or you want to switch models, you run the suite, read the pass rate, and decide on evidence — turning "the AI feels worse today" into a number you can act on.

FAQ

Should I set temperature to 0 to make tests deterministic?

It reduces variance but does not eliminate it, and it often degrades output quality on creative tasks. Lower temperature for tasks that should be deterministic (extraction, classification), but still test for properties rather than exact strings, because provider-side drift persists at any temperature.

How big should my golden set be to start?

Fifty real, well-chosen cases beat 500 synthetic ones. Start at 50, weighted toward your actual traffic distribution and your highest-risk slices, then grow it every time a production bug escapes — each escaped bug becomes a new golden case so it can never escape twice.

Isn't LLM-as-judge just testing one black box with another?

Yes, which is why you validate the judge against human grades and pin its model version. A calibrated judge that agrees with your reviewers 90%+ of the time is a force multiplier; an unvalidated one is theatre. Treat the judge as a system under test, not an oracle.

Who owns AI QA — QA or engineering?

Both. Engineering owns the deterministic and structural layers; QA owns the golden set, the rubrics, and judge calibration. The handoff that breaks teams is when nobody owns the rubric — write it down and assign it.