Testing a feature that answers differently every time

By on

Rows of laboratory test tubes in a rack

Traditional QA asks whether the output matches the expected value. With a model behind the feature there is no expected value, and most teams respond by not testing it at all.

Every test suite ever written assumes the same input produces the same output. Feed a model the same prompt twice and you get two different sentences, both arguably correct. The assertion has nothing to attach to.

What I see teams do in practice is skip it. The AI feature is the one part of the product with no automated coverage, which is also the part most likely to embarrass you in front of a customer.

Test the properties, not the string

You cannot assert the answer. You can assert things that must be true of any acceptable answer, and those are usually easy to write.

  • It cites at least one source, and every cited document exists.
  • It never claims a number that does not appear in the retrieved context.
  • It refuses when the context is empty rather than answering from memory.
  • It stays under the length the interface can display without truncation.
  • It never returns the user's own input verbatim as the answer.

None of those care what the answer says. All of them catch real failures, and they run in CI like any other test.

Build the ugly set

The examples in your demo are not a test set. The useful set is the one built from cases that already went wrong: the support ticket where it hallucinated a refund policy, the account with no data, the customer whose name broke the prompt template.

Thirty of those, run on every change, will tell you more than a thousand synthetic prompts. And unlike a benchmark score, each failure points at something you can actually fix.

Watch the rate, not the run

A non-deterministic feature does not pass or fail. It passes at some rate, and the number worth tracking is whether that rate moved after a change.

Run each case a handful of times, record the pass rate, and compare against the previous build. A drop from 96% to 78% on the same set is a real signal even though no individual run is definitive. This is also the only way to catch the quiet regression when a model provider updates something underneath you.

Where to start

  • Write down five properties every acceptable answer must have. Not what it should say, what must be true of it.
  • Collect ten real failures from support, sales calls or your own use. That is your first test set and it costs an afternoon.
  • Run the set on every deploy and record a pass rate. One number, tracked over time.
  • Add each new production failure to the set as it happens. The set is the asset, not the harness.

Tags: ai, process, engineering