October 27, 2026

Real bug or model failure? How to tell the difference in AI tests

When an AI test goes red, it's either a genuine defect or the model behaving differently. Here's a practical way to separate the two instead of guessing.

Short answer: a failing AI test is a real bug when the output violates a rule that must always hold (a contract, invariant, or ground-truth check), and a model failure when the output is merely different but still valid. The fix is to stop asserting exact strings and start asserting the properties that define “correct” — then a red test tells you which of the two you’re looking at.

Most teams can’t answer the question because their tests only know one thing: “did the output equal this exact value?” For deterministic code that’s fine. For AI it’s useless — the same prompt can return three different, equally correct answers, so exact-match tests flap at random and you learn to ignore red. That’s the real danger: a suite you’ve trained yourself not to trust.

Why “did it match?” is the wrong question

Consider a support-bot that classifies a message and drafts a reply. You wrote:

def test_reply():
    out = bot("my card was declined")
    assert out == "Please check your card balance and try again."

This passes today and fails tomorrow when the model says “It looks like your payment didn’t go through — could you verify your card details?” — which is better. The test is red; nothing is broken. That’s a model failure (of your test, not the system).

The three checks that actually separate bug from drift

Replace the exact-match with assertions on what must be true regardless of wording.

1. Contract checks — the output’s shape must hold

def test_reply_contract():
    out = bot("my card was declined")
    assert out.intent == "billing"            # classification is deterministic enough to pin
    assert 1 <= len(out.text) <= 600          # length bounds
    assert "http" not in out.text             # never invent a link
    assert out.pii_leaked is False            # never echo card numbers

If any of these fail, it’s a real bug — the system broke a promise, no matter how fluent the text was.

2. Property / invariant checks — rules that are true for every valid answer

import re

def test_no_hallucinated_amounts():
    out = bot("what's my balance?")
    # the bot must not state a dollar figure it was never given
    assert not re.search(r"\$\s?\d", out.text)

Properties catch the failures exact-match never will: a confident, well-written, completely made-up answer.

3. Graded evals — “is this answer acceptable?” instead of “is it identical?”

For open-ended output, score it against a rubric (keyword coverage, an embedding-similarity floor, or an LLM-as-judge) and assert a threshold, not equality:

def test_reply_quality():
    out = bot("my card was declined")
    score = judge(out.text, rubric="acknowledges the decline, asks one clarifying question, no blame")
    assert score >= 0.8

A drop below the threshold is a real regression. A score of 0.85 vs 0.92 is just drift — ignore it.

A decision table you can keep

The one habit that fixes most of this

Pin your model and prompt versions in the test run, and separate two signals: “did a contract break?” (hard fail, blocks CI) from “did quality move?” (tracked metric, alerts but doesn’t block on noise). The day you split those two, “real bug or model failure?” stops being a guess and becomes something your pipeline answers for you.

FAQ

Is a flaky AI test always a bad test? Usually, yes — if it flaps on identical input, you’re asserting the wrong thing. Move from exact-match to property and graded-eval assertions.

Can I just use an LLM to judge the output? Yes, as one signal. LLM-as-judge is great for open-ended quality but is itself non-deterministic — pin its model, give it a strict rubric, and pair it with deterministic contract checks so a hard rule never depends on a soft judge.

How is this different from normal testing? Deterministic systems have one correct output; you assert equality. AI systems have a set of valid outputs; you assert the properties that define the set. Everything else — CI, mutation testing, coverage — still applies.

← All guides