November 1, 2026

Flaky AI tests are usually bad tests: how to stabilize a non-deterministic suite

Most AI test flakiness isn't the model's fault — it's exact-match assertions on output that was never guaranteed to stay the same. Fix the assertion, not the AI.

Short answer: most AI test flakiness comes from two sources — exact-match assertions on output that was never guaranteed to match, and missing pins on model version and temperature. Fix those two things and 80% of your flapping tests stabilize. The remaining 20% are genuine non-determinism, and the answer there is a different kind of assertion, not a different kind of model.

The “AI is untestable” complaint comes up constantly in QA teams that are six months into an LLM project. The suite runs, half the tests pass, half fail, nobody trusts it, CI goes red, someone marks the failures “known flaky” and disables them. Now you have a test suite that runs green and tells you nothing.

That’s not an AI problem. That’s a test-design problem that AI exposed.

Why do AI tests flap when nothing has changed?

Two reasons, and they’re distinct enough that fixing the wrong one wastes a week.

Reason 1: the assertion is wrong. You called the API once, got a string back, and wrote assert response == "Here are the three items: apples, bananas, oranges." The model gives the same information every time but rephrases it — “The three items are apples, bananas, and oranges.” or “I found three items: apples, bananas, oranges.” Your assertion fails. The model isn’t broken. The test is.

Reason 2: the test isn’t pinned. You’re running at the default temperature (often 0.7–1.0), and you haven’t nailed down the model version. Even at temperature=0, some providers add noise. At temperature=1.0, the same prompt produces genuinely different outputs run-to-run. If your test doesn’t control these, you’re testing a moving target.

How do you tell which problem you have?

Run the failing test 10 times in a row against the same model, same prompt, with temperature=0 and an explicit model version pinned. Count the failures.

• 10/10 fail: the assertion is fundamentally wrong — it’s asserting something that was never stable.
• 0/10 fail: your test was running at high temperature, or the model version was drifting. Pinning fixed it.
• 3/10 fail at temperature=0: you’ve found genuine residual non-determinism (most providers don’t guarantee exact reproducibility even at temp=0). You need a statistical assertion or a threshold check.

The diagnosis tells you which fix to apply.

How do you fix wrong assertions?

Stop asserting the string. Assert the properties that must be true of any valid output.

# Before: brittle, flaps on every rephrasing
def test_item_list():
    response = ai_app.query("List the three items.")
    assert response == "Here are the three items: apples, bananas, oranges."

# After: stable, tests what actually matters
def test_item_list_properties():
    response = ai_app.query("List the three items.")

    # all three items must be present, order doesn't matter
    for item in ["apple", "banana", "orange"]:
        assert item in response.lower()

    # must not invent a fourth item
    unexpected = ["grape", "mango", "pear", "melon"]
    assert not any(u in response.lower() for u in unexpected)

    # must stay concise — this isn't an essay prompt
    assert len(response) < 300

These assertions are deterministic. They never flap, they run at CI speed, and when one fails you know exactly what class of failure occurred. The first version was testing the model’s wording. These versions test whether the model was correct.

The same logic applies to structured output. If your app returns JSON, don’t assert the whole serialized string — assert the schema and the values.

import jsonschema

ITEM_SCHEMA = {
    "type": "object",
    "required": ["items", "count"],
    "properties": {
        "items": {"type": "array", "items": {"type": "string"}, "minItems": 1},
        "count": {"type": "integer", "minimum": 1},
    },
}

def test_structured_output_shape():
    result = ai_app.query_json("List the items.")
    jsonschema.validate(result, ITEM_SCHEMA)
    assert result["count"] == len(result["items"])

Schema validation never flaps. A miscount between count and len(items) is a real logic bug — and it’ll catch it reliably every time.

How do you pin model version and temperature?

This is the simplest fix and the most commonly skipped one.

import anthropic

client = anthropic.Anthropic()

def query_for_tests(prompt: str) -> str:
    """Use this wrapper in all tests — never call the client directly."""
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",  # explicit version, not "claude-haiku-latest"
        max_tokens=512,
        temperature=0,                       # deterministic mode for unit tests
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.content[0].text

Two things to note. First, pin the full model ID — claude-haiku-4-5-20251001, not a latest alias. When the provider bumps the alias, your test behavior changes silently. Second, run your property checks at temperature=0, but keep a separate eval suite that runs at your production temperature. A bug that only surfaces at temperature=1.0 is still a bug, and you want to find it before users do.

What about tests that still flap after pinning?

At temperature=0, most models produce identical output for identical inputs most of the time. “Most of the time” isn’t all the time — some providers are explicit that exact reproducibility isn’t guaranteed. If you’re still seeing flakiness after pinning, you have two options.

Option A — statistical assertion. Run the query N times and assert a pass rate, not a single result.

def test_sentiment_is_positive_most_of_the_time():
    results = [ai_app.classify_sentiment("Great product!") for _ in range(20)]
    positive_rate = results.count("positive") / len(results)
    assert positive_rate >= 0.90, f"Only {positive_rate:.0%} positive — expected ≥90%"

Twenty runs is usually enough to distinguish a real regression (drops to 60%) from noise (stays above 90%). Don’t run this in fast CI — it’s for a nightly or pre-release gate.

Option B — property check on the distribution. If your app is supposed to return a confident label 90% of the time, write that as the spec directly. A test that runs once and sometimes passes isn’t a spec — it’s a guess.

Which flakiness isn’t worth fighting?

Genuine creative variation in open-ended generation. If your feature is “give me five marketing tagline ideas,” the outputs should vary. Testing that they don’t vary is testing the wrong thing. What you can assert: every tagline is under 15 words, none contain the competitor’s name, all use imperative tense. Stable properties on variable output. Writing those for LLM and RAG output takes some practice but it’s the same skill.

Also not worth fighting: latency variation. A 200ms swing on a generation call doesn’t mean anything. Set a hard ceiling (5s, 10s, whatever production SLA is) and assert that — not an exact time.

What this doesn’t fix

Bad property coverage. If you write properties that are too loose — “response must be non-empty” — you’ll have a green suite that misses every quality regression. Properties have to actually capture the contract of the feature.

The deeper problem: figuring out what properties to assert for your specific system is the design work. There’s no universal answer. The properties for a customer-support chatbot are different from those for a code-generation tool. You have to understand what “correct” looks like for your feature before you can write a test that catches when it isn’t.

That design work is what AI Software Testing in Practice is built around — working out the right assertion strategy per feature type, with examples across real AI product categories. The free test lab has working property-check templates you can adapt today.

FAQ

Are flaky AI tests a sign that the model is broken? Rarely. Most AI test flakiness is caused by exact-match assertions on non-deterministic output, or by tests that don’t pin temperature and model version. Fix the assertion and the test stops flapping without touching the model.

Should I quarantine flaky AI tests instead of fixing them? Quarantine for 48 hours while you diagnose — don’t leave them skipped permanently. A flaky test is either a bad assertion (fix it now) or real non-determinism you need to understand (build a statistical version). Skipping it forever means a category of bugs stops being caught.

How do I stabilize an AI test that sometimes passes and sometimes fails on the same input? First check if you’re pinning model version and temperature=0. If yes and it still flaps, the test is asserting something that varies legitimately — rewrite it as a property check or a scored threshold instead of an exact match.

← All guides