October 29, 2026

How to test an LLM or RAG app when the same prompt gives different valid answers

Use property checks, scored evals, and a golden dataset — never exact-match assertions. Non-determinism isn't the problem; asserting the wrong thing is.

Short answer: stop asserting what the answer says and start asserting what must be true of any valid answer. Property checks (format, grounding, forbidden patterns) are deterministic and belong in CI. Scored evals with a threshold handle open-ended quality. Exact-match assertions on non-deterministic output are just noise.

The complaint I hear most from testers moving into LLM work: “I can’t write a test — I don’t know what the right answer is.” That’s not an LLM problem. It’s an assertion-design problem. Write deterministic tests your whole career and you start thinking the spec for “acceptable output” is always a single string.

Why does exact-match fail for LLM and RAG output?

A RAG pipeline answers “What’s the refund policy?” three different valid ways:

• “You have 30 days to return items for a full refund.”
• “Returns are accepted within 30 days of purchase.”
• “Our return window is 30 days from the delivery date.”

All three are correct. An exact-match test passes once, then flaps forever on every slightly different rephrasing. Worse — if you pin it to the first string you saw, the test will fail when the model is upgraded to one that writes better answers. You end up locking in regressions.

The fix isn’t “find the right string.” It’s to stop asserting strings. (If your suite is already flapping across the board, the root cause is almost always bad assertions rather than the model itself — worth diagnosing that first.)

What is a property check and how do you write one for RAG?

A property is something that must be true of every valid answer. Those are what you assert.

import re
from urllib.parse import urlparse

def test_refund_policy_properties():
    out = rag_app.query("What's the refund policy?")

    # must mention the number 30 — it's in every source doc
    assert "30" in out.answer

    # must not invent a different timeframe
    assert not re.search(r"\b(7|14|60|90)\s*-?\s*day", out.answer)

    # must cite at least one source chunk
    assert len(out.source_chunks) >= 1

    # must stay under 300 chars — this is a snippet, not an essay
    assert len(out.answer) <= 300

    # must not hallucinate links the retriever never returned
    retrieved_domains = {urlparse(c.url).netloc for c in out.source_chunks}
    for url in re.findall(r"https?://\S+", out.answer):
        assert urlparse(url).netloc in retrieved_domains

All of these are deterministic. They never flap, they run in CI at test speed, and when one fails you know exactly what broke. The last check — “did you invent a URL?” — catches a failure class that no exact-match test would ever find: a fluent, well-written, completely made-up link.

How do you build a scored eval suite for open-ended quality?

Property checks catch hard failures. For quality — does the answer actually answer the question well? — you need scored evals.

Start with a golden dataset: 50–100 question/answer pairs where “answer” means expected properties plus a rubric, not an expected string.

{"question": "What's the refund policy?", "must_contain": ["30"], "rubric": "Clearly states the return window. No extra conditions invented. Concise.", "min_score": 0.80}
{"question": "How do I cancel my subscription?", "must_contain": ["cancel", "account"], "rubric": "Step-by-step process, no invented phone numbers or URLs.", "min_score": 0.80}

Then run it:

import json
from pathlib import Path

def run_eval_suite(dataset_path: str, hard_fail_on_props: bool = True):
    results = []
    for line in Path(dataset_path).read_text().splitlines():
        case = json.loads(line)
        out = rag_app.query(case["question"])

        # hard checks first — any failure here is a bug, full stop
        props_ok = all(kw in out.answer for kw in case["must_contain"])
        if hard_fail_on_props:
            assert props_ok, f"Property check failed for: {case['question']}"

        score = judge(out.answer, rubric=case["rubric"])
        results.append({
            "question": case["question"],
            "props_ok": props_ok,
            "score": score,
            "pass": score >= case["min_score"],
        })

    pass_rate = sum(r["pass"] and r["props_ok"] for r in results) / len(results)
    assert pass_rate >= 0.90, f"Eval suite: {pass_rate:.0%} pass rate, need 90%"
    return results

The 90% threshold is deliberate. Demand 100% and a single noisy judge call blocks your deploy. Set it at 80% and real regressions hide in the noise. 90% holds the line without making the suite brittle.

When does LLM-as-judge actually work?

When you give it a tight rubric and pin both model and temperature. It falls apart on vague questions.

Bad prompt: “Is this answer good? Rate 0–10.”

Good prompt:

JUDGE_PROMPT = """\
You are a strict evaluator. Respond ONLY with a decimal between 0.0 and 1.0.

Rubric: {rubric}

Answer to evaluate:
{answer}

Score (0.0 = fails rubric entirely, 1.0 = satisfies it completely):"""

def judge(answer: str, rubric: str) -> float:
    resp = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=10,
        temperature=0,
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            rubric=rubric, answer=answer,
        )}],
    )
    return float(resp.content[0].text.strip())

Pin temperature=0 on the judge. It still isn’t perfectly stable — same input, slightly different float — but stable enough to assert a threshold. Pin the model version too. When you upgrade the judge, re-baseline your thresholds against the existing dataset before trusting the new scores.

RAGAS does this for you out of the box with four standard RAG metrics: faithfulness, answer relevancy, context precision, context recall. I’d use it for RAG evals specifically. Just don’t replace your hard property checks with RAGAS scores — a faithfulness of 0.72 doesn’t tell you whether you hallucinated a phone number.

How do you test retrieval and generation failures separately?

RAG has two distinct failure modes: the retriever fetches the wrong chunk, or the LLM hallucinates despite correct chunks. End-to-end tests can’t tell the difference.

def test_retrieval_surfaces_right_doc():
    chunks = retriever.fetch("What's the refund policy?", top_k=5)
    # at least one chunk must contain the return window
    assert any("30" in c.text or "return" in c.text.lower() for c in chunks)
    # shipping docs shouldn't rank in the top 3 for a refund question
    assert all(c.category != "shipping" for c in chunks[:3])


def test_generation_stays_grounded():
    """Feed the LLM controlled context and verify it doesn't invent facts."""
    question = "What is the late-fee rate?"
    context = [Chunk(text="Late fees are 2% per month.", source="policy.pdf")]

    out = llm.generate(question, context=context)
    assert "2%" in out
    assert not re.search(r"\b[3-9]\d*%", out)  # no invented rate

Separating these turns “the RAG app is broken” into “the retriever is broken” or “the generator is hallucinating.” That distinction changes how you fix it — and it changes how long the fix takes.

What doesn’t this solve?

A golden dataset built when everything is working tends not to cover the edge cases that break things later. The 50 questions you write at launch won’t include the 11 weird phrasings real users try in month three. Every production bug should add two new eval cases: the exact input that triggered it, plus one related case you can imagine.

Property checks also don’t catch semantic failures. An answer that contains “30” but inverts the policy — “returns are not accepted within 30 days” — passes the keyword check. The judge is supposed to catch that, but only if your rubric says “correctly states the direction of the policy.”

The eval-design problem — deciding what properties and rubrics to write for your system — is the hard part, and there’s no shortcut. The free test lab has a working RAG eval harness with a small golden dataset you can clone and adapt. AI Software Testing in Practice covers the full pyramid — from fast property checks through eval suites through regression tracking across model upgrades.

FAQ

Should I set temperature to 0 to make LLM tests deterministic? For contract and property checks, yes — pin temperature=0 so fast unit tests don’t flap. But run your eval suite at your production temperature too; a bug that only appears at high temperature is still a bug.

How many examples do I need in a golden eval dataset? Fifty is enough to start. You’re looking for coverage of your failure modes, not statistical significance. Add ten more every time a bug reaches production that your suite missed.

What is RAGAS and should I use it for RAG testing? RAGAS is an open-source eval framework that scores retrieval-augmented pipelines on faithfulness, answer relevancy, context precision, and context recall. Use it for your eval suite, but don’t gate CI on its scores alone — pair it with deterministic contract checks for hard failures.

← All guides