Green was never the goal. Catching the bug is.
When AI writes the code and the tests, “all passing” stops meaning “working.” This is the verification layer that tells a real bug from a model just being different — from a Lead Test Engineer who ships the tests, not the theory.
Your suite is green. 90% coverage. The bug shipped anyway.
Copilot or Claude generates forty tests in ninety seconds. They pass. Coverage looks great. But generated tests have a specific failure mode: they assert what the code currently does — so if the function has a bug, the test faithfully locks it in and marks it green.
Add LLM features and it gets worse. The same prompt returns different valid answers, so your exact-match assertions flake, and a red test could be a genuine defect or the model behaving differently. The central question of modern QA is short and brutal: real bug, or model failure? You answer it with evidence, not trust.
A green test makes three different claims. Only the third earns trust.
- 01 — runs
The test executes
It imports, it runs, it goes green. That’s table stakes, not confidence.
- 02 — agrees
The test agrees with the code
It matches what the code does today — bugs included. This is where AI-written tests stop.
- 03 — discriminates
The test tells right from wrong
It fails when the behaviour is wrong. This is the only claim worth having — and the one you have to engineer.
One real defect, two tests, two different verdicts.
Straight from the free lab: a pricing function with a seeded off-by-one bug. Same broken code, both tests below.
// The AI's happy-path test
calculateOrderTotal({
items: [{ unitPriceCents: 2500, quantity: 1 }],
couponCode: "SAVE10",
});
// $25.00 order → 10% discount applied.
// Nowhere near the boundary, so it
// stays green on the broken code. ✔// The discriminating test
calculateOrderTotal({
items: [{ unitPriceCents: 2000, quantity: 1 }],
couponCode: "SAVE10", // exactly $20.00
});
// expects couponDiscountCents: 200
// flawed code checks subtotal > 2000
// instead of subtotal >= 2000
// → returns 0. Off-by-one caught. ✘The failing test is the win. The AI’s test never touches the inclusive boundary, so it can’t see the bug. The verification test hits exactly $20.00 and fails — proving it can tell correct pricing from a real defect. The whole method is one habit: design the discriminating test, then generate.
Run AI as a testing system — in a loop, not a one-shot prompt.
The book’s core workflow. Every code change re-enters it at step one; the verify step is the one you never skip.
Design the discriminating test
Decide what a real defect looks like before you generate anything.
Generate with AI
A prompt contract and context pack, so the output is controlled — not improvised.
Prove it catches the defect
Run it against a known-bad version. If nothing fails, the test proves nothing.
Govern & gate
Independent review, agent separation, CI gates — then merge with evidence.
↺ every change re-enters the loop at 01
The free 10-minute lab
Run a suite that passes 3/3 against code with four real bugs, then watch an independent
verification suite catch every one. No install, no account — clone it and run node run-lab.js all.
A field guide, not a hype tour.
17 chapters, 6 parts
Plus appendices and a glossary written for testers, not ML researchers.
Runnable labs
Unit, web/API, instrument-cluster, and CAN/UDS bus domains — not just toy examples.
Prompt + agent kit
Copyable prompts, a context pack, and governed agent role cards you can drop into your team.
The System Builder
A governed assistant that interviews you, previews a setup, and never writes to live systems.
Local & offline model chapter
For regulated teams who can’t send code or test data to a hosted API.
Reproducible bundle
Rebuild the whole thing with one command: ./verify.sh → BUNDLE GATE: PASS.
The arc, in three moves
Operate, apply, govern
The prompt contract, the operating loop, then verified unit/API/E2E testing, and independent review with agent separation.
Run it privately · orchestrate
Configure and run a local model offline, package a workflow as a reusable skill, and orchestrate independent agents.
Scale without losing control
Decide where AI earns its place, keep the gates honest, and treat AI output as a sample — not a verdict.
AI Testing in Practice
The verification-first field guide + runnable kit for QA engineers who use AI. Honest scope, no hype.
$9.99 Amazon (ebook)
Standards alignment is alignment, not endorsement.
Latest from the site
- Codex or Claude wrote my tests — how do I actually trust them? Oct 30, 2026A green AI-generated suite proves the code does what it already does, bugs and all. The only way to earn confidence is to break the code on purpose and watch the tests fail.
- From experienced tester to AI testing: the roadmap, in order Oct 29, 2026You don't need a data-science degree to test AI. Here's the order to actually learn it, built on what you already know as a tester.
- Real bug or model failure? How to tell the difference in AI tests Oct 27, 2026When an AI test goes red, it's either a genuine defect or the model behaving differently. Here's a practical way to separate the two instead of guessing.
FAQ
Who is this for?
Experienced manual or automation testers (6–10+ years) who are strong on traditional QA but feel like juniors again on the AI/LLM side. If you can read a test and smell a bug, you already have 70% of what you need.
Why is testing AI-written code and LLM output different?
Two reasons. AI-generated tests tend to assert what the code already does — bugs included — so "green" stops meaning "correct". And LLM outputs legitimately vary: the same prompt can return different valid answers, so a red test might be a real bug or just the model being itself. The skill is telling those apart with verification, evals, and prompt contracts instead of brittle exact-match assertions.
Can I try the method before buying?
Yes — the free 10-minute lab. You run a test suite that passes 3/3 against code with four real bugs in it, then watch an independent verification suite catch every one. No install, no account, just Node.
What exactly does the book cover?
A prompt-contract + context-pack workflow, an operating loop (design → generate → verify → govern), verified unit/API/contract testing, browser/E2E flake control, mutation testing, local-model testing for regulated teams, and governed multi-agent setups — across unit, web/API, and automotive instrument-cluster and CAN/UDS bus domains.
Is it tied to any standards?
It is written in alignment with ISTQB CT-GenAI and ISO/IEC/IEEE 29119 (alignment, not endorsement), by a Lead Test Engineer from an automotive and embedded background — so the governance and independence chapters are built for teams that answer to auditors.