Codex or Claude wrote my tests — how do I actually trust them?

A green AI-generated suite proves the code does what it already does, bugs and all. The only way to earn confidence is to break the code on purpose and watch the tests fail.

Short answer: don’t trust them because they’re green. AI writes tests that assert the code’s current behavior — including the bugs. Earn the confidence by mutation testing: change the code on purpose and check the tests fail. The ones that stay green are lying to you.

Last month I let Claude generate a test file for a pricing function. Forty tests, all passing, 94% coverage, took ninety seconds. It felt like a gift. Then I changed a <= to < in the discount tier — a real off-by-one I’d have shipped — and all forty tests still passed. The suite wasn’t testing the pricing logic. It was describing it.

That’s the trap, and it’s specific to generated tests.

Why AI tests pass against broken code

When you write a test, you usually start from “here’s what this should do.” When a model writes a test, it reads the implementation and writes assertions that match what the code does. Those are not the same job. If the function has a bug, the model faithfully writes a test that locks the bug in place and marks it green.

Three failure patterns show up again and again:

Coverage tools can’t see any of this. A line is “covered” the moment it executes — whether or not a single assertion cares about the result. This is why I stopped treating coverage as a quality signal years ago, and why it’s actively dangerous with AI-generated suites: it gives you a big green number that measures the wrong thing.

The test that proves your tests: mutation testing

Here’s the move. Instead of asking “do my tests pass?”, ask “if the code were wrong, would my tests notice?” You answer it by deliberately introducing bugs — mutants — and running the suite against each one.

Every mutant is a tiny, realistic bug. A good test suite kills the mutant — at least one test fails. A mutant that survives is a bug your tests would have let through. The score is simple: killed ÷ total. That number, not coverage, is how much your suite is actually worth.

You don’t do this by hand. The tooling is mature:

A first run on that AI-generated pricing suite looked like this:

$ mutmut run
# ... a few minutes later
$ mutmut results
Killed: 22  Survived: 18  Timeout: 0

# the survivors are the interesting part:
- discount.py:14  `<=` → `<`     (survived)   ← my off-by-one
- discount.py:27  `return total` → `return None`  (survived)
- tax.py:9        `rate * amount` → `rate + amount` (survived)

Eighteen survivors out of forty mutants. The suite that looked airtight was catching a little over half of injected bugs. Now I had a to-do list: eighteen specific holes, each pointing at the exact line and the exact kind of mistake my tests ignored. I wrote real assertions for those, re-ran, and got survivors down to two (both equivalent mutants — changes that don’t actually alter behavior, which you learn to recognize).

The workflow I’d actually recommend

Keep letting the AI write tests. It’s genuinely fast and the boilerplate is fine. Just add one gate:

  1. Generate the tests. Let the model do the grunt work.
  2. Run mutation testing on the module, not just coverage.
  3. Treat every surviving mutant as a missing assertion. Fix the test, not the score.
  4. Set a mutation-score threshold in CI for the code that matters (payments, auth, anything that loses money or leaks data). Start at 70%, raise it as you go.

The point isn’t to mutation-test your entire repo on every commit — that’s slow and you’ll turn it off within a week. Run it on the risky modules, and run it especially on anything an AI wrote, because that’s where the mirror tests hide.

What this doesn’t fix

Mutation testing proves your tests can detect code changes. It does not tell you the original spec was right — if the requirement itself is wrong, you’ll have a high mutation score on the wrong behavior. It’s also not free: runtimes are minutes, not seconds, so it belongs on critical paths and pre-merge, not on every keystroke. And it says nothing about non-deterministic AI outputs — testing an LLM’s answers is a different problem (that’s evals, and telling a real bug from model drift).

What it does do is convert “the tests are green” into “the tests would actually catch a bug,” which is the only version of green worth having.

This verification layer — proving AI-written tests before you rely on them — is the spine of AI Testing in Practice. If you want to feel it before reading, the free lab is a 10-minute run: a green suite sitting on top of four real bugs, and the mutation pass that exposes them.

FAQ

Isn’t high coverage enough if the tests are well written? Coverage tells you a line ran, not that anything checked the result. You can hit 100% coverage with zero meaningful assertions — and AI-generated suites do this constantly. Mutation score measures whether the assertions actually bite.

How is mutation testing different from just writing more tests? More tests can all be mirror tests. Mutation testing finds the specific behaviors nothing is checking, so you add the three assertions that matter instead of thirty that don’t.

My mutation run is too slow to put in CI. Now what? Don’t run it repo-wide. Scope it to the modules that carry real risk and run it pre-merge on changed files. A 90-second mutation run on the payment module is worth more than a 40-minute run across code nobody will break.

Go deeper

This is one piece of the puzzle. “AI Testing in Practice” gives you the full, systematic approach — runnable examples, honest scope, no hype.

Get the book · $9.99 Try the free lab

← All guides