Lean Team
Lean Team blog

5 mistakes AI coding agents make, and how to catch them before merge

AI coding agents write code fast. They also make the same handful of mistakes over and over. None of them are exotic, which is exactly why they slip through review: the diff looks plausible, the summary sounds confident, and the reviewer is busy.

Here's what to check every AI-assisted diff for before it merges.

1. Invented APIs

How to spot it: calls to functions, methods, flags or packages that don't exist in the versions you actually depend on. The name is usually reasonable, which is the problem.

What to do: search the codebase and the library's docs for every unfamiliar call. Delete or replace anything you can't find. If the tests "pass," check that they actually execute that line.

2. Swallowed errors

How to spot it: empty catch blocks, except: pass, or an error that's logged and then ignored so the function carries on with bad state.

What to do: handle the error, rethrow it, or write a comment explaining why ignoring it is safe. "It made the test pass" isn't a reason.

3. Fake tests

How to spot it: tests with no assertions, tests that only check a mock was called, or tests that mirror the implementation line by line so they can never fail.

What to do: rewrite them to assert behavior someone outside the function would notice: the return value, the saved record, the HTTP response.

4. Confident wrong summaries

How to spot it: "All tests pass" or "Fixed the bug," with no command output shown.

What to do: run the command yourself and read the output. Every time. An agent's summary of a test run isn't a test run.

5. Silent behavior changes inside "refactors"

How to spot it: a diff labeled as a refactor that also changes defaults, error messages, log lines or the order things happen in.

What to do: move behavior changes into their own commit, with a test that shows the new behavior on purpose.

The one-question test

Before merging, ask:

Would I be comfortable defending this diff in review if I had written every line myself?

If not, it isn't ready, no matter who or what wrote it.

Making it stick

A checklist only helps if it's used on every diff, not just the ones that look suspicious. Two things that help:

The full 12-point slop filter, the pre-merge checklist and a ready-to-use code-reviewer subagent for Claude Code are free on GitHub: lean-team-products/ai-code-review-gates. Copy them into your repo and you're set up in two minutes.