Flaky tests: a triage rule that actually gets them fixed
A flaky test passes and fails without any code change. One flaky test is an annoyance. A handful of them teaches the whole team to hit "re-run" by reflex, and once that habit sets in, real failures slip through with everyone else.
So treat flaky tests as bugs with a deadline.
The rule
- Spot it. A test that fails, then passes on re-run with no code change, is flaky until proven otherwise.
- Quarantine within a day. Move it out of the blocking suite. Never just add retries and forget it: retries hide the problem and slow every build.
- Fix or delete within two weeks. A quarantined test nobody fixes is a deleted test with extra steps. If it isn't worth fixing, delete it on purpose.
- Track the count. The target is zero. Put the number somewhere the team sees it.
The usual causes
| Symptom | Usual cause | Fix |
|---|---|---|
| Fails only alongside other tests | Shared state: globals, singletons, static fields, a shared DB row, an env var | Fresh state per test; unique test data; reset in afterEach |
| Fails only in parallel | Two tests using the same port, file, table or account | Unique resources per worker; separate schemas |
| Fails around midnight, month end or in another time zone | Code reads the real clock | Inject or fake the clock; never call now() in logic under test |
| Fails on slow machines | Fixed sleeps or tight timeouts | Wait for a condition, not for a length of time |
| Fails only in CI | Different OS, locale, time zone or filesystem case sensitivity | Pin time zone and locale in test config; match CI and local environments |
Reproducing it
Most flaky tests are hard to fix because they're hard to reproduce. Run the single test in a loop until it fails, then run it alongside its neighbors in the same order CI uses. Most test runners can do both: repeat a test N times, and run in a fixed or random order with a seed you can replay.
Once you can make it fail on demand, the table above usually tells you where to look.
Closing it out
When you fix one, write down the cause in the commit message. After a few months, the pattern tells you which of the causes your codebase is prone to, and that's where a lint rule or a shared test helper pays off.