When a suite stops meaning anything#
The day someone says “just rerun it, that one is flaky” and everyone nods, a red build stops meaning anything. The suite still runs. It just no longer tells you the truth.
The five causes#
| Cause | The sign | The fix |
|---|---|---|
| Timing | Passes locally, fails in CI | Wait for the condition — never sleep |
| Shared data | Passes alone, fails in the suite | Each test owns its data |
| Test order | Fails when order changes | Shuffle order in CI, fix what breaks |
| Outside services | Fails when they deploy | Stub them, except in a small smoke set |
| A real bug | Rare, shows up under load | Celebrate — the test found something |
Don’t sleep. Wait for the condition.#
// Slow when it's ready in 50 ms. Still flaky at 3.1 s.
Thread.sleep(3000);
// Returns as soon as it's true. Fails with the last value it saw.
await(() -> getClaim(id), c -> "SETTLED".equals(c.getStatus()),
Duration.ofSeconds(30), Duration.ofMillis(200));Put the last value in the error message. “Timed out” starts an investigation. “Timed out, last status was PENDING_APPROVAL” usually ends it.
Each test owns its data#
A test that passes alone but fails in the suite is reading another test’s data. Create the data in setup with a unique name, and delete it in teardown with alwaysRun = true — so a failing test still cleans up.
Shuffle, then run in parallel#
Random order finds tests that depend on each other. Parallel runs find shared state. Both will break things at first. That is the point — fix what breaks instead of turning them off.
Retry once, and count it#
Retrying a network blip is fine. Retrying until green hides real bugs forever. Allow one retry, and show every retry on a dashboard. A test that needed a retry did not really pass.
Quarantine, with a deadline#
- Move the flaky test out of the gate, so green is honest again.
- Keep running it on a schedule.
- Give it an owner and a fix-by date.
- Keep the quarantine short. If it grows, stop and fix the suite.
Give the suite a time budget#
Fail the build when the suite gets slower than its budget, and print the ten slowest tests on every run. A few tests usually take most of the time — and they are easy to find once you look.