Flake Hunt
Finds out why a test is intermittent, instead of retrying it until the pipeline turns green.
When to reach for it
The second time a test fails and then passes on a re-run — before anyone adds a retry, a sleep, or a skip with no note.
What changes
- You finish with a named cause — shared state, a real clock, an unordered collection, an unawaited promise, a port collision — rather than a retry count.
- The flake is made reproducible on demand through repetition, seeding, forced ordering or constrained parallelism, so “fixed” becomes something you can test.
- Failure rate is measured before and after with the same repeat count, so “7 red in 200 runs” becomes “200 green in 200 runs” instead of one lucky pass.
- When the cause turns out to be a genuine race in the product, it is filed as a product bug rather than quietly silenced as a test problem.
- Retries, fixed sleeps and skipped tests are removed on the way out instead of accumulating as permanent scar tissue.
Pairs with
- Failing Test FirstMakes the agent prove the bug exists — with a test that fails for the right reason — before it is allowed to fix anything.
- Behaviour, Not CoverageTurns “add some tests” into tests that state what the code promises, instead of tests that exist to move a percentage.
- Fixture DietCuts test setup down to the few facts the test actually depends on, so the test says what it means.
Flake Hunt
An intermittent test is a message about the system. Read it before silencing it.
1. Make it fail on demand
You cannot fix what you cannot summon. Try, in order:
- Run that test alone a hundred times and count the failures
- Run the file in a different order, and with a different random seed
- Run the suite with one worker, then with many
- Run it on a loaded machine, or with a slower disk
- Run it against a different clock: a month boundary, a daylight saving change, one minute before midnight
Write down the failure rate you measured. That number is the baseline you will be judged against later.
2. Work through the usual suspects
Nearly every flake is one of these:
- Time — real clocks, timeouts sitting near the wire, dates that roll over during the run
- Order — map or object iteration, a query with no sort, a test that only passes because its neighbour ran first
- Shared state — module-level caches, singletons, rows or files left behind, environment variables set by another test
- Concurrency — a promise nobody awaited, a fixed sleep standing in for a condition, a listener attached after the event fired
- Environment — a hardcoded port, a colliding temporary path, a network call you had forgotten was real
3. Prove the cause by causing it
Do not settle for a plausible story. Force the condition and watch the test fail every time: freeze the clock on the boundary, reverse the collection order, drop to a single worker, occupy the port. If you cannot make it fail on purpose, you have not found it yet.
4. Fix at the source
Inject the clock instead of reading it. Sort explicitly instead of trusting insertion order. Build state per test instead of sharing it. Wait for a condition with a timeout instead of sleeping for a guess. Ask the system for a free port instead of picking one.
5. Re-measure the same way
Use the repeat count from step one. "Green once" says nothing about a test that was already green most of the time.
6. Decide whether the bug is in the product
If the cause is a real race, a real ordering assumption or a real timeout that production shares, the test was right and the code is wrong. File it as a product bug and say so plainly.
Rules
- Do not add a retry. A retried flake is the same bug with a larger budget and a quieter voice.
- Do not add a sleep. Wait on the condition you actually need, with a timeout that fails loudly when it is not met.
- Do not skip or delete the test without recording what behaviour is now uncovered and what would bring it back.
- Do not declare victory on one green run.
- Do not blame the build machine without evidence. Slowness is a trigger, not a cause; the assumption it exposed is the cause.