8 September 2026 · Rohit Swami

Validating a flaky-test detector

Building it took about a week. Working out whether it actually worked took considerably longer, and produced better material.

I once spent an afternoon on a flaky test that wasn't flaky.

It sat at the top of a list ranked by failure rate, which seemed like a sensible way to rank flaky tests until I noticed it had failed on every run for three weeks. It wasn't nondeterministic. It was broken, and had been broken so consistently that the ranking put it exactly where the most nondeterministic test in the suite should have been.

That is the whole design argument for the tool I ended up writing, so I'll state it once and move on: flakiness is inconsistency, not failure. A test that fails every time tells you nothing about nondeterminism. What you want to know is how often a test disagrees with itself. Score that instead, and the always-failing test drops to zero and gets filed under a different heading, where it belongs.

flakestat does that. It reads the JUnit XML your runner already emits, scores transitions rather than failures, and attaches a confidence level to every verdict, because 0.62 from three runs and 0.62 from three hundred are not the same claim. One Go binary, no account, nothing leaves the machine.

That part was a week. This post is about the rest.

The awkward question#

How do you know a flaky-test detector works?

It's a genuinely annoying question. You can write fixtures with planted flakes, and I did. A test that fails on a coin flip is easy to detect and proves almost nothing. Real flakiness is rarer, weirder, and entangled with the environment. And the failure mode you care about is silent: a detector that misses things looks identical to a codebase that isn't flaky.

What I wanted was a case where somebody else had established the ground truth. Not my fixture, not my judgement.

ConduitIO had one. Their contributors filed an issue documenting several flaky tests, named them, and later merged a commit that fixed them. That gives two commits, one either side of a repair I had nothing to do with, and a detector that works should say different things about them.

You can run that yourself:

git clone https://github.com/rowhitswami/flakestat && cd flakestat
./scripts/reproduce-validation.sh

Two minutes. It fetches their repository at both commits, runs the same sampling protocol at each, and prints the table. It also runs in CI on every build, so if their commits move or the script rots, it breaks on my end before it breaks on yours.

I'd rather be judged on that than on anything I assert. The protocol was written down before any of it ran, which is the only way the results mean anything.

Three things I got wrong#

The bugs were more instructive than the features, and they share a shape.

The measurement was the signal#

I added a --parallel mode to run the suite several times at once. Pointed at spf13/cobra, it reported TestDeadcodeElimination as flaky at 0.60.

It isn't. The test builds a binary at a fixed path, and concurrent copies were deleting each other's build. The tool had manufactured the exact phenomenon it was built to detect, then reported it with a straight face.

Candidates found under --parallel are now re-run sequentially and demoted if they don't reproduce. I kept the mode rather than deleting it, because failing to reproduce in five runs is weak evidence of innocence, not proof. But the default is sequential, and the default is trustworthy.

Duplicates suppress flakiness#

CI artifacts get collected twice. A job re-runs, two aggregators pick up the same shard, someone re-uploads. I assumed the risk was inflated confidence: more observations agreeing, tighter bound, smaller p-value, all resting on evidence that only existed once.

It does the opposite, and it took a measurement to notice.

A duplicate carries its original's timestamp. So it sorts next to its original, and a duplicate always agrees with itself. Duplication doesn't add noise, it adds artificial agreement. On a test failing 4 of 12 runs, ingesting the same reports twice moved the score from 0.64 to 0.30 while the apparent evidence doubled.

The error runs towards false negatives. Duplication makes flaky tests look stable, which is the direction that loses them quietly. Every observation now carries the identity of the execution it describes, so a copy counts once. A genuine retry, which really did run the tests again, still counts.

The same invariant, three times#

Two outcomes are evidence of nondeterminism only if everything that could legitimately change the result was held constant. Obvious once written down. I broke it three times.

First branches: a test passing on main and failing on an in-progress feature branch, read as one chronological series, looks like repeated disagreement.

Then platforms. A test that always fails on Windows and always passes everywhere else is perfectly deterministic on every platform. It scored 0.45, with an explanation asserting "direct evidence of nondeterminism". That one surfaced within minutes of pointing the tool at its own CI matrix.

Then the display layer, which kept grouping by branch after scoring had moved on to execution context. The verdict was right; the evidence printed underneath it was wrong, marking "flip on identical code" at exactly the boundaries the verdict had ruled out. Arguably worse than being wrong outright, since the number a careful reader checks against was the one still misbehaving.

There is now one definition of comparability and everything that displays transition evidence calls it. Three implementations that agreed by convention was the actual bug.

A night spent failing to find something#

One hole I couldn't argue away: every flaky test the tool had found was already known to somebody. Conduit's were filed. So were the ones in the regression corpus. A detector that only rediscovers documented bugs is a plausible detector, not a demonstrated one.

So I pointed it at six active Go projects overnight. Five finished the protocol: 335 executions, each a fresh process, shuffled, with a recorded seed so anything found would be reproducible by its maintainer rather than "run 47 failed". The sixth had a ten-minute baseline and would have taken fifty hours.

It found nine real flaky tests.

Seven were already filed. One had a fix open and unmerged since March. One had been caught by litestream's nightly race-detector sweep the week before. flakestat hit it in ten runs, which is a reasonable argument for concentrated repetition over nightly sampling, and not an argument for discovery.

The two that weren't known looked promising for about an hour. One was an unreported intermittent failure in a 9.1k-star consensus library, clean against four separate cross-checks: no issue, no pull request, no commit, and absent from the project's own community-maintained list of flaky tests.

Then I ran the experiment that mattered.

test alone,  -count=1                  25/25 pass
test alone,  -count=3                  25/25 pass
full suite,  -count=1, no shuffle      12/12 pass
full suite,  -count=1, shuffled        12/12 pass
full suite,  ./... -count=1            10/10 pass
full suite,  ./... -count=3, shuffled   2/31 FAIL

It only failed under the exact conditions my harness imposed. Their CI runs -count=1. The other unreported finding turned out to be the same category: a package that can't be run with -count>1 at all, which is a real problem worth telling them about and is not flakiness.

So: zero previously-unknown flaky tests.

Stopping there was tempting. It was four in the morning and the result was exactly what I had been hoping for. Had I stopped, I would have filed a bug report that misattributed its own cause, and written a post claiming the tool found something nobody knew about.

That's the --parallel mistake again. Same shape, different layer: the apparatus producing the signal. I've now made it twice in one project, which suggests it's less a bug than a standing hazard of measuring anything.

What I'll claim#

Two things the evidence supports.

Speed. Concentrated repetition beats nightly sampling. Litestream's sweep needed a week of nightly runs to surface a flake; ten runs on a laptop found it.

Restraint. Across two clean repositories, 202 executions and 231 distinct tests, it reported nothing at all. It refused to classify a dramatic ten-minute hang on two observations, correctly, because two observations aren't a verdict. It called nine always-failing tests broken rather than flaky. A detector that finds something everywhere you point it is just a mirror.

What I won't claim is discovery. It has never surfaced a flaky test nobody had already filed, and the write-up says so in the same words I'd use if it had.

I think that's the more useful post anyway. There is a lot of tooling that tells you what it found. There is less that tells you what it looked for and missed.

flakestat is MIT-licensed and reads the JUnit XML your test runner already writes. brew install rowhitswami/tap/flakestat, npm install --save-dev flakestat or pip install flakestat. Documentation · Source · Full validation record · The overnight hunt