flakestat vs Trunk Flaky Tests

What each one is for, what flakestat does differently, and the cases where Trunk Flaky Tests is the better choice.

Trunk is a hosted service with dashboards, org-wide rollups and alerting. It does same-commit detection too, and does the hosted part well. flakestat is a single binary that does the detection part locally, for free, with your test data never leaving your machines.

Both read the JUnit XML your runner already writes. The difference is where the analysis happens, what it is allowed to claim, and whether you can check that it works.

What flakestat does differently#

It ranks by inconsistency, not failure rate. A test that fails every single time is not flaky, it is broken, and the two need completely different fixes. flakestat scores how often a test disagrees with itself between runs, so an always-failing test scores 0.00 and is reported separately as consistently-failing instead of topping the list and costing somebody an afternoon.

Every verdict carries a confidence level. A score of 0.62 from three runs and 0.62 from three hundred are different claims. Classification uses a lower bound on the score rather than the score itself, so a small sample can never reach a confident verdict no matter how dramatic the failures look.

You can verify the detection works. Two commands reproduce a before and after against somebody else’s bug: ConduitIO’s repository at the commit before their own deflaking fix, and at the fix. Three tests come back flaky on one side and stable on the other. No hosted service lets you audit its scoring this way, because the scoring is the product.

git clone https://github.com/rowhitswami/flakestat && cd flakestat
./scripts/reproduce-validation.sh

It refuses to claim causation. When failures cluster on Windows it says they cluster on Windows. When os and arch vary together, as they do on most CI matrices, it says the observations cannot tell which one matters rather than picking one. More on that.

The history is a file you own. Append-only NDJSON in your repository or your artifact store. You can read it, diff it, merge shards with cat, and take it with you. There is no export step because there is nothing to export from.

Side by side#

When flakestat is the better choice#

• Test data cannot leave your infrastructure, whether from regulation, private code, or policy.
• You want to catch a flake before pushing, not after CI reports it.
• The budget for this is zero, or the volume makes per-run pricing awkward.
• You want to read the scoring and check it yourself rather than trust a number.
• You want one tool across a polyglot monorepo, because the input is JUnit XML rather than a language integration.

When Trunk Flaky Tests is the better choice#

Worth saying plainly, because a comparison that finds no case for the alternative is an advertisement.

• You want a dashboard non-engineers can read.
• You need rollups across many repositories and teams.
• You want alerting, ownership routing and SLA tracking.
• You would rather buy the whole workflow than assemble it.

flakestat does not try to do any of that.

Can I use both?#

Yes, and it is a reasonable setup: the hosted service for org-wide visibility, flakestat locally so an engineer can reproduce and confirm a flake in ten minutes without pushing. They read the same JUnit XML and neither interferes with the other.