flakestat

Find flaky tests in Jest

Jest already writes everything flakestat needs. Here is the shortest path from “CI is flaky” to a ranked list.

1. Emit JUnit XML#

Jest
npx jest --reporters=default --reporters=jest-junit

Install jest-junit and point it at a per-run path. Jest's --retryTimes hides flakiness rather than measuring it. Useful to keep CI moving, but it is why flaky suites stay flaky for years. Vitest works the same way with --reporter=junit.

2. Hunt for disagreement#

run the suite 20 times
flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- npx jest --reporters=jest-junit

That runs your suite 20 times on unchanged code and reports which tests changed their answer. Nothing is modified in your repo, and the whole thing is one binary.

3. Read the result#

VERDICT               SCORE  CONF  RUNS  PASS/FAIL  TEST
flaky                  0.62  high    20       14/6  test_checkout::test_race
consistently-failing   0.00  high    20       0/20  test_api::test_broken

A test failing every time scores 0.00 and is reported separately, because it is broken, not flaky, and mixing the two wastes the time of whoever is hunting nondeterminism. Full explanation of the columns.

4. Record CI runs so it keeps working#

A local burst finds the obvious flakes. Recording the CI runs you already pay for finds the ones that only appear on another platform, another runtime, or under load, at no extra compute.

flakestat ingest 'reports/**/*.xml'
flakestat report --top 20

See GitHub Actions or GitLab and others for a complete workflow, and gating CI to stop new flaky tests getting in.

Why not just retry?#

Retrying keeps the pipeline moving, and you should. But it hides the problem. A test that needs a retry still fails sometimes in ways that matter, and a suite with retries on stays flaky for years because nothing ever measures whether it is getting better. Retry to unblock; measure so the list shrinks.

Install#

install
brew install rowhitswami/tap/flakestat

Then the quickstart takes about two minutes.

On this page

1. Emit JUnit XML2. Hunt for disagreement3. Read the result4. Record CI runs so it keeps workingWhy not just retry?Install