flakestat

Find flaky tests in pytest

pytest already writes everything flakestat needs. Here is the shortest path from “CI is flaky” to a ranked list.

1. Emit JUnit XML#

pytest
pytest --junitxml=reports/junit.xml

pytest writes JUnit XML natively, so there is no plugin to install. If you already use pytest-rerunfailures, keep it. Retries keep the pipeline moving, but they hide the flakiness rather than measuring it. flakestat measures it so the list actually shrinks.

2. Hunt for disagreement#

run the suite 20 times
flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- pytest --junitxml='{junit}'

That runs your suite 20 times on unchanged code and reports which tests changed their answer. Nothing is modified in your repo, and the whole thing is one binary.

3. Read the result#

VERDICT               SCORE  CONF  RUNS  PASS/FAIL  TEST
flaky                  0.62  high    20       14/6  test_checkout::test_race
consistently-failing   0.00  high    20       0/20  test_api::test_broken

A test failing every time scores 0.00 and is reported separately, because it is broken, not flaky, and mixing the two wastes the time of whoever is hunting nondeterminism. Full explanation of the columns.

4. Record CI runs so it keeps working#

A local burst finds the obvious flakes. Recording the CI runs you already pay for finds the ones that only appear on another platform, another runtime, or under load, at no extra compute.

flakestat ingest 'reports/**/*.xml'
flakestat report --top 20

See GitHub Actions or GitLab and others for a complete workflow, and gating CI to stop new flaky tests getting in.

Why not just retry?#

Retrying keeps the pipeline moving, and you should. But it hides the problem. A test that needs a retry still fails sometimes in ways that matter, and a suite with retries on stays flaky for years because nothing ever measures whether it is getting better. Retry to unblock; measure so the list shrinks.

Install#

install
brew install rowhitswami/tap/flakestat

Then the quickstart takes about two minutes.

On this page

1. Emit JUnit XML2. Hunt for disagreement3. Read the result4. Record CI runs so it keeps workingWhy not just retry?Install