Documentation

Install flakestat#

Same binary whichever route you pick. No runtime to install alongside it.

brew install rowhitswami/tap/flakestat

npm install --save-dev flakestat
# or, without installing:
npx flakestat report

pip install flakestat

go install github.com/rowhitswami/flakestat/cmd/flakestat@latest

curl -sSfL https://raw.githubusercontent.com/rowhitswami/flakestat/main/scripts/install.sh \
  | sh -s -- -b /usr/local/bin

docker run --rm -v "$PWD:/workspace" ghcr.io/rowhitswami/flakestat report

Or download a binary directly from Releases. macOS, Linux and Windows, on x86_64 and arm64. The binary is statically linked and has no runtime dependencies.

Verify the install#

flakestat version
flakestat --help

Which install should I use?#

Next#

Head to the quickstart. Two commands, and you'll know whether your suite is flaky.

Quickstart#

Two ways to use flakestat. They answer different questions, and most projects end up doing both.

1. Hunt: is this suite flaky right now?#

Runs your test command N times on unchanged code and reports what disagreed. Use it when you already suspect a suite, or before opening a pull request.

flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- pytest --junitxml='{junit}'

flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- npx jest --reporters=jest-junit

flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- gotestsum --junitfile='{junit}' -- ./...

flakestat hunt --runs 20 --junit 'target/surefire-reports/*.xml' \
  -- mvn -q test

{run} becomes the run number so each run writes its own report, and {junit} becomes that path inside your command. Everything after -- is your command, passed through untouched.

Let it detect your setup#

flakestat init      # writes .flakestat.json
flakestat hunt      # no flags needed

init looks at your project and writes the command and report path once, so the day-to-day invocation is a single word. See configuration.

2. Ingest: which tests are flaky over time?#

Records the CI runs you already pay for. This is the one that catches flakes that only appear on one platform, one runtime, or under CI load, and it costs no extra compute.

# in CI, after your tests run
flakestat ingest 'reports/**/*.xml'

# any time
flakestat report --top 20

See tracking flakiness over time for where to keep the history, and GitHub Actions for a complete workflow.

What you get back#

VERDICT               SCORE  CONF  RUNS  PASS/FAIL  TEST
flaky                  0.62  high    20       14/6  test_demo::test_flaky_race
suspect                0.08   low     6        5/1  test_api::test_timeout
consistently-failing   0.00  high    20       0/20  test_demo::test_broken

Read the columns in reading a report, or ask about one test with flakestat explain.

If it finds nothing#

That is a real result, not a failure. It means nothing disagreed in that many runs. A test that fails 2% of the time will usually survive 20 runs untouched. Raise --runs, or record CI history where the sample grows for free.

Reading a report#

Six verdicts, one score and one confidence level. Here is what each of them is claiming.

VERDICT               SCORE  CONF  RUNS  PASS/FAIL  TEST
flaky                  0.62  high    20       14/6  test_demo::test_flaky_race
suspect                0.08   low     6        5/1  test_api::test_timeout
consistently-failing   0.00  high    20       0/20  test_demo::test_broken

Verdicts#

Score is not a failure rate#

Score is how often a test disagrees with itself: 0.62 means roughly six of ten consecutive runs changed their answer. A test that fails every single time scores 0.00, because its outcome never disagrees with itself. It is broken, and it shows up as consistently-failing so you can fix it as the different problem it is.

Confidence#

Confidence is low, medium or high depending on how much evidence sits behind the score. A score of 0.62 from three runs and the same score from three hundred are very different claims, and a small sample can never reach high no matter how flaky the test looks.

Classification uses a lower bound on the score rather than the score itself, so a verdict requires evidence rather than a lucky flip in a short run. The mechanics are in how scoring works.

Other output formats#

flakestat report --format json      # for tooling
flakestat report --format markdown  # for a PR comment or wiki
flakestat report --all              # include stable and unscored tests
flakestat report --top 20           # worst 20 only

Hunting flaky tests locally#

Run the suite many times on unchanged code and see what disagrees.

flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- pytest --junitxml='{junit}'

flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- npx jest --reporters=jest-junit

flakestat hunt --runs 20 --junit 'reports/junit-{run}.xml' \
  -- gotestsum --junitfile='{junit}' -- ./...

flakestat hunt --runs 20 --junit 'target/surefire-reports/*.xml' \
  -- mvn -q test

How many runs do I need?#

It depends entirely on how rare the flake is. These are measured against synthetic tests with known failure probabilities:

Chasing one test#

flakestat hunt --runs 50 --junit 'reports/junit-{run}.xml' \
  -- pytest tests/test_checkout.py::test_race --junitxml='{junit}'

Narrowing to one test is usually much faster per run, so you can afford far more runs, which is exactly what a rare flake needs.

If it finds nothing#

That is a real result. It means nothing disagreed in that many runs, which is evidence the test is not flaky at a rate this sample could detect. It is not evidence the test is clean. Raise --runs, or let CI history accumulate.

Scoring the whole history instead#

By default hunt scores only the burst it just ran. Add --history to score everything recorded so far, burst and CI runs together.

flakestat hunt --runs 20 --history

Tracking flakiness over time#

A burst proves flakiness exists. History measures it, and catches flakes a local run never will.

flakestat ingest 'reports/**/*.xml' \
  --commit "$GIT_SHA" --branch "$BRANCH"

Commit and branch default to the current git checkout, so in most CI setups you can omit them. Recording the commit is what lets flakestat tell genuine flakiness, same code and different result, from a regression someone later fixed.

History lives in .flakestat/runs.ndjson, one JSON object per line. Because it is append-only NDJSON, results from parallel CI shards concatenate with cat and no merge step.

Re-ingesting is safe#

Every observation carries the identity of the execution it describes, so an artifact uploaded twice, a re-run aggregation step, or a shard collected by two jobs is counted once, while a genuine retry, which really did run the tests again, still counts.

Copies are ignored on read, so nothing is required of you. flakestat compact removes them from the file as well, which is worth doing when the file is the record you keep.

Where to keep the history#

matrix jobs → per-job NDJSON artifacts → aggregate job
  → fetch history branch → merge + compact → analyze
  → commit back to the history branch

concurrency:
  group: flakestat-history-writer
  cancel-in-progress: false

flakestat's own CI workflow implements this end to end, including re-merging a push that lost a race against another run.

Trimming the file#

flakestat compact             # drop duplicate executions
flakestat compact --dry-run   # report what would go, change nothing

compact refuses to rewrite a log containing unreadable lines, since reading skips those and rewriting would delete evidence you might still recover.

Recording where tests ran#

Attach context to observations and flakestat will tell you where failures concentrate.

flakestat ingest 'reports/**/*.xml' \
  --dimension os=windows \
  --dimension runtime.version=3.13 \
  --dimension database=postgres-17

Where the failures concentrate

  os=windows
    failures here:  12 / 12 (100.0%)
    elsewhere:      0 / 24 (0.0%)
    difference:     +100.0 pp

Two rules that keep it honest#

• Nothing is scraped. Only a whitelist of known CI variables is read, and only JUnit <property> names flakestat recognizes. The process environment is never walked, so secrets, tokens and build ids cannot end up in your history.
• Host details are recorded only when flakestat ran the tests. If one job downloads other jobs' artifacts and ingests them centrally, pass --no-host: otherwise the aggregator's platform is stamped onto results from everywhere else, and analysis can conclude the exact opposite of the truth.

Canonical keys#

Anything else you pass is a user dimension and is analysed normally: database version, browser, feature flag, region.

Correlation, never causation#

flakestat says failures cluster on Windows. It will not claim Windows is why. Observational data cannot distinguish a cause from anything perfectly correlated with it, and when several dimensions vary together, as os and arch usually do on a CI matrix, it reports all of them and says the observations cannot tell which one matters.

Three guards stop it producing confident nonsense over sparse data: an evidence floor, an effect-size floor, and a Benjamini–Hochberg correction for how many dimensions were tested. Testing os, arch, runtime, browser and database will eventually turn up something that looks significant purely by chance.

Gating CI without a permanently red build#

Accept today's flakiness, then fail only on what is new or measurably worse.

Most repos already have flaky tests when they adopt a tool like this. Failing on all of them means a red build on day one, and a deleted gate by day three. So check is a ratchet rather than a threshold.

flakestat check --update-baseline    # once; commit .flakestat/baseline.json
flakestat check --fail-on-new        # in CI from then on

Keeping jitter out of your builds#

Scores move a little as evidence accumulates. --regression-delta (default 0.1) is how much an accepted test has to worsen before it counts as a regression, so ordinary drift doesn't fail builds.

Tightening the ratchet#

Tests that stop being flaky are reported as FIXED. That is your cue to re-run --update-baseline so the newly clean test can never silently regress.

FIXED   test_api::test_timeout      was 0.31, now 0.00
NEW     test_cart::test_checkout    0.42 (high)

1 newly flaky test. Baseline has 4 accepted.

Unblocking the pipeline#

Emit a skip list your runner already understands, so merges are unblocked while you fix things.

flakestat quarantine --format pytest -o quarantine.txt
pytest @quarantine.txt

flakestat quarantine --format jest -o quarantine.json

flakestat quarantine --format go

flakestat quarantine --format yaml -o quarantine.yaml

A workable rhythm#

1. Quarantine what is blocking merges today.
2. Commit a baseline so nothing new gets in.
3. Fix the worst-ranked test, remove it from quarantine, update the baseline.
4. Repeat. The list only shrinks, because the gate stops it growing.

Explaining a verdict#

Every verdict can be interrogated. Nothing is asserted that the evidence does not show.

flakestat explain test_checkout_flow

tests/test_checkout.py::test_checkout_flow
  id 4c1f9a2e77b3d810

  Classification: flaky
  Score:          0.55
  Confidence:     high  (lower bound 0.402 over 41 transition(s))

  Observations:   42
  Passed:         27
  Failed:         15
  Failure rate:   35.7%

  Transitions:               41
  Same-commit disagreements: 9
  Seen across:               6 commit(s), 2 branch(es)

  History  (last 20 of 42, oldest first)

  P F P P F P F F P P P F P P F P P F P P
    ^ ^   ^ ^ ^   ^     ^ ^   ^ ^ ^   ^

  P pass   F fail   -  skip   ^ flip   ! flip on identical code

Reading the strip#

Markers appear only between observations that are actually comparable: same branch, same platform, same runtime. A test that always passes on Linux and always fails on Windows is deterministic, so its strip carries no flip markers even though the outcomes differ.

Options#

flakestat explain test_name --json        # the same evidence, for tooling
flakestat explain test_name --history 80  # widen the strip
flakestat explain 4c1f9a2e77b3d810        # by id, when names are ambiguous

If a name matches several tests, flakestat lists the candidates with their ids rather than guessing which you meant.

GitHub Actions#

Record every run, comment on the pull request, and gate on regressions rather than on flakiness itself.

- name: Run tests
  run: pytest --junitxml=reports/junit.xml
  continue-on-error: true

- uses: rowhitswami/flakestat@v0.2.1
  with:
    args: ingest 'reports/**/*.xml'
    comment: true          # post/update a PR comment
    annotations: true      # annotate new flakes in the diff

That records the run, writes a report to the job summary, and posts it as a pull request comment. A re-run updates the same comment rather than adding another.

The intended sequence#

- run: pytest --junitxml=reports/junit.xml
  continue-on-error: true          # record the run even when it fails

- uses: rowhitswami/flakestat@v0.2.1
  with: { args: "ingest 'reports/**/*.xml'" }

- uses: rowhitswami/flakestat@v0.2.1
  with: { args: "check --fail-on-new", comment: true }

Inputs#

Outputs: flaky-count, new-flaky-count, report-file.

Permissions#

permissions:
  contents: read
  pull-requests: write   # only if comment: true

Matrix builds#

Let each matrix job ingest its own results, then merge the NDJSON in an aggregate job. That way every observation records the platform it actually ran on, rather than the aggregator's.

- uses: rowhitswami/flakestat@v0.2.1
  with:
    args: >-
      ingest 'reports/**/*.xml'
      --dimension os=${{ matrix.os }}

- uses: actions/upload-artifact@v4
  with:
    name: observations-${{ matrix.os }}
    path: .flakestat/runs.ndjson

See recording where tests ran for why this matters, and tracking over time for where to keep the merged history.

GitLab, CircleCI and others#

Nothing is GitHub-specific. Install the binary and call it.

Provider context (run id, job id, attempt and shard) is detected automatically for GitLab CI, CircleCI, Buildkite, Jenkins and Azure Pipelines.

test:
  script:
    - pytest --junitxml=reports/junit.xml || true
    - curl -sSfL https://raw.githubusercontent.com/rowhitswami/flakestat/main/scripts/install.sh
        | sh -s -- -b /usr/local/bin
    - flakestat ingest 'reports/**/*.xml'
    - flakestat check --fail-on-new
  artifacts:
    paths: [.flakestat/runs.ndjson]
    when: always

- run:
    name: Tests
    command: pytest --junitxml=reports/junit.xml
    when: always
- run:
    name: Record flakiness
    command: |
      curl -sSfL https://raw.githubusercontent.com/rowhitswami/flakestat/main/scripts/install.sh \
        | sh -s -- -b /usr/local/bin
      flakestat ingest 'reports/**/*.xml'
      flakestat check --fail-on-new
    when: always

stage('Test') {
  steps {
    sh 'pytest --junitxml=reports/junit.xml || true'
    sh 'flakestat ingest "reports/**/*.xml"'
    sh 'flakestat check --fail-on-new'
  }
}

steps:
  - command:
      - "pytest --junitxml=reports/junit.xml || true"
      - "flakestat ingest 'reports/**/*.xml'"
      - "flakestat check --fail-on-new"
    artifact_paths: ".flakestat/runs.ndjson"

Sharded pipelines#

Let each shard ingest its own results and concatenate the NDJSON afterwards. Each shard then records the context it actually ran in, rather than the aggregator's.

# in each shard
flakestat ingest 'reports/**/*.xml' --dimension ci.shard="$CI_NODE_INDEX"

# in the aggregate job
cat shard-*/runs.ndjson > .flakestat/runs.ndjson
flakestat compact
flakestat report --top 20

No CI at all#

flakestat is just a binary reading local files. Point it at any directory of JUnit XML you have, from any source.

flakestat ingest 'archive/2026-*/junit.xml'
flakestat report

Command reference#

Nine commands. Run flakestat <command> -h for the full flag list of any of them.

Scoring flags#

Accepted by hunt, report, explain, check and quarantine. Defaults are calibrated against a ground-truth corpus, not chosen by intuition. See how scoring works before changing them.

Output flags#

Command-specific flags#

hunt#

ingest#

check#

Configuration#

Write the command and report path once, then run a single word.

flakestat init            # detects your project
flakestat init --project pytest   # or skip detection
flakestat init --force    # overwrite an existing config

{{
  "command": ["pytest", "--junitxml={junit}"],
  "junit": "reports/junit-{run}.xml",
  "runs": 20,
  "threshold": 0.10
}}

With that file in place, the whole invocation becomes:

flakestat hunt

Precedence#

Flags always win over the file, so a one-off flakestat hunt --runs 50 works without editing anything. Nothing in the file is required, and every key has a default.

What gets written where#

How scoring works#

Five steps, and a live demo you can poke at to see each of them.

Try it#

Click any run to cycle it between pass, fail and skip. The verdict below is computed with the same rules the binary uses.

The demo covers one execution context on one commit, which is where the interesting behaviour is. The binary additionally weights same-commit and cross-branch evidence, and decays by age, in steps 3 and 4 below.

The five steps#

1. Group by execution context. Two outcomes are only comparable if branch, os, arch and runtime were the same. Without this, interleaving platforms makes a test that always fails on Windows and always passes elsewhere read as constant disagreement, a phantom signal that has caught this project twice.
2. Count transitions. Every adjacent pair within a context that disagreed is a flip. Failure rate is deliberately not used, which is why an always-failing test scores zero.
3. Weight them. A flip on the same commit counts three times a cross-commit one, because the latter may be a regression someone fixed. Disagreement off the default branch counts for half, because failures there are expected.
4. Decay by age. Age is measured in commits, not observations, so a 200-run burst on one commit uses all of its evidence instead of only the tail.
5. Take a lower bound. Classification uses a Wilson score lower bound with an effective sample size, so a verdict requires evidence rather than a lucky flip in a short run.

Skips are not outcomes#

pass    -> pass evidence
failure -> fail evidence
error   -> fail evidence
skip    -> neither

A skipped run is not evidence either way, so it never becomes an outcome a neighbour can disagree with, and never enters the failure rate. P S P is one comparison between two passes, not two disagreements.

Where failures concentrate#

Association analysis is separate and never touches the score. A test is flaky because its outcomes demonstrate flakiness; associations only say where that flakiness concentrates. Findings must clear an evidence floor, an effect-size floor, and a Benjamini–Hochberg correction for how many dimensions were tested. See recording where tests ran.

Going deeper#

The design notes record the reasoning in full, including the mistakes that shaped it: a recency weight that made 100 runs no better than 10, a parallel mode that manufactured the flakiness it reported, and three separate occasions where incomparable observations were compared. The validation record covers whether any of it holds up against somebody else’s bugs.

Supported test runners#

Anything that writes JUnit XML, which in practice is everything.

pytestJestVitest go testJUnit 5TestNG SurefireRSpecPHPUnit MochaPlaywrightCypress xUnitNUnitcargo test

Dialect tolerance#

JUnit XML has no official schema, so every framework writes it slightly differently. The parser handles both root elements, nested suites, locale-formatted durations like 4,521, bytes that are illegal in XML 1.0, and Surefire's <flakyFailure> markers, which are a direct flakiness signal and are read as one.

Playwright emits <error> for some failures and <failure> for others; both count as failures. A <skipped> test counts as neither.

No JUnit XML?#

flakestat falls back to exit codes and reports suite-level flakiness. That is less precise, because you learn the suite is flaky rather than which test, but it works with anything that returns a status code.

flakestat hunt --runs 20 -- ./run-tests.sh

Frequently asked questions#

Short answers. Each links to the longer explanation.

Does my test data leave my machine?#

No. flakestat is a binary that reads local files and writes a local file. There is no network call, no account and no telemetry.

My test failed 100% of the time. Why is the score zero?#

Because it isn't flaky, it's broken. flakestat scores how often a test disagrees with itself, not how often it fails, so a consistently failing test scores zero and appears as consistently-failing, a different problem with a different fix. See reading a report.

How many runs do I need?#

Depends on the flake rate. ~25% is reliably caught within 20 runs; 10% usually needs 50 or more; below about 5% a local burst is the wrong instrument and CI history is right. insufficient-data means exactly that. There is not enough evidence yet.

Can I use it with a monorepo or sharded CI?#

Yes. Let each shard ingest its own results and concatenate the NDJSON. It is append-only and one observation per line specifically so that works with cat and no merge step.

Does re-running a job double-count?#

No. Each observation carries the identity of the execution it describes, so the same artifact ingested twice counts once, while a genuine retry, which really did run the tests again, counts as the second execution it is.

Will it slow down my CI?#

ingest parses XML files you already produce; it is milliseconds. hunt runs your suite N times and costs exactly that, which is why it is a local tool rather than something on every build.

What if my runner doesn't write JUnit XML?#

Almost all of them can, with one flag or one reporter package. See supported runners. Failing that, exit-code mode still gives suite-level results.

Is it free?#

Yes. MIT licensed, free at any volume, and no per-seat or per-run pricing, because it runs on your own machines, so there is nothing to meter.

How is this different from just retrying failed tests?#

Retries hide flakiness; flakestat measures it. A retried test still costs time and still fails in ways that matter. Retry to keep the pipeline moving, and measure so the list actually shrinks. Gating on regressions is how you stop it growing.

Why not just use a hosted service?#

Use one if you want dashboards, org-wide rollups and alerting. They do that well. flakestat is for the case where you want detection to be local, free and yours. See how it compares.