Flaky test hunter
Finds why a test passes and fails intermittently by looping runs, shuffling order and checking timing, shared state, time zones, network and unordered collections, then fixes the cause. Use when CI is flaky or a test fails only sometimes.
flaky-test-hunterInstall
Run one command — it saves SKILL.md into your agent's skills folder. Restart the agent and the skill is picked up automatically.
mkdir -p ~/.claude/skills/flaky-test-hunter && curl -fsSL https://anymodel.org/en/skills/debugging/flaky-test-hunter/SKILL.md -o ~/.claude/skills/flaky-test-hunter/SKILL.md
mkdir -p ~/.agents/skills/flaky-test-hunter && curl -fsSL https://anymodel.org/en/skills/debugging/flaky-test-hunter/SKILL.md -o ~/.agents/skills/flaky-test-hunter/SKILL.md
New-Item -ItemType Directory -Force "$HOME\.claude\skills\flaky-test-hunter" | Out-Null; Invoke-WebRequest https://anymodel.org/en/skills/debugging/flaky-test-hunter/SKILL.md -UseBasicParsing -OutFile "$HOME\.claude\skills\flaky-test-hunter\SKILL.md"
New-Item -ItemType Directory -Force "$HOME\.agents\skills\flaky-test-hunter" | Out-Null; Invoke-WebRequest https://anymodel.org/en/skills/debugging/flaky-test-hunter/SKILL.md -UseBasicParsing -OutFile "$HOME\.agents\skills\flaky-test-hunter\SKILL.md"
Any other agent: save the file into its skills folder, or give it the link to the file: https://anymodel.org/en/skills/debugging/flaky-test-hunter/SKILL.md
SKILL.md
Flaky Test Hunter
A flaky test is a real bug, either in the test or in the code. Retrying until green hides it. Find the nondeterminism and remove it.
Workflow
1. Measure the flakiness
- Identify the test: name, file, how often it fails in CI (look at the last runs), and the exact failure message. Different messages mean different causes.
- Run it alone many times to get a failure rate:
# JavaScript (vitest/jest)
for i in $(seq 1 50); do npx vitest run path/to/file.test.ts -t "test name" || echo "FAIL $i"; done
# Python
pytest path/to/test_file.py::test_name --count=100 # needs pytest-repeat
for i in $(seq 1 50); do pytest -x -q path/to/test_file.py::test_name || echo "FAIL $i"; done
# Go
go test ./pkg -run TestName -count=200 -race
- If it never fails alone but fails in the full suite, the cause is shared state or ordering. If it fails alone, the cause is inside the test or the code.
2. Vary the environment to find the axis
Change one thing at a time and see which change makes it fail more often:
- Order: run the whole file or suite in random or reversed order (
pytest --random-orderwith pytest-random-order, replay with--random-order-seed=<seed>;jest --randomize,go test -shuffle=on). Record the seed printed so you can replay it. - Parallelism: run serially and then in parallel (
-n 4,--runInBand,-p 1). Failures only under parallel runs mean shared resources (ports, files, database rows, temp dirs). - Load: run under CPU stress or with fewer cores. Timing-dependent tests fail when the machine is slow.
- Time: change the clock and time zone (
TZ=Pacific/Kiritimati,TZ=America/Los_Angeles), run near midnight, month ends, DST changes, leap days. - Data: random seeds, generated ids, locale, current directory.
3. Check the usual causes
- Sleeps and races.
sleep(1)or fixed timeouts waiting for async work. Replace with waiting on a condition: poll with a deadline, await the promise, use the event or callback. - Shared state. Global variables, singletons, module caches, static fields, environment variables, files in a shared temp path, database rows left from another test, mocks not restored. Each test must set up and clean up its own state.
- Time and dates. Dependence on "now", the local zone, or a date arithmetic crossing a boundary. Inject a clock or freeze time.
- Network. Real calls to external services, DNS, ports already in use. Use fakes, recorded responses, or port 0 and read the assigned port.
- Ordering. Iterating sets, maps or dictionaries, unordered database results without
ORDER BY, filesystem listing order, goroutines or threads finishing in any order. Sort before comparing or assert on sets. - Randomness. Unseeded random, UUIDs in snapshots, floating point equality. Seed it or compare with tolerance.
- Resources. Leaked handles or ports from previous tests, tests relying on a warm cache, async work from a finished test hitting the next one.
- Concurrency in the code itself. If the failure shows a true race (lost update, double processing), the product code is wrong; do not just fix the test.
4. Make the failure reproducible
Once you have a failing seed or condition, keep it. Reduce it to the smallest failing combination (two tests, one seed). Add temporary logging at the point of divergence and compare a passing run with a failing run.
5. Fix the cause
- Prefer deterministic setup over waiting: inject dependencies, freeze clocks, use isolated data per test.
- If a wait is unavoidable, wait for the specific condition with a bounded deadline.
- After the fix, repeat the loop from step 1 with at least the same count, plus under the condition that exposed it. Zero failures over 100 runs is the bar for a typical 5% flake.
6. Quarantine only as a last resort
If you cannot fix it now:
- Ask the user before disabling or skipping a test.
- Mark it with the runner's skip or quarantine mechanism, with a comment containing a ticket reference and the observed failure rate and cause hypothesis. Do not skip silently.
- Do not add automatic retries as a "fix"; if the team uses retries, keep the test visible as flaky.
Report format
- Test, failure rate before, message.
- Cause found, with the evidence (the seed, order or condition that triggers it).
- Fix and failure rate after (runs count).
- Anything quarantined and the ticket.
Pitfalls
- A passing run proves nothing about a flaky test; only many runs do.
- Do not increase timeouts as the first move; it makes the test slower and still racy.
- Do not delete the test because it is annoying.
- Do not rerun CI repeatedly to "see if it passes" instead of reproducing locally.
Source: AnyModel skills catalog · License: CC BY 4.0
Connect through AnyModel API
This skill runs on whatever model your agent uses. Point Claude Code, Codex or Cursor at AnyModel to run it on Claude, GPT, Gemini and more with one key — pay per token, no subscription.
AnyModel