Flaky Test Diagnosis: Find the Cause Before Adding Retries
By Xusheng YangA test fails in CI, passes on retry, and disappears into a green summary. Next week, a customer reports the same intermittent missing result. The retry helped the pipeline continue but did not explain the behavior.
Flaky-test diagnosis investigates why apparently comparable executions produce different outcomes. A changing test environment, shared state, an incorrect wait, or a real product race can all create intermittent failures.
This is part 20 of Testing Methods in Practice. The investigation below uses an illustrative project-creation test, not a reported incident or an executed reliability study.
Preserve the first failure
Save the build, environment, fixture identifiers, test attempt, timing, screenshot, and available trace or logs. Include the expected result and the earliest observation that diverged from it.
Playwright distinguishes passed, flaky, and failed outcomes when retries are configured. Preserve that distinction in reporting: a pass after retry should not erase the initial failure or the opportunity to investigate it.
Avoid repeated reruns against a shared environment before saving evidence. They may mutate the fixture, consume quota, or remove the exact state that made the first attempt informative.
Classify the symptom before choosing a fix
| Symptom | Initial hypothesis | Controlled experiment |
|---|---|---|
| Element appears shortly after timeout | Wait targets the wrong state or service is slow | Observe the actual state transition with bounded timestamps |
| Test passes alone but fails in parallel | Shared mutable fixture | Give cases separate records or workspaces |
| Click hits another element | Overlay or layout state changed | Inspect screenshot and actionability evidence at the click |
| Saved result disappears after reload | Product write or consistency problem | Compare committed state with a fresh read |
| Only CI fails | Environment or resource difference | Compare browser, fonts, locale, CPU pressure, and dependencies |
These are hypotheses, not conclusions. A timeout can reflect a genuinely missing result. Increasing the timeout may hide that symptom without identifying the cause.
Reproduce under recorded conditions
Start with the same build and a fresh, equivalent fixture. Run the case alone and then beside its suspected neighbor. Record first-attempt failures separately from retry outcomes.
Change one factor at a time: shared account, parallelism, response delay, viewport, or dependency stub. If independent data removes the failure, inspect which resource was shared and how it was modified. Serial execution can be a temporary containment, but is not evidence that the product or test is now independent.
The test-data isolation guide gives a resource-ownership model for this experiment. Use exact object identifiers rather than names when correlating runs.
Replace fragile timing with meaningful observation
Suppose the case clicks Create and immediately reads the list's text once. The list updates later, so the assertion compares a stale string.
Use a bounded assertion against the live intended record. Playwright's auto-retrying assertions refetch the locator state while waiting. That addresses observation timing; it does not justify waiting indefinitely or accepting an unrelated record with the same name.
If the service exceeds its expected response budget, keep a separate performance or reliability finding. A longer test timeout can allow diagnosis while still leaving the product issue open.
Do not repair a failure by changing its purpose
Replacing an exact expected project with the first visible project can make the case pass while dropping the original claim. Removing a permission assertion or skipping the failing step has the same problem.
Review the test diff against the requirement. A sound repair improves setup, observation, or application behavior while preserving the intended outcome. If the requirement itself changed, record and approve that decision explicitly.
For a product race, add a coordinated regression at the layer that can control the interleaving. Keep the visible browser recovery case too. The concurrency guide explains why an uncontrolled quick double click is weak evidence of race coverage.
Define repair evidence and temporary quarantine
Repeat the relevant isolated and parallel conditions after the fix, with a chosen run count and configuration recorded in advance. Report observed failure counts and remaining limitations. Zero failures in a finite sample increases confidence in those conditions; it does not prove every schedule is safe.
If a case must be quarantined, give it an owner, issue, review date, and replacement coverage or an explicit accepted gap. Silent indefinite skipping removes the signal the test was meant to provide.
Your exercise: select one intermittent failure, preserve its first-attempt evidence, write two competing hypotheses, and design one experiment that distinguishes them. Finish with a cause, a reviewed repair, and the conditions under which you verified it. Use run evidence to keep the user-visible observations attached to the investigation.
Return to the start: Exploratory testing, or browse all Testing Methods articles.