Flaky Test Management and Test Reliability Questions

Direct answer: Restrict retry eligibility to a maintained ALLOWLIST of failure signatures with prior evidence of being transient (network timeouts, specific 5xx codes, recognized intermittent-infra patterns), never a blanket "retry on any failure," and make every retry outcome, not just the final result, visible in both the CI report and the bug tracker, so masking is a visible risk to monitor rather than an invisible one.

Structured elaboration

Detection heuristics for eligible failures: match the failure's normalized signature (per the fingerprinting approach) against a maintained allowlist, network-timeout exceptions, specific HTTP status codes (502, 503, 429), and recognized intermittent-infrastructure-error patterns (a specific, previously-diagnosed flaky-runner symptom, the third example category named alongside network timeouts and 502/503 responses). A failure NOT matching any allowlisted signature is NOT retry-eligible by default, this is the single most important design decision distinguishing a narrow, defensible retry policy from a broad, masking-prone one.

Retry policy parameters: max attempts capped small (2-3), exponential backoff with jitter (per the retry-decorator patterns covered throughout this topic), and a PER-TEST retry-budget cap over a rolling window (not just per-failure), so a test that needs retry-rescuing repeatedly over time gets flagged for escalation rather than the policy quietly absorbing it forever.

Metrics to collect: retry-rate (fraction of runs needing at least one retry, a direct cost/friction signal); pass-after-retry rate PER TEST (the specific signal distinguishing a test that's occasionally, genuinely rescued from one that's chronically being rescued, which should trigger investigation regardless of how the aggregate retry-rate looks); retries-per-test (how many attempts were typically needed, a granularity finer than a bare rate that can reveal a test needing MORE retries over time even if its overall pass-after-retry rate looks stable).

Surfacing retry outcomes in reports and bug-tracking systems: every CI report shows, per test, whether it passed clean, passed-after-N-retries, or failed after exhausting retries, as THREE distinct, visually distinguishable states, not collapsed into a binary pass/fail that would hide the retry entirely; a test whose pass-after-retry rate crosses a threshold over a rolling window automatically opens (or updates) a tracking issue, giving the retry-masking risk a concrete, actionable artifact rather than a number nobody looks at.

Avoiding masking application bugs, concretely: (1) the allowlist itself must be reviewed periodically (per the ongoing-audit pattern covered throughout this topic), since a signature that WAS reliably transient can start correlating with a real, new bug over time, and yesterday's safe allowlist entry isn't guaranteed to stay safe forever; (2) the per-test retry-budget cap (not just per-failure) ensures a test that's being rescued REPEATEDLY, even if each individual rescue looks like an isolated, allowlist-matching transient failure, eventually gets forcibly escalated rather than the policy treating each occurrence as independently acceptable; (3) the three-state (clean/rescued/failed) reporting, rather than binary pass/fail, is what keeps the masking risk VISIBLE to anyone looking at CI results day-to-day, not just to someone deliberately auditing retry logs.

Worked example: a specific network-timeout signature has been reliably transient for months (a low, stable pass-after-retry rate across dozens of tests matching it). Then, following a specific backend deployment, tests matching that SAME signature start showing an elevated pass-after-retry rate specifically for one particular test, crossing the per-test threshold within a week; the automated tracking-issue creation flags it promptly, and investigation confirms the deployment introduced a genuine, intermittent backend regression that happens to produce the SAME timeout signature the allowlist has long trusted as transient, exactly the "yesterday's safe entry isn't guaranteed to stay safe" pattern the periodic allowlist review and per-test threshold are both designed to catch.

Trade-offs & pitfalls: an allowlist that's built once and never revisited is a slowly-decaying safety guarantee, not a permanent one; treat "when was this allowlist last reviewed against recent pass-after-retry data" as a standing metric worth tracking on its own, since an allowlist review that never happens is functionally the same risk as having no allowlist review process at all, just with a false sense of security in the meantime.

Direct answer: At a 97% historical pass rate on a release-blocking smoke test, the right immediate action is (B), quarantine it out of the blocking gate into a visible, owned quarantine, while filing (D), a proper fix, in parallel; blindly retrying (A) treats the symptom without evidence it's not masking something real, and doing nothing is not on the table since it's actively blocking releases right now.

Structured elaboration

A decision framework:

1. Gather the metrics first: 97% pass rate over how many runs, and what does the FAILURE pattern look like, clustered in time/environment, or genuinely random? A 97% rate over only 10 runs (meaning roughly 0.3 failures expected, so 1 observed failure) carries far less statistical confidence than 97% over 200 runs; per the statistical-detection framework, don't treat a small-sample rate with the same confidence as a well-sampled one.
2. Classify the failure signature: does the failure match a KNOWN, previously-diagnosed pattern, or is it new? A known, previously-understood transient pattern is a stronger case for a scoped retry allowance; a new, unclassified failure signature is a stronger case for NOT retrying blind, since retrying an unknown failure risks masking a genuine new regression.
3. Weigh the options against the release context:

• (A) Add retries: reasonable ONLY if the failure matches a known-transient signature and the test isn't gating something where a false pass is unacceptable; otherwise risky, since it treats the symptom without addressing (or even confirming) the cause.
• (B) Quarantine: removes it from BLOCKING but keeps it visibly running and tracked, the right default when you don't yet have enough evidence to be confident a retry is safe, and you need to unblock the release NOW.
• (C) Remove it from the gate entirely: similar to quarantine but implies a longer-term or permanent decision that this test shouldn't gate releases; appropriate if, on reflection, this test's coverage doesn't actually warrant blocking power (a judgment call about the test's VALUE, not just its reliability).
• (D) File a fix: always warranted in parallel with whichever immediate mitigation is chosen, since none of A, B, or C actually resolves the underlying flakiness.

4. Stakeholders to consult: the test's owning team (do they have context on recent changes that might explain new flakiness), the release manager (how much release-schedule pressure exists right now, which affects how much risk is acceptable), and, for anything touching a business-critical path, a product or compliance stakeholder who can weigh in on acceptable risk for THIS specific release.
5. Acceptable risk: this depends on what the smoke test actually verifies, a 97% pass rate on a smoke test covering a rarely-changed, low-risk area carries different acceptable risk than the same rate on a test covering active development in a critical path; the decision isn't just about the NUMBER, it's about what that number means for THIS specific test's coverage.
6. Immediate mitigations if you can't fix it quickly: quarantine with a TIGHT SLA (days, not the standard 30-day default, given release pressure), paired with a manual verification step covering what the quarantined test would have caught, so removing its blocking power doesn't silently remove ALL coverage for that release specifically.

Worked example: investigating shows the 97% figure comes from 300 runs (9 failures), and all 9 failures share the same "element not found, timing" signature that's previously been traced to a known async-render race, not a new regression. Given release pressure is currently high (a hotfix needs to ship today) and the failure pattern is well-understood and low-risk, the decision is: quarantine (B) immediately to unblock the release, paired with a manual smoke-check of the specific flow this test covers before shipping (the immediate mitigation), and file a proper fix (D) targeting the underlying async-render race, with a 5-day SLA given the urgency rather than the standard 30-day default.

Trade-offs & pitfalls: choosing (A) retries as a first response without doing steps 1 to 3 first is the most common shortcut under release pressure, and it's exactly the shortcut that risks masking a genuine new regression behind a convenient-looking "it's just flaky" narrative; the fastest SAFE path under pressure is still quarantine-plus-manual-verification, not blind retry, even though retry feels like less work in the moment.

Direct answer: A custom pytest marker with metadata (issue, owner, and typically a flakiness threshold or a strict flag) lets a test declare its own known-flaky status directly in code, and a CI pipeline reads that marker via pytest's marker-introspection API to change behavior BEFORE and AFTER the run, without needing a separate, out-of-band registry for tests that are already known to be flaky.

Approach and code

import pytest

@pytest.mark.known_flaky(issue="PROJ-123", owner="team-a", max_reruns=2)
def test_checkout_confirmation_banner():
    #... test body that occasionally races an async render...
    assert confirmation_banner.is_visible

Registering the marker (so pytest doesn't warn about an unknown mark, and so its metadata is documented) in conftest.py or pytest.ini:

# pytest.ini
[pytest]
markers =
    known_flaky(issue, owner, max_reruns): mark a test as known-flaky, with a linked
        issue, an owning team, and an allowed rerun count for CI to act on.

How a CI pipeline reads this metadata

A pytest plugin hook (pytest_collection_modifyitems, run after test collection but before execution) inspects each collected test item for the known_flaky marker (named to avoid colliding with the
widely-used pytest-rerunfailures plugin, which already defines its own @pytest.mark.flaky(reruns=..., reruns_delay=...)
marker) and adjusts execution behavior accordingly:

# conftest.py
import pytest


def pytest_collection_modifyitems(config, items):
    for item in items:
        marker = item.get_closest_marker("known_flaky")
        if marker is None:
            continue
        max_reruns = marker.kwargs.get("max_reruns", 0)
        owner = marker.kwargs.get("owner", "unassigned")
        issue = marker.kwargs.get("issue")
        # 1) allow reruns: attach rerun metadata for pytest-rerunfailures
        # to pick up, scoped ONLY to marked tests, not applied blanket.
        item.add_marker(pytest.mark.flaky(reruns=max_reruns, reruns_delay=1))
        # 2) skip in gating builds: if this run is a merge-blocking gate, exclude
        # known-flaky tests from the blocking result while still running them
        # for visibility (report-only), rather than letting them block merges.
        if config.getoption("--gating-mode", default=False):
            item.user_properties.append(("gating_excluded", True))
        # 3) annotate reports: attach owner/issue so the JUnit/HTML report links
        # directly to the tracking ticket for anyone triaging a red run.
        item.user_properties.append(("flaky_owner", owner))
        item.user_properties.append(("flaky_issue", issue))

Key implementation notes: the marker is read at COLLECTION time (pytest_collection_modifyitems), not at run time, so the rerun and gating-exclusion behavior is decided before any test executes, keeping the logic centralized in one hook rather than scattered per-test. max_reruns is intentionally per-test (via the marker's kwargs), not a single suite-wide constant, since different known-flaky tests legitimately warrant different rerun budgets depending on how well-understood their flakiness pattern is. The gating_excluded property (attached via user_properties, which pytest surfaces into JUnit XML) is what a CI system's gating logic reads to decide whether a marked test's failure blocks a merge, WITHOUT skipping the test outright, so its results still populate the flakiness dashboard.

Edge cases and complexity: a test marked known_flaky that STOPS actually being flaky (its owner fixed it but forgot to remove the marker) will keep getting rerun-allowance and gating-exclusion indefinitely; a CI-side automated check comparing the marker's presence against the test's actual recent pass rate (flagging markers on tests that have been consistently passing for, say, 30 days) closes this gap rather than relying on someone remembering to clean it up. The hook itself is O(number of collected tests), a single pass over the test items, negligible overhead relative to actually running the suite.

Trade-offs & pitfalls: putting owner and issue directly in code (rather than in an external registry) has the advantage of staying co-located with the test and reviewed alongside it in a PR, but the disadvantage that it's easy to forget to update (or remove) as ownership changes or the issue closes; pairing it with the "marker on a now-reliable test" check above is what keeps the in-code metadata trustworthy over time rather than slowly drifting stale, the same staleness risk record/replay test doubles face for a different reason.

Verified (executed this session, python3/pytest 9.1.1 + pytest-rerunfailures 16.4): the corrected code above (marker renamed to known_flaky, def test_...(): with parentheses, and proper 4-space nested indentation restored -- the original code block's indentation was also flattened to a single space per line regardless of nesting depth, which independently prevents it from compiling) was run end-to-end against a test that fails its first 2 calls then passes: pytest reran it twice (per the hook attaching pytest.mark.flaky(reruns=2, reruns_delay=1) for the real plugin to pick up) and passed on the 3rd attempt, output 1 passed, 2 rerun in 2.05s, confirming the collection-time hook design actually works once both bugs are fixed.

Direct answer: Evaluate each tool against the SAME four criteria, but weight them differently depending on what the tool actually is (a detection ANALYTICS platform, a dedicated flaky-test detector, or a service-virtualization tool solve genuinely different problems), and structure the integration as a REVERSIBLE pilot with an explicit rollback trigger, not a one-way commitment, given how much of this topic's remediation work depends on trusting the tool's output.

Structured elaboration

Evaluation criteria, applied to three illustrative categories:

1. A dedicated flaky-test-detector product (purpose-built for detecting and scoring flakiness from CI history, the category a named example like FlakyTestDetector falls into):

• Integration effort: typically LOW, these tools are built specifically to plug into common CI systems with minimal custom work.
• Accuracy: the most important criterion here specifically, since the tool's entire value proposition IS its detection accuracy; validate against your OWN historical data (feed it a known period of results and check whether its flags match what you already know was genuinely flaky) rather than trusting vendor-claimed accuracy figures alone.
• Cost: usually a recurring per-seat or per-test-volume subscription, worth modeling against your ACTUAL test volume, not a rough estimate, since detection tools often price by scale.
• Privacy: your test names, failure messages, and potentially stack traces (which can incidentally contain sensitive data, per the earlier discussion on HAR/log scrubbing) leave your infrastructure to a third party; needs explicit review of what data the tool ingests and its data-handling/retention policy.

2. A general test-analytics platform (broader observability, of which flaky-test detection is one feature among several):

• Integration effort: typically HIGHER than a dedicated tool, since you're adopting a broader platform, not a narrow point solution.
• Accuracy: harder to isolate and validate specifically for flakiness, since it's one feature among several, worth a focused validation specifically on THAT feature rather than assuming overall platform quality implies feature-specific accuracy.
• Cost: often bundled with broader observability value, which may or may not be worth it depending on whether you'd want the other features anyway, a genuinely different cost calculus than a narrow point tool.
• Privacy: broader platform access to your CI/test data generally, worth a more thorough review given the wider scope.

3. Service virtualization (a different category entirely, not detection, but a REMEDIATION tool for the external-dependency root-cause category):

• Integration effort: MODERATE to high, requires actually integrating the virtualization layer into your test execution path, a more invasive change than a passive detection tool.
• Accuracy: reframed as FIDELITY here, does the virtualized service's behavior stay representative of the real dependency's actual contract, which ties directly to the staleness-risk discussion covered for test doubles elsewhere in this topic.
• Cost: often licensing plus the ongoing maintenance cost of keeping virtualized services in sync with real ones.
• Privacy: lower concern here specifically, since virtualization typically runs within your own infrastructure rather than sending data to a third party, though this depends on the specific product.

Integration plan with rollback: (1) run the chosen tool in a PARALLEL, non-blocking pilot mode first, its detections logged and compared against your existing process's output, without yet acting on its flags automatically; (2) after a defined pilot period (say, one month), compare the tool's flags against ground truth (using the same retrospective genuine-catch-ratio-style validation covered in the disable/quarantine-decision sub-area) to measure real accuracy on YOUR data, not the vendor's; (3) only THEN promote it to actively driving automated actions (quarantine, retry-decisions), with an explicit rollback trigger defined UPFRONT (for example, "if accuracy drops below X% or the false-positive rate exceeds Y% for two consecutive weeks, revert to the pre-tool process") rather than deciding on rollback criteria only after a problem has already emerged and trust has already been damaged.

Worked example: piloting a dedicated flaky-detector product in parallel mode for one month against a team's existing manual triage process shows it correctly flags flaky tests with roughly 85% precision when checked against the team's own retrospective classification, a reasonable bar to promote it to actively driving quarantine decisions; a defined rollback trigger (precision dropping below 70% for two consecutive weeks, checked via the SAME ongoing audit process covered in the self-healing-runner sub-area) is documented and agreed before promotion, so if the tool's accuracy degrades later (a vendor model update behaving worse on your specific test patterns, for instance), there's already an agreed, unambiguous criterion for reverting rather than a fresh debate under pressure.

Trade-offs & pitfalls: evaluating a tool ONLY on vendor-provided accuracy claims, without validating against your own historical data first, is the single most common mistake, a detection model's accuracy is highly dependent on the specific characteristics of YOUR test suite and failure patterns, and a tool that performs well on the vendor's benchmark or another company's suite may perform meaningfully differently on yours; the parallel-pilot validation step above exists specifically to catch that gap before committing to automated actions driven by the tool's output.

Direct answer: Collect evidence that shows WHICH two operations actually collided and WHY (query logs with timestamps, the database's own deadlock-detection output, and lock-wait statistics), reproduce it locally by deliberately forcing the same interleaving rather than hoping for luck, and fix it either by choosing an appropriate isolation level/explicit locking strategy at the database layer or by restructuring the application logic to avoid the contention entirely.

Structured elaboration

Evidence to collect:

• Query logs with precise timestamps: enable statement-level logging around the failing operation, capturing exactly which queries ran, in what order, from which connection/transaction, so you can reconstruct the actual interleaving that occurred during a failure, not just infer it after the fact.
• Deadlock traces: most relational databases (Postgres, MySQL/InnoDB) can log full deadlock details when one occurs, showing exactly which two transactions held which locks and were waiting on each other; this is often the single most direct piece of evidence, converting "the test failed" into "transaction A held lock on row X waiting for row Y, while transaction B held row Y waiting for row X."
• Lock wait statistics: even short of an outright deadlock, elevated lock-wait times (visible via the database's own lock-monitoring views) on the tables/rows involved indicate genuine contention, useful for confirming a race even when it manifests as a slow, eventually-successful operation rather than an outright deadlock error.

Reproducing the race locally: rather than relying on timing luck, deliberately construct the interleaving, open two separate connections/transactions in a test harness, and use explicit synchronization (start transaction A, pause it at the specific point BEFORE its commit via a debugger breakpoint or an injected delay, start and complete transaction B's conflicting operation, then resume and complete A) to force the exact race condition on demand, converting an intermittent production symptom into a reliable, on-demand local reproduction.

SQL/database techniques to fix the root cause:

• Transaction isolation levels: if the race stems from a lower isolation level (READ COMMITTED, the common default) allowing a read-then-write sequence to be interleaved with another transaction's write, a stricter isolation level (REPEATABLE READ or SERIALIZABLE) can eliminate the specific anomaly, at the cost of increased lock contention and a higher rate of transactions needing to retry due to serialization failures, a real trade-off, not a free upgrade.
• SELECT... FOR UPDATE: explicitly lock the specific rows a transaction is about to modify at READ time, rather than relying on the isolation level alone, preventing another transaction from acquiring a conflicting lock on those same rows until the first transaction completes; this is a more targeted fix than raising the isolation level suite-wide, since it applies only to the specific query pattern that needs it.
• Optimistic locking: instead of locking rows pessimistically, add a version column (or a timestamp) checked at UPDATE time (UPDATE... SET version = version + 1 WHERE id = ? AND version = ?), and detect a conflict when the update affects zero rows (meaning someone else updated it first), retrying at the APPLICATION level; this avoids holding database locks during the race window entirely, trading a possible retry for reduced lock contention, well suited to scenarios where CONTENTION is rare but must still be handled correctly when it does occur.
• Application-side mitigations: where a database-level fix is impractical (a legacy schema, a third-party-managed database), serialize the conflicting operations at the APPLICATION layer instead, via an application-level lock (a distributed lock service) or by routing conflicting operations through a single, ordered queue/worker, moving the mutual-exclusion guarantee out of the database and into code you directly control.

Worked example: a flaky integration test intermittently fails with a unique-constraint violation when two test setup routines concurrently attempt to create the SAME default record for a shared reference table (an "unless exists, insert" pattern implemented as a plain read-then-conditional-insert, without any locking). Deadlock/lock-wait evidence isn't directly implicated here (it's a race, not a deadlock, since there's no CIRCULAR wait), but query-log evidence shows both connections' SELECT-then-INSERT sequences interleaving under concurrent test setup. The fix: replace the read-then-conditional-insert pattern with a single atomic INSERT... ON CONFLICT DO NOTHING (Postgres) statement, which the database itself handles atomically, removing the race entirely rather than trying to synchronize around it with application-level locking.

Trade-offs & pitfalls: raising the isolation level suite-wide to fix ONE specific race is a blunt instrument that can introduce new, different flakiness elsewhere (increased serialization-failure retries on OTHER, previously-fine transactions under the new stricter isolation level); prefer a targeted fix (SELECT... FOR UPDATE on the specific query, or an atomic single-statement operation as in the worked example) scoped to the actual contention point, reserving a suite-wide isolation-level change for cases where the contention pattern is genuinely pervasive rather than localized to one specific operation.