Flaky Test Management and Test Reliability Questions

Direct answer: Prioritize by expected IMPACT of fixing a test, not by raw flakiness rate, since a test that fails rarely but blocks every release is worth more engineering time than a test that fails often but nobody notices, and build the scoring so it scales to thousands of candidates without manual review of each one.

Structured elaboration

• Prioritization criteria, combined into a score: (a) developer-velocity impact, how often this test's failures block a merge or require a rerun, a directly measurable count from CI history; (b) release-blocking frequency, specifically whether this test sits in a release-gating suite versus a non-blocking one, since the same failure rate matters far more in a gating suite; (c) runtime/compute cost, tests that are both flaky AND slow consume disproportionate CI budget on retries; (d) business-criticality of the code path it covers (checkout, auth) weighted higher than a low-traffic internal tool; (e) ownership clarity, a test with a clear, responsive owner is cheaper to fix (lower marginal cost) than one whose owner has to be tracked down first, which affects how much VALUE a unit of triage effort returns.
• A concrete combined score: rank candidates by (impact score) divided by (estimated fix effort), a cost-effectiveness ranking rather than an impact-only ranking, so the limited monthly fix capacity is spent where it returns the most value per unit of effort, not just on the highest-impact-but-also-hardest-to-fix tests.
• Scale consideration for a monorepo with 200k tests: at this scale, manual per-test triage is infeasible; the scoring above must be computed automatically from CI telemetry (impact and frequency signals) and the effort ESTIMATE can start as a coarse heuristic (grouped by root-cause category inferred from failure signature; a "shared fixture" category is typically cheaper to fix at scale than a "genuine application race condition" category) refined over time as actual fix effort data accumulates.
• Incident-driven justification: when a production incident traces back to a bug that a flaky test WOULD have caught, but its failure was dismissed or its owner never got to it, that incident is powerful evidence for the prioritization rule and should feed back into the scoring, a test whose flakiness has previously masked a real incident should get an impact-score boost beyond what its raw metrics alone would suggest.
• Metrics and process to convince stakeholders: present the SAME cost-effectiveness framing to stakeholders that drives the internal ranking, "fixing these 20 tests addresses 60% of last month's blocked-merge time" is a concrete, business-relevant number leadership can act on, versus an abstract "we have 3,000 flaky tests" that doesn't translate into a decision. Pair it with the incident evidence above when available, since a concrete near-miss or realized incident is the most persuasive single data point for securing dedicated fix time.

Worked example: with capacity to fix 20 tests this month out of a backlog of thousands, ranking by (blocked-merge-minutes caused) divided by (estimated fix hours, from root-cause category) surfaces that the top 20 by this ratio account for a disproportionate share of blocked-merge time (a realistic pattern given flaky-test impact is typically concentrated: a small fraction of tests on high-traffic, release-gating paths cause most of the developer-facing pain), a far stronger business case than picking the 20 tests with the numerically highest raw failure rate, some of which may be low-traffic, non-blocking tests nobody was actually affected by.

Trade-offs & pitfalls: a pure impact-only ranking (ignoring fix cost) systematically deprioritizes cheap, easy wins in favor of hard, high-impact tests that may take disproportionately long, actually reducing the NUMBER of tests fixed and the overall trend in a fixed monthly budget; the cost-effectiveness framing above avoids that trap, but requires a reasonably honest fix-effort estimate, which is itself uncertain early on and should be revised as real data comes in rather than trusted blindly from day one.

Direct answer: Score each failure on how confident you are it's a KNOWN transient pattern (from history and error-type classification) versus how much environmental strain the CI system is currently under, and only auto-retry when both point toward "probably transient," escalating to a hard failure by default whenever business priority makes a false pass too risky to allow.

Structured elaboration

Inputs and how each contributes to the decision:

• Recent pass/fail history: a test with a well-established intermittent pattern (passes most of the time, historically recovers on retry) is a stronger retry candidate than a test failing for the first time ever, an unprecedented failure is more likely to be a genuine new regression.
• Error type classification: failures matching a KNOWN transient signature (a specific timeout error, a recognized flaky-infra pattern) score higher toward "retry"; failures matching an assertion mismatch on business logic, or an error type never seen before for this test, score toward "hard failure," since retrying a genuine logic bug just wastes time and risks a lucky pass hiding it.
• CI environment metrics (queue time, agent health): if the CURRENT environment shows signs of strain (unusually long queue times, a runner reporting degraded health), that raises the prior probability THIS failure is environmental rather than a real bug, nudging toward retry; a healthy, unremarkable environment at the time of failure removes that mitigating signal.
• Business priority: a test covering a business-critical path (payments, auth) should have a HIGHER bar for retry-and-pass, since the cost of a false pass (a real regression slipping through) is asymmetrically higher there; a lower-priority test can tolerate a more permissive retry policy since a missed regression there is cheaper.

Scoring approach, high-level: combine these into a composite score, for example a weighted sum or a simple decision-tree: retry_score = f(historical_pattern_match, error_type_transient_probability, environment_strain_signal) - priority_penalty(business_priority), where priority_penalty is large enough for critical-path tests that it overrides an otherwise-retry-favorable score from the other three inputs, expressing the principle that criticality can veto an otherwise reasonable retry decision.

Thresholds: retry automatically above a HIGH-confidence threshold; treat as hard failure below a LOW threshold (unrecognized error type, no historical transient pattern, healthy environment); for the MIDDLE band, apply a bounded, small number of retries (1 or 2, not open-ended) with the outcome logged distinctly (which inputs drove the decision, and whether the retry ultimately passed or not) so a repeated pattern in the middle band gets flagged for review rather than silently retried indefinitely.

Safeguards to avoid masking regressions:

1. Every retry, whether it eventually passed or not, is logged with the FULL context (which inputs drove the decision), so a post-hoc audit can review whether the decision logic is behaving as intended.
2. A CAP on total retries per test per time window (not just per single failure), so a test degrading into chronic flakiness triggers escalation (quarantine, ticket) rather than silently accumulating individual retry-rescues forever.
3. Business-priority-based OVERRIDE: for the highest-priority tests, consider disabling automated retry-to-pass entirely and routing straight to hard failure plus immediate escalation, accepting the velocity cost in exchange for zero tolerance on a masked regression in that specific path.

Worked example: a payment-confirmation test fails with a timeout error under a CI run where queue times are currently 3x normal (environment strain signal is high) and this specific test has a well-established history of occasional timeout-related transient failures. Under a naive rule, this would score toward "retry." But because the test is tagged business-critical, the priority penalty pushes the combined decision to "hard failure plus immediate escalation" rather than a silent retry-to-pass, accepting that this specific run blocks the merge in exchange for guaranteeing a human looks at it before anything ships.

Trade-offs & pitfalls: an algorithm this data-driven is only as good as the error-type classification and historical-pattern data feeding it; a new, previously-unseen transient failure MODE (a new kind of infrastructure flakiness the classifier hasn't learned yet) will initially score toward "hard failure" even though it's genuinely transient, a reasonable, conservative default (fail closed on uncertainty) but one that will generate some avoidable friction until the pattern is recognized and incorporated. Regularly reviewing hard-failures-that-later-turned-out-transient is how the classification data stays current rather than going stale.

Direct answer: Poll job status, and on failure, schedule a retry via POST /jobs/{id}/retry with exponential backoff up to 3 attempts, tracking a per-job failure count across calls so that crossing a quarantine threshold short-circuits further retries; make every retry call idempotent via a deterministic key derived from (job_id, attempt_number), so a duplicate call from a retried HTTP request or a second concurrent worker never double-schedules the same retry.

Approach and code

import time
import random


def reschedule_with_backoff(api, job_id, max_attempts=3, quarantine_threshold=3,
                             base_delay=0.01, quarantined_jobs=None, sleep_fn=time.sleep):
    """Poll a job's status; on failure, retry with exponential backoff up to
    max_attempts. If accumulated failures reach quarantine_threshold, mark
    quarantined instead of continuing to retry. Idempotent per attempt.
    """
    if quarantined_jobs is None:
        quarantined_jobs = {}
    failure_count = quarantined_jobs.get(job_id, 0)

    for attempt in range(max_attempts):
        status = api.get_status(job_id)  # GET /jobs/{id}/status
        if status == "passed":
            quarantined_jobs.pop(job_id, None)  # clear on recovery
            return {"job_id": job_id, "final_status": "passed", "attempts_used": attempt + 1}

        failure_count += 1
        if failure_count >= quarantine_threshold:
            quarantined_jobs[job_id] = failure_count
            return {"job_id": job_id, "final_status": "quarantined", "failure_count": failure_count}

        idempotency_key = f"{job_id}:attempt-{attempt}"  # deterministic, not random
        delay = base_delay * (2 ** attempt) + random.uniform(0, base_delay)  # backoff + jitter
        sleep_fn(delay)
        api.retry(job_id, idempotency_key)  # POST /jobs/{id}/retry

    quarantined_jobs[job_id] = failure_count
    return {"job_id": job_id, "final_status": "quarantined", "failure_count": failure_count}

Idempotency handling: the idempotency_key is deterministically derived from (job_id, attempt), NOT a fresh random value per call, so if the SAME logical retry request is sent twice (a client-side timeout causing a retried HTTP call, or two concurrent workers racing to process the same failed job), the scheduler API can recognize the duplicate key and treat the second call as a no-op rather than scheduling a genuinely second retry attempt. This requires the server-side POST /jobs/{id}/retry endpoint to itself support idempotency keys (a common REST API pattern), which is an explicit assumption worth stating rather than silently relying on.

Concurrent workers: the failure_count must be tracked in SHARED state (here modeled as the quarantined_jobs dict passed in, standing in for a real shared store like a database row or a distributed counter) so two workers independently processing failures for the SAME job don't each independently think they're only on attempt 1 of 3 and collectively exceed max_attempts before either individually notices; in a real distributed deployment this shared counter needs to be an atomic increment against a real datastore (a SQL UPDATE... SET failure_count = failure_count + 1... RETURNING, or an atomic Redis INCR), not a plain in-memory dict, which the code above uses only as a stand-in for executable verification.

Verification (executed this session, python3, real fake-scheduler harness): the code block previously published for this answer had a real defect: every indented line had been flattened to a single leading space regardless of true nesting depth, so the if quarantined_jobs is None: block's body appeared at the SAME indentation as the if itself, an IndentationError: expected an indented block after 'if' statement on load, meaning the code could not have run at all despite this answer's prior claim of having executed it. This is fixed above with correct, consistent multi-level indentation; the logic itself was not otherwise wrong. I implemented the code exactly as shown against a fake in-memory scheduler API and ran four adversarial cases: (1) a job that fails twice then passes, confirming it correctly RECOVERS without hitting quarantine; (2) a job that fails every single time, confirming it QUARANTINES after max_attempts rather than retrying forever; (3) calling the retry API twice with the SAME idempotency key (simulating a duplicate call), confirming the underlying attempt counter increments only ONCE, not twice; (4) three sequential calls against SHARED failure-count state (simulating multiple workers touching the same job over time), confirming the job correctly quarantines once the shared counter crosses the threshold, rather than each call resetting its own independent count. All four passed:

case1 PASS: {'job_id': 'job-A', 'final_status': 'passed', 'attempts_used': 3}
case2 PASS: {'job_id': 'job-B', 'final_status': 'quarantined', 'failure_count': 3} retry_calls made: 2
case3 PASS: duplicate idempotency key correctly treated as a no-op, unique_keys=1
case4 PASS: shared failure-count state correctly quarantines after threshold across
sequential worker calls: {'job_id': 'job-D', 'final_status': 'quarantined', 'failure_count': 3}
ALL S58 CASES PASS (measured this session)

Trade-offs & pitfalls: the in-memory quarantined_jobs dict used above for executable verification is explicitly NOT safe for real concurrent distributed workers (a plain dict has no atomicity guarantee across processes/machines); the production version must replace it with an atomic operation against a real shared store, and this is exactly the kind of detail that looks fine in a quick local test (as case 4 above shows, sequential calls behave correctly) but would silently break under GENUINE concurrent access without the atomicity guarantee, worth flagging explicitly rather than letting the passing local test create false confidence about the concurrent case it doesn't actually exercise.

Direct answer: Balance automation and ownership by making the SYSTEM responsible for detecting and mechanically enforcing the quarantine action, while keeping HUMANS responsible for every decision that requires judgment, approving an exception, deciding a test's real value, and ultimately fixing it, with time limits and metrics ensuring neither side can quietly abdicate its role.

Structured elaboration

• Approval workflows: automated quarantine ENTRY (crossing the flakiness threshold) doesn't need human approval, speed matters more than judgment for that specific action, since quarantine is reversible and low-risk. Automated quarantine EXIT (a test resuming its blocking-gate power) should require an explicit owner confirmation, not a fully automatic re-promotion, since a wrongly-early exit reintroduces exactly the disruption the system exists to prevent; this asymmetry (fast automated entry, human-gated exit) is deliberate, not an oversight.
• Time limits on quarantine: every quarantined test gets an explicit SLA (for example, 30 days), tracked as quarantine-age; this is a governance mechanism, not just a metric, since a test with no time limit at all reliably becomes permanent, exactly the failure mode this governance model exists to prevent.
• Escalation paths: when a quarantine's SLA is about to expire without resolution, escalate, first a reminder to the owning team, then, if still unresolved, visibility to that team's engineering manager, and finally, if still unresolved past a further grace period, a decision point (extend with explicit written justification, or delete the test) rather than silent indefinite extension, which is what a governance model without a REAL escalation consequence degrades into.
• Metrics to track: quarantine-age (how long has this specific test been quarantined, the leading indicator of a stalling case); fix-rate (what fraction of quarantined tests get fixed within their SLA, the organization-level health metric); and a quarantine-backlog-size trend (is the overall count growing or shrinking over time, the signal for whether prevention and remediation are keeping pace with new flakiness entering the suite).
• Incentives to ensure quarantined tests get fixed rather than forgotten: make fix-rate visible in the same forum as other engineering-health metrics a team is already accountable for (the same principle covered in the culture-and-ownership sub-area), so a team's SLA-breach rate is something leadership can see, not a private, easily-ignored backlog. Avoid PURELY punitive framing (which incentivizes deleting tests or gaming the metric rather than genuinely fixing them); pair visibility with genuine, protected engineering time allocated for fixing (via the prioritized-triage approach) so teams have both the incentive AND the actual capacity to act on it.
• Balancing automation and ownership, explicitly: the system automates DETECTION and mechanical ENFORCEMENT (moving a test in and out of the blocking gate based on clear, consistent rules); humans retain OWNERSHIP of judgment calls (is this test worth fixing versus deleting, does this specific SLA breach warrant an exception, is the underlying root cause understood well enough to safely re-promote). Conflating the two, either fully automating judgment calls or requiring human approval for purely mechanical actions, is what breaks the balance in either direction.

Worked example: a test crosses the quarantine threshold and is automatically quarantined within minutes (no approval needed), with an issue auto-created and a 30-day SLA attached. At day 25 with no fix landed, an automated reminder alerts the owning team; at day 30 with still no resolution, the SLA breach becomes visible on the team's own engineering-health dashboard (the same one leadership reviews), and the team must EITHER land a fix, request an explicit, justified extension (visible to their manager, not a silent default), or accept the test's deletion. This structure ensures the test doesn't simply sit in quarantine indefinitely, which the earlier post-incident-policy sub-area of this topic showed is a real, costly failure mode when it happens.

Trade-offs & pitfalls: an overly strict, zero-exception SLA can pressure teams into rushed, low-quality fixes (or premature deletions of tests that still have real coverage value) just to avoid the deadline; the "explicit, justified extension" option in the worked example exists specifically to allow legitimate exceptions without reverting to the silent-indefinite-extension failure mode, the justification requirement (visible, reviewed) is what keeps that escape valve from becoming a routine loophole.

Direct answer: Treat "passes locally, fails intermittently in CI" as a diagnostic question with three candidate explanations (test-code timing, application-level async behavior, or environment difference), and design your reproduction to distinguish between them rather than guessing and patching the first plausible cause.

Structured elaboration

• Reproduce locally under CI-like conditions: running the test once locally rarely reproduces a CI-only intermittent failure, because CI runners are typically slower and noisier (shared CPU, network variance) than a developer machine. Instead: (a) run the SAME test many times in a loop locally (50 to 100 iterations) to establish a baseline local failure rate, even if it is near zero; (b) if it stays at zero locally, artificially degrade the local environment (CPU throttling, added network latency) to see if the failure reproduces under load; (c) increase logging verbosity around the async boundary you suspect, and run in CI itself with that extra logging to capture evidence from an actual failure rather than a hypothesized one.
• Deterministic time control: if the suspected timeout is timing-sensitive, replace real waits with a controllable clock (a time-mocking library, or an injected clock dependency) so you can deterministically simulate the "just barely too slow" window that CI's variance occasionally produces, rather than waiting for it to happen by chance.
• Which tests to run repeatedly: prioritize the specific failing test in isolation many times (to separate "this test is flaky" from "the suite has cross-test pollution"), and separately run the full suite in the SAME order and parallelism CI uses, since order- or parallelism-dependent flakiness will not show up running the test alone.
• Changing tests to be deterministic or tolerant: two different fixes address two different diagnoses. If the root cause is a genuine async race in test code (asserting before an operation completes), the fix is an explicit, condition-based wait instead of a fixed sleep. If the root cause is real environmental variance (CI is sometimes just slower), the fix is a more GENEROUS but still bounded timeout, paired with clear logging that distinguishes "genuinely stuck" from "was slow but finished."
• Deciding test vs application vs infra: if the same operation, called directly (bypassing the test framework) under a tight timing loop, occasionally takes longer than the test's timeout, the root cause is likely genuine async application latency, not a test bug; if the operation is always fast when isolated but the TEST still times out under full-suite load, the root cause is more likely resource contention from concurrent tests (CPU, DB connections) rather than the operation itself.
• Trade-offs between flakiness fixes and test coverage: the fastest way to make a flaky test "pass reliably" is to make its assertions weaker or its scope narrower, but that silently reduces the coverage it provides. A better trade-off is to keep the assertion strength and instead make the WAIT mechanism deterministic (explicit condition waiting) rather than trading away what the test actually verifies.

Worked example: a checkout test intermittently times out waiting for an order-confirmation API call in CI, never locally. Looping the isolated call 500 times locally under artificial CPU throttling reproduces occasional latency spikes above the test's 2-second timeout, roughly 1 in 150 iterations, confirming this is real async latency variance rather than a test-code bug. The fix: raise the wait to a bounded 5 seconds AND log the actual observed latency on every run (not just on timeout), so a future latency regression is visible as a trend in the logs rather than only as an intermittent CI failure.

Trade-offs & pitfalls: a common mistake is fixing the FIRST plausible cause found (usually "just increase the timeout") without confirming it against the reproduction evidence; this frequently just delays the next flake rather than resolving it. Another pitfall: adding verbose logging only in CI and not validating it locally first means you may ship logging that itself changes timing (e.g., synchronous log writes slowing the exact operation you're diagnosing), which can mask or shift the very race you are trying to observe.