Flaky Test Management and Test Reliability Questions

Direct answer: The system needs four cooperating pieces: a scorer that continuously computes a flakiness metric per test from CI history, a state machine that moves a test between healthy, quarantined, and fixed states with an audit trail, an integration layer that talks to CI (to exclude a quarantined test from gating) and the issue tracker (to open and update a diagnostic ticket), and a re-evaluation loop that automatically proposes removing a test from quarantine, never leaving that decision to silently expire.

Structured elaboration

System components and data flow

flowchart LR
 A[CI test-result stream] --> B[Flakiness scorer]
 B -->|score > threshold| C[Quarantine state machine]
 C -->|enter quarantine| D[Test registry: tag quarantined]
 C -->|enter quarantine| E[Issue tracker: open ticket + attach artifacts]
 D --> F[CI gate: excludes quarantined tests from blocking]
 C -->|re-eval trigger: time elapsed or new runs| G[Re-evaluation job]
 G -->|score improved| H[Propose un-quarantine, owner confirms]
 G -->|no improvement, SLA breached| I[Escalate]
 H --> C

• Detection: the scorer reads recent CI results (a rolling window of, say, the last 20 to 50 runs) and computes a flakiness threshold breach, the same computation the topic's detection sub-area covers; this system consumes that score rather than redefining it.
• Tagging: on breach, the registry marks the test quarantined with a timestamp, the triggering score, and a reference to the CI run(s) that caused the breach, so the state is queryable and auditable, not a hidden flag.
• Issue creation: an issue is opened automatically with diagnostic artifacts attached (recent failure logs, the flakiness score history, and links to the offending CI runs) and assigned to the test's registered owner, or to a default triage queue if no owner is on file.
• Owner notification: a notification (chat, email, or both depending on team norms) goes to the owner with a direct link to the ticket, not just a dashboard update, since a silent dashboard entry is easy to miss.
• CI gate integration: quarantined tests still RUN (so their signal is not lost entirely) but do not block merges; their results are recorded and rolled up into the flakiness dashboard so a genuinely fixed test's improving trend is visible before the formal re-evaluation.
• Automatic re-evaluation: on a configurable schedule (for example, weekly) or after N new runs, the re-evaluation job recomputes the score. If it has improved below a (slightly stricter, to avoid flapping) un-quarantine threshold, it proposes removal from quarantine, which an owner confirms rather than the system silently re-enabling gating. If the SLA (for example, 30 days in quarantine) is breached with no improvement, it escalates rather than extending indefinitely.
• APIs: a minimal contract is POST /quarantine {test_id, score, evidence}, GET /quarantine/{test_id} for status, and POST /quarantine/{test_id}/reevaluate, plus a webhook the CI system calls after each run so the scorer stays current without polling.

Trade-offs & pitfalls: two failure modes need explicit design attention. First, noisy or oscillating quarantine: a test that hovers right at the threshold can flap in and out of quarantine every re-evaluation cycle, which is worse than either staying quarantined or staying gated, since it destabilizes the gate unpredictably. The fix is hysteresis: use a HIGHER score to enter quarantine than to exit it (for example, enter at a 15% failure rate, require dropping below 5% to exit), so a test sitting near the boundary does not flap. Second, metrics integrity: because quarantined tests keep running but stop blocking merges, it is tempting to let their reruns silently pad an "all green" dashboard; the design must keep quarantined-test results visible and clearly labeled so the team's headline pass rate is not artificially inflated by hiding the exact signal the system exists to surface.

Worked example: suppose a test has run 40 times in the last week with 6 failures (15% failure rate), crossing a 10%-enter threshold. It is auto-quarantined, a ticket is opened with the 6 failing run links attached, and the owner (from the test's registered CODEOWNERS-style metadata) is notified. Two weeks later, after the owner's fix, the test has run 25 more times with 0 failures; the re-evaluation job recomputes a rolling failure rate of roughly 0% over the most recent window, well under a 5%-exit threshold, and proposes un-quarantine, which the owner confirms in one click from the ticket.

Direct answer: Balance automation and ownership by making the SYSTEM responsible for detecting and mechanically enforcing the quarantine action, while keeping HUMANS responsible for every decision that requires judgment, approving an exception, deciding a test's real value, and ultimately fixing it, with time limits and metrics ensuring neither side can quietly abdicate its role.

Structured elaboration

• Approval workflows: automated quarantine ENTRY (crossing the flakiness threshold) doesn't need human approval, speed matters more than judgment for that specific action, since quarantine is reversible and low-risk. Automated quarantine EXIT (a test resuming its blocking-gate power) should require an explicit owner confirmation, not a fully automatic re-promotion, since a wrongly-early exit reintroduces exactly the disruption the system exists to prevent; this asymmetry (fast automated entry, human-gated exit) is deliberate, not an oversight.
• Time limits on quarantine: every quarantined test gets an explicit SLA (for example, 30 days), tracked as quarantine-age; this is a governance mechanism, not just a metric, since a test with no time limit at all reliably becomes permanent, exactly the failure mode this governance model exists to prevent.
• Escalation paths: when a quarantine's SLA is about to expire without resolution, escalate, first a reminder to the owning team, then, if still unresolved, visibility to that team's engineering manager, and finally, if still unresolved past a further grace period, a decision point (extend with explicit written justification, or delete the test) rather than silent indefinite extension, which is what a governance model without a REAL escalation consequence degrades into.
• Metrics to track: quarantine-age (how long has this specific test been quarantined, the leading indicator of a stalling case); fix-rate (what fraction of quarantined tests get fixed within their SLA, the organization-level health metric); and a quarantine-backlog-size trend (is the overall count growing or shrinking over time, the signal for whether prevention and remediation are keeping pace with new flakiness entering the suite).
• Incentives to ensure quarantined tests get fixed rather than forgotten: make fix-rate visible in the same forum as other engineering-health metrics a team is already accountable for (the same principle covered in the culture-and-ownership sub-area), so a team's SLA-breach rate is something leadership can see, not a private, easily-ignored backlog. Avoid PURELY punitive framing (which incentivizes deleting tests or gaming the metric rather than genuinely fixing them); pair visibility with genuine, protected engineering time allocated for fixing (via the prioritized-triage approach) so teams have both the incentive AND the actual capacity to act on it.
• Balancing automation and ownership, explicitly: the system automates DETECTION and mechanical ENFORCEMENT (moving a test in and out of the blocking gate based on clear, consistent rules); humans retain OWNERSHIP of judgment calls (is this test worth fixing versus deleting, does this specific SLA breach warrant an exception, is the underlying root cause understood well enough to safely re-promote). Conflating the two, either fully automating judgment calls or requiring human approval for purely mechanical actions, is what breaks the balance in either direction.

Worked example: a test crosses the quarantine threshold and is automatically quarantined within minutes (no approval needed), with an issue auto-created and a 30-day SLA attached. At day 25 with no fix landed, an automated reminder alerts the owning team; at day 30 with still no resolution, the SLA breach becomes visible on the team's own engineering-health dashboard (the same one leadership reviews), and the team must EITHER land a fix, request an explicit, justified extension (visible to their manager, not a silent default), or accept the test's deletion. This structure ensures the test doesn't simply sit in quarantine indefinitely, which the earlier post-incident-policy sub-area of this topic showed is a real, costly failure mode when it happens.

Trade-offs & pitfalls: an overly strict, zero-exception SLA can pressure teams into rushed, low-quality fixes (or premature deletions of tests that still have real coverage value) just to avoid the deadline; the "explicit, justified extension" option in the worked example exists specifically to allow legitimate exceptions without reverting to the silent-indefinite-extension failure mode, the justification requirement (visible, reviewed) is what keeps that escape valve from becoming a routine loophole.

Direct answer: A prioritized plan has two parallel tracks running from day one, not sequential phases: immediate mitigation (quarantine plus disciplined retries) to restore trust in CI right away, and a longer root-cause program (measurement, ownership, tooling, culture) that actually drives the flakiness rate down over the following quarter, with named stakeholders and milestones at 30/60/90 days.

Structured elaboration

Short-term mitigation (week 1)

• Instrument: before touching anything, get a flakiness score per test from recent CI history (a simple failure-rate-over-a-rolling-window metric is enough to start).
• Quarantine: automatically move any test above an agreed threshold (for example, failure rate over 10% over the last 20 runs) out of the merge-blocking gate into a visible, still-running quarantine suite. This is reversible and auditable, not a silent skip.
• Controlled retries: for tests just below the quarantine threshold, allow a small number of automatic reruns on failure, but log every retry and report a "flaky-but-passed-on-retry" signal separately from a clean pass, so the signal is not silently lost.

Long-term fixes (weeks 2 to 90 and beyond)

• Root-cause analysis: triage the quarantined list by IMPACT (how often it blocked a release, not just raw failure count) and assign root-cause categories (timing, shared state, external dependency, environment) so fixes can be batched by pattern rather than handled one at a time.
• Tooling: build or adopt the flakiness dashboard and quarantine automation as first-class CI infrastructure, not a side script, so the mitigation from week 1 stays sustainable at scale.
• Ownership: every quarantined test gets a named owner and an SLA (for example, fixed or deleted within 30 days, or escalated). Ownership is what prevents quarantine from becoming a graveyard.
• Culture: report flakiness metrics in the same forum as other quality metrics (uptime, defect escape rate) so leadership treats it as a first-class engineering-health signal, not a nuisance.

Measuring progress: track (a) overall suite flakiness rate (weighted by how often each test runs), (b) time-in-quarantine distribution (are tests getting fixed or accumulating?), and (c) developer-facing signal: how often engineers report ignoring or re-running failures without investigating. A plan that only tracks (a) can look successful while (c) stays bad if teams just get used to routing around flaky tests.

Stakeholders: the engineering teams that own the flaky tests (they do the fixing), a QA/test-infrastructure function (owns the quarantine tooling and dashboard), and engineering leadership (needs the 30/60/90 trend to justify the time investment against feature work). Depending on who is driving this plan, the framing shifts, but the substance does not: an individual-contributor lead runs the same short-term/long-term split as a 3-month cross-team initiative; an engineering manager migrating a team's suite out of a blocking gate is doing the same quarantine-plus-root-cause work at team scale; a cross-functional architect coordinating with Product and QA is adding the stakeholder-alignment layer on top of the identical technical plan. State explicitly in your answer which seat you are describing the plan from, since the mechanics are shared but the escalation and reporting lines differ.

90-day timeline: days 0 to 7, instrument and quarantine (stop the bleeding); days 8 to 30, root-cause and fix the top-impact 20% of quarantined tests (this typically recovers most of the trust, since a small fraction of flaky tests usually causes most of the blocked merges); days 31 to 90, build the sustaining process (ownership SLAs, dashboard as a standing artifact, a lint/review check to catch new flaky patterns before merge) and report the trend to leadership.

Worked example: A team starts with 150 quarantined tests and 40% of CI minutes spent on retries. Triage by impact finds that 18 tests (12% of the quarantined set) accounted for over 70% of blocked merges in the prior month, because they sit on the critical checkout path most PRs touch. Fixing those 18 first (mostly by replacing shared database fixtures with per-test isolation) recovers most of the developer-trust problem well before the remaining 132 lower-impact tests are worked through on their normal SLA.

Trade-offs & pitfalls: the most common failure mode is doing ONLY the short-term mitigation and calling it done, quarantine without an SLA becomes a permanent hiding place for real bugs and the suite's effective coverage silently erodes. The opposite failure mode is refusing to quarantine anything until it's "properly fixed," which keeps CI red and developers ignoring it for months while the root-cause work is still in progress. The plan above avoids both by running mitigation and root-cause work in parallel from day one, with quarantine time-boxed and owned rather than open-ended.

Direct answer: Split the test into two layers with different purposes, a fast, deterministic UNIT test that verifies your error-handling LOGIC against a controlled test double (no real dependency at all), and a small number of separately-scoped INTEGRATION tests that exercise the real dependency but tolerate its real instability with generous timeouts and are not part of the fast, PR-blocking gate.

Structured elaboration

• Mocking and test doubles for the unit layer: replace the slow or rate-limited dependency with a controllable stub that can be told to return specific responses ON DEMAND, including the exact error conditions (a timeout, a 429 rate-limit response, a malformed response) your error-handling code needs to be verified against. This makes the ERROR PATHS themselves fully deterministic and fast to test, which is usually where the most valuable test coverage lives anyway (verifying your code degrades gracefully under a KNOWN failure condition), rather than depending on the real dependency actually failing at the right moment to exercise that code path.
• Timeouts, scoped appropriately per layer: the unit-level tests, using stubs, need no meaningful timeout tolerance at all (a stub responds instantly); a SMALL number of true integration tests against the real dependency need a generous, explicit timeout appropriate to that dependency's real-world latency characteristics, and should be clearly separated (a distinct test suite or tag) from the fast unit suite so a slow or rate-limited real call doesn't block routine development velocity.
• Retries, only where they mirror production behavior: if your PRODUCTION code has retry logic for handling a rate-limited dependency, the unit test should verify THAT retry logic works correctly (using a stub that simulates a rate-limit response on the first call and success on a retry), which is a deterministic, fast test of real logic, distinct from "retrying the CI test itself" as a flakiness workaround, which papers over the problem instead of testing the actual handling code.

Concrete rewrite example: a flaky integration test that calls a real payment gateway and asserts the application correctly reports "payment temporarily unavailable" when the gateway rate-limits:

# BEFORE: a flaky integration test hitting the real gateway, which
# occasionally times out or isn't actually rate-limited when the test runs,
# making the test itself unreliable regardless of what it's trying to verify.
def test_payment_rate_limit_handling_integration():
 # relies on the REAL gateway actually being rate-limited right now,
 # which is nondeterministic and slow.
 response = real_payment_client.charge(test_card, amount=100)
 assert response.user_message == "payment temporarily unavailable"

# AFTER, layer 1: a fast, deterministic unit test using a stub that
# DETERMINISTICALLY returns a 429, verifying the application's own
# error-handling logic without depending on the real gateway's actual state.
def test_payment_rate_limit_handling_unit(stub_payment_client):
 stub_payment_client.set_next_response(status=429, body={"error": "rate_limited"})
 response = charge_with_error_handling(stub_payment_client, test_card, amount=100)
 assert response.user_message == "payment temporarily unavailable"
 assert stub_payment_client.call_count == 1 # verifies no unintended retry storm

# AFTER, layer 2: a small, separately-tagged integration test (not in the
# fast PR-blocking gate) that exercises the REAL gateway's sandbox mode,
# tolerating its real latency/instability with a generous timeout, run less
# frequently (e.g. nightly) to catch real contract drift the stub might miss.
@pytest.mark.integration_slow
def test_payment_gateway_contract_integration():
 response = real_payment_client.charge(sandbox_test_card, amount=100, timeout=30)
 assert response.status_code in (200, 429) # a looser, contract-level check

Trade-offs & pitfalls: moving the bulk of coverage to stub-based unit tests risks the stub DRIFTING out of sync with the real dependency's actual current contract (the same staleness risk that record/replay test doubles face); the small, separately-scoped integration tier exists specifically to catch that drift, and skipping it entirely in favor of pure stub-based testing trades away real-world fidelity for speed and determinism, a trade worth making for the BULK of tests but not worth making entirely, some real-dependency verification should remain, just not in the fast, blocking path.

Direct answer: Poll job status, and on failure, schedule a retry via POST /jobs/{id}/retry with exponential backoff up to 3 attempts, tracking a per-job failure count across calls so that crossing a quarantine threshold short-circuits further retries; make every retry call idempotent via a deterministic key derived from (job_id, attempt_number), so a duplicate call from a retried HTTP request or a second concurrent worker never double-schedules the same retry.

Approach and code

import time
import random


def reschedule_with_backoff(api, job_id, max_attempts=3, quarantine_threshold=3,
                             base_delay=0.01, quarantined_jobs=None, sleep_fn=time.sleep):
    """Poll a job's status; on failure, retry with exponential backoff up to
    max_attempts. If accumulated failures reach quarantine_threshold, mark
    quarantined instead of continuing to retry. Idempotent per attempt.
    """
    if quarantined_jobs is None:
        quarantined_jobs = {}
    failure_count = quarantined_jobs.get(job_id, 0)

    for attempt in range(max_attempts):
        status = api.get_status(job_id)  # GET /jobs/{id}/status
        if status == "passed":
            quarantined_jobs.pop(job_id, None)  # clear on recovery
            return {"job_id": job_id, "final_status": "passed", "attempts_used": attempt + 1}

        failure_count += 1
        if failure_count >= quarantine_threshold:
            quarantined_jobs[job_id] = failure_count
            return {"job_id": job_id, "final_status": "quarantined", "failure_count": failure_count}

        idempotency_key = f"{job_id}:attempt-{attempt}"  # deterministic, not random
        delay = base_delay * (2 ** attempt) + random.uniform(0, base_delay)  # backoff + jitter
        sleep_fn(delay)
        api.retry(job_id, idempotency_key)  # POST /jobs/{id}/retry

    quarantined_jobs[job_id] = failure_count
    return {"job_id": job_id, "final_status": "quarantined", "failure_count": failure_count}

Idempotency handling: the idempotency_key is deterministically derived from (job_id, attempt), NOT a fresh random value per call, so if the SAME logical retry request is sent twice (a client-side timeout causing a retried HTTP call, or two concurrent workers racing to process the same failed job), the scheduler API can recognize the duplicate key and treat the second call as a no-op rather than scheduling a genuinely second retry attempt. This requires the server-side POST /jobs/{id}/retry endpoint to itself support idempotency keys (a common REST API pattern), which is an explicit assumption worth stating rather than silently relying on.

Concurrent workers: the failure_count must be tracked in SHARED state (here modeled as the quarantined_jobs dict passed in, standing in for a real shared store like a database row or a distributed counter) so two workers independently processing failures for the SAME job don't each independently think they're only on attempt 1 of 3 and collectively exceed max_attempts before either individually notices; in a real distributed deployment this shared counter needs to be an atomic increment against a real datastore (a SQL UPDATE... SET failure_count = failure_count + 1... RETURNING, or an atomic Redis INCR), not a plain in-memory dict, which the code above uses only as a stand-in for executable verification.

Verification (executed this session, python3, real fake-scheduler harness): the code block previously published for this answer had a real defect: every indented line had been flattened to a single leading space regardless of true nesting depth, so the if quarantined_jobs is None: block's body appeared at the SAME indentation as the if itself, an IndentationError: expected an indented block after 'if' statement on load, meaning the code could not have run at all despite this answer's prior claim of having executed it. This is fixed above with correct, consistent multi-level indentation; the logic itself was not otherwise wrong. I implemented the code exactly as shown against a fake in-memory scheduler API and ran four adversarial cases: (1) a job that fails twice then passes, confirming it correctly RECOVERS without hitting quarantine; (2) a job that fails every single time, confirming it QUARANTINES after max_attempts rather than retrying forever; (3) calling the retry API twice with the SAME idempotency key (simulating a duplicate call), confirming the underlying attempt counter increments only ONCE, not twice; (4) three sequential calls against SHARED failure-count state (simulating multiple workers touching the same job over time), confirming the job correctly quarantines once the shared counter crosses the threshold, rather than each call resetting its own independent count. All four passed:

case1 PASS: {'job_id': 'job-A', 'final_status': 'passed', 'attempts_used': 3}
case2 PASS: {'job_id': 'job-B', 'final_status': 'quarantined', 'failure_count': 3} retry_calls made: 2
case3 PASS: duplicate idempotency key correctly treated as a no-op, unique_keys=1
case4 PASS: shared failure-count state correctly quarantines after threshold across
sequential worker calls: {'job_id': 'job-D', 'final_status': 'quarantined', 'failure_count': 3}
ALL S58 CASES PASS (measured this session)

Trade-offs & pitfalls: the in-memory quarantined_jobs dict used above for executable verification is explicitly NOT safe for real concurrent distributed workers (a plain dict has no atomicity guarantee across processes/machines); the production version must replace it with an atomic operation against a real shared store, and this is exactly the kind of detail that looks fine in a quick local test (as case 4 above shows, sequential calls behave correctly) but would silently break under GENUINE concurrent access without the atomicity guarantee, worth flagging explicitly rather than letting the passing local test create false confidence about the concurrent case it doesn't actually exercise.