Direct answer: A prioritized plan has two parallel tracks running from day one, not sequential phases: immediate mitigation (quarantine plus disciplined retries) to restore trust in CI right away, and a longer root-cause program (measurement, ownership, tooling, culture) that actually drives the flakiness rate down over the following quarter, with named stakeholders and milestones at 30/60/90 days.
Structured elaboration
Short-term mitigation (week 1)
- Instrument: before touching anything, get a flakiness score per test from recent CI history (a simple failure-rate-over-a-rolling-window metric is enough to start).
- Quarantine: automatically move any test above an agreed threshold (for example, failure rate over 10% over the last 20 runs) out of the merge-blocking gate into a visible, still-running quarantine suite. This is reversible and auditable, not a silent skip.
- Controlled retries: for tests just below the quarantine threshold, allow a small number of automatic reruns on failure, but log every retry and report a "flaky-but-passed-on-retry" signal separately from a clean pass, so the signal is not silently lost.
Long-term fixes (weeks 2 to 90 and beyond)
- Root-cause analysis: triage the quarantined list by IMPACT (how often it blocked a release, not just raw failure count) and assign root-cause categories (timing, shared state, external dependency, environment) so fixes can be batched by pattern rather than handled one at a time.
- Tooling: build or adopt the flakiness dashboard and quarantine automation as first-class CI infrastructure, not a side script, so the mitigation from week 1 stays sustainable at scale.
- Ownership: every quarantined test gets a named owner and an SLA (for example, fixed or deleted within 30 days, or escalated). Ownership is what prevents quarantine from becoming a graveyard.
- Culture: report flakiness metrics in the same forum as other quality metrics (uptime, defect escape rate) so leadership treats it as a first-class engineering-health signal, not a nuisance.
Measuring progress: track (a) overall suite flakiness rate (weighted by how often each test runs), (b) time-in-quarantine distribution (are tests getting fixed or accumulating?), and (c) developer-facing signal: how often engineers report ignoring or re-running failures without investigating. A plan that only tracks (a) can look successful while (c) stays bad if teams just get used to routing around flaky tests.
Stakeholders: the engineering teams that own the flaky tests (they do the fixing), a QA/test-infrastructure function (owns the quarantine tooling and dashboard), and engineering leadership (needs the 30/60/90 trend to justify the time investment against feature work). Depending on who is driving this plan, the framing shifts, but the substance does not: an individual-contributor lead runs the same short-term/long-term split as a 3-month cross-team initiative; an engineering manager migrating a team's suite out of a blocking gate is doing the same quarantine-plus-root-cause work at team scale; a cross-functional architect coordinating with Product and QA is adding the stakeholder-alignment layer on top of the identical technical plan. State explicitly in your answer which seat you are describing the plan from, since the mechanics are shared but the escalation and reporting lines differ.
90-day timeline: days 0 to 7, instrument and quarantine (stop the bleeding); days 8 to 30, root-cause and fix the top-impact 20% of quarantined tests (this typically recovers most of the trust, since a small fraction of flaky tests usually causes most of the blocked merges); days 31 to 90, build the sustaining process (ownership SLAs, dashboard as a standing artifact, a lint/review check to catch new flaky patterns before merge) and report the trend to leadership.
Worked example: A team starts with 150 quarantined tests and 40% of CI minutes spent on retries. Triage by impact finds that 18 tests (12% of the quarantined set) accounted for over 70% of blocked merges in the prior month, because they sit on the critical checkout path most PRs touch. Fixing those 18 first (mostly by replacing shared database fixtures with per-test isolation) recovers most of the developer-trust problem well before the remaining 132 lower-impact tests are worked through on their normal SLA.
Trade-offs & pitfalls: the most common failure mode is doing ONLY the short-term mitigation and calling it done, quarantine without an SLA becomes a permanent hiding place for real bugs and the suite's effective coverage silently erodes. The opposite failure mode is refusing to quarantine anything until it's "properly fixed," which keeps CI red and developers ignoring it for months while the root-cause work is still in progress. The plan above avoids both by running mitigation and root-cause work in parallel from day one, with quarantine time-boxed and owned rather than open-ended.
