Quality Culture and Ownership Questions

Direct answer

Strict, hard-blocking enforcement buys a consistent quality floor and removes the social cost of anyone having to say no, but it slows every team by the same amount regardless of whether their actual risk justifies it, and a miscalibrated gate breeds a bypass culture that quietly defeats the whole point. The governance model I would propose is tiered rather than binary: a small set of universal, non-negotiable floors, and everything else calibrated to each team's actual blast radius (how much of the system, or how many users, would be affected if something there goes wrong) through explicit, written decision rules instead of case-by-case judgment calls.

Structured elaboration

The trade-off itself. Hard-blocking checks (for example, blocking a pull request, PR, for a coverage decrease or a failed end-to-end, e2e, test) give you a real, consistent floor and prevent the "just this once" exception that erodes standards over time. Their cost is that they cannot distinguish a genuinely low-risk change from a risky one unless you design them to, so they slow every team uniformly, and when the gate is noisy or miscalibrated, engineers find workarounds rather than fixing the underlying issue, which is worse than no gate at all. Advisory, non-blocking checks preserve velocity and trust engineer judgment, but rely on consistent voluntary compliance, which is exactly what erodes under deadline pressure, the moment the check matters most.

A tiered governance model:

• Tier 1, high blast radius (customer-facing, or touching money or safety): hard-blocking gates on the core checks (coverage regression, e2e pass, security scan), with essentially no unreviewed exceptions.
• Tier 2, medium blast radius (internal tools, moderate traffic): advisory with visibility. The check runs and reports, and merging past a failure requires a lightweight, logged justification rather than a full approval chain, so there is friction without a hard stop.
• Tier 3, low blast radius (experiments, prototypes): checks run for information only; the team owns its own bar.
• Assign each service to a tier using objective, periodically reviewed criteria (customer-facing traffic, whether it touches money or safety, historical incident rate), not a permanent, one-time label, since a service's risk profile changes as it matures.

Example decision rules:

1. If a service is customer-facing and handles payment or personal data, a coverage regression blocks the merge, and any exception requires sign-off above the requester's own manager.
2. If a change only touches test files or documentation, skip the e2e gate entirely, removing friction where it clearly cannot matter.
3. If a specific e2e check has failed intermittently on unrelated changes for a defined recent period, it is temporarily removed from blocking status until fixed, so one known-unreliable check does not erode trust in the whole gate.
4. A team can request a temporary tier downgrade for a well-scoped experiment, with an expiry date and their manager's sign-off, rather than an informal, undocumented exception.

Worked example

Apply the model to three real teams: a payments team sits in Tier 1, so a coverage-regressing change is blocked outright and an exception needs a sign-off well above the engineer's own manager. An internal admin-tools team sits in Tier 2, so the same kind of regression is visible and logged but does not block the merge, since the blast radius of a bug there is much smaller. A small prototype team sits in Tier 3, where the checks run for information only. When the payments engineer requests an exception under deadline pressure, decision rule 1 gives a clear, pre-agreed answer (escalate for senior sign-off) rather than an ad hoc negotiation, while the internal-tools team's equivalent request was never actually blocked in the first place, since their tier does not gate on that check.

Trade-offs and pitfalls

• Too many tiers becomes its own bureaucracy; keep the number small (three is usually enough) and put the effort into keeping tier assignment itself objective and current rather than into adding more tiers.
• Even the strictest tier needs a defined, logged emergency bypass with mandatory follow-up review; without one, a genuine incident under time pressure pushes people to route around the entire system, not just the one gate.
• A tier assignment that never gets revisited goes stale: a service that started as a low-risk internal tool and later became customer-facing needs to move tiers, or the framework quietly stops matching reality.
• A purely bottom-up model, where each team sets its own bar, maximizes velocity but forfeits consistency and makes cross-team incident review harder to standardize; a purely top-down bar maximizes consistency but ignores real differences in risk. The tiered model is a deliberate middle path, and it has its own ongoing cost: someone has to keep the tier assignments and decision rules current.

Direct answer

A pattern of repeated releases with insufficient testing is a process failure, not a string of individual mistakes, so the fix is not "write more tests." I would change three things at once: what blocks a release in the CI (continuous integration) pipeline, who is explicitly accountable for a service's test coverage, and what production signal tells us a release actually behaved. I would sequence a set of short-term stopgaps to stop the bleeding within weeks and a set of longer-term structural changes to prevent recurrence over the following quarter or two, and I would define upfront how I'd know it worked.

Structured elaboration

Short-term actions (first two to three weeks):

• Test gating: if the existing tests are advisory (visible but non-blocking) in the pull request (PR), make them a hard-blocking gate for the specific area the postmortem implicated. Add a short "risk checklist" a reviewer must sign off on for changes touching that fragile area, so the team isn't waiting on a large test-suite rewrite to get some protection immediately.
• Ownership: assign a named owner for each service that shipped broken, someone whose review is required before merge to that service. "The team owns it" diffuses into nobody owning it; a name on the door changes behavior fast.
• Monitoring: add or tighten alerting on the exact symptom that escaped (the specific error rate, latency percentile, or business metric the postmortem traced back to), and make sure it pages a real person, not just logs to a dashboard nobody watches.

Long-term actions (over the following one to two quarters):

• Test gating: replace ad hoc gating with an explicit definition of done per service (a coverage floor, required test types for the risk profile of that service, required review depth) codified as an automated merge check, not a wiki page people forget.
• Ownership: rotate a quality-owner or on-call role so the responsibility survives any one person leaving, and put escaped-defect counts into the team's own goals alongside velocity, so quality has a visible cost when it slips.
• Monitoring: build a small dashboard tracking defect escape rate (bugs caught in production instead of before release) and change failure rate (the fraction of releases that cause an incident or rollback) over time, so a regression in practice shows up before it becomes the next postmortem.

Worked example

Say the postmortem shows this team shipped 20 releases last quarter and 4 of them caused a customer-visible incident traceable to insufficient testing: that's a 4/20 = 20% change failure rate attributable to this cause. I would use that 20% as the baseline, not a target I've already hit. The short-term gate plus named ownership would be aimed at services responsible for those 4 incidents specifically, since that is where the risk actually concentrated. Over the next two quarters I would expect the same 20-release volume to show that number trending down (for example, 2 of 20 next quarter, 1 of 20 the quarter after) tracked on the dashboard above, and I would present the trend, not a single before/after snapshot, since one quarter can be lucky or unlucky.

Trade-offs and pitfalls

• Making every check blocking on day one can stall the team's velocity and provoke bypasses (self-approved overrides) that quietly defeat the gate; sequence advisory-then-blocking, or blocking only for the specific known-risky area first, rather than gating everything at once.
• A single named owner without a shared bar just creates a new bottleneck or single point of failure; pair ownership with a written standard so the owner is enforcing a shared rule, not their personal taste.
• Watch the metric you actually care about (defect escape rate, change failure rate) rather than a proxy like raw ticket count or number of tests added, which can go up while quality stays flat or drops.
• A postmortem-driven change that never gets revisited becomes theater; put a review date on the gate itself so it gets loosened if it turns out to be miscalibrated, not just tightened forever.

Direct answer

Roll mandatory gates out advisory-then-blocking through a small set of volunteer pilot teams first, pair the rollout with real tooling and performance work that removes the actual friction, not just the policy, run a time-boxed exemption process so a team isn't stuck failing a gate it can't yet pass, and gate the decision to widen enforcement on a published dashboard threshold rather than a fixed calendar date.

Structured elaboration

Pilot selection: two to four teams, a mix of one high-trust, high-maturity team to prove the gate can work smoothly and one team with known CI pain to surface real friction early, running in advisory mode (the gate reports failures but doesn't block) for four to six weeks before flipping to blocking.

Exemption policy: any team can request a temporary, time-boxed exemption, two to four weeks, with a named reason and an owner. Exemptions are visible on the dashboard, not silent, and expire automatically instead of becoming permanent by default. A growing exemption backlog is itself a signal to slow the rollout, not a reason to push through it.

Tooling and performance improvements to minimize friction: parallelized test execution, test-impact analysis so a pull request only re-runs the tests its change actually affects, cached build artifacts between runs, and clear, actionable failure messages linked to the specific failing check instead of a bare red X. These matter because a slow or opaque gate is what actually drives resistance, not the underlying policy.

Metrics dashboard: gate pass rate, median time the gate adds to a pull request, active exemption count, and defect-escape trend, all visible to every team, not only leadership. This dashboard is the widen-enforcement trigger itself, not a status page reviewed after the decision is already made.

Widen-enforcement decision: define the threshold in advance, for example the pilot teams sustaining blocking enforcement with a stable or declining pull request cycle time and a shrinking exemption count over two consecutive review cycles, before expanding to the next wave, and expand in readiness-based waves rather than all at once.

Worked example

The pilot pairs the data-platform team, which has real CI pain from a flaky integration test (one that sometimes passes and sometimes fails with no code change, usually from timing or environment issues), with the payments team, which has high process maturity. During four weeks of advisory mode, the dashboard shows the gate would have blocked roughly a third of the data-platform team's pull requests, almost entirely on the one flaky test, while the payments team is barely affected. The fix quarantines and repairs the flaky test and adds test-impact analysis so data-platform pull requests only re-run affected suites, meaningfully cutting typical gate runtime. Both teams flip to blocking in week five; two smaller teams each request a three-week exemption for a pending migration. At the week-eight steering review, the dashboard shows the exemption count trending toward zero and pass rate stabilizing, so wave two, the next six teams chosen by lowest current gate-failure rate, is approved to start in advisory mode.

Trade-offs and pitfalls

Advisory-then-blocking is slower than flipping a switch company-wide, but a mandate imposed before the tooling is fast enough just relocates the pain into missed deadlines and exemption requests, and burns the credibility the later waves depend on. An exemption process that's too easy to get becomes the de facto opt-out and the gate never actually enforces anything; one that's too hard invites shadow workarounds, like disabling the check locally. The time-box and the visible dashboard exist specifically to keep the exemption a pressure-release valve, not a loophole. A dashboard nobody actually looks at doesn't change behavior, it has to be part of the real widen-or-hold decision meeting, not just a status page people glance at occasionally.