Quality Metrics and Test Reporting Questions

Direct answer

As shown, not ready: I would recommend a conditional no-go, meaning hold the release until three specific checks pass, then ship it gradually. The process numbers (pass rate, coverage) look tolerable, but the two numbers closest to customer harm are the weak ones: 2 critical escapes (severity-1 defects, meaning outage or data loss, found in production after release) in 30 days, and a 36-hour time to restore.

Structured elaboration

How I read each number

What else I need

• The failing tests, filtered to areas this release touched.
• Open defects by severity.
• Whether a fast rollback or feature flag exists. A working rollback or flag usually restores service in minutes to an hour, so a 36-hour restore suggests fixes are being built, tested and deployed from scratch. That is an inference to confirm, not a fact.
• The trend for each number, not only the snapshot: is 2 escapes better or worse than last quarter?
• The customer impact of the change: a payment feature and a colour change carry different risk.

Recommendation for the product manager (the go/no-go). Go only if all three hold by the agreed date:

1. Every failing test is classified as real, flaky or environment, with no open real failure in changed areas.
2. Both critical escapes are fixed and each has a regression test (a test that fails if that same defect returns).
3. The release ships behind a feature flag (a switch that turns the change on or off without a new deploy) or staged (for example 5 percent, then 25, then 100) with a rollback (return to the previous version) rehearsed and a restore target of under an hour.

Otherwise hold. I would not make the coverage number a blocker: 42 percent is a risk to monitor, not a reason to stop this release.

Worked example

Why pass rate alone misleads: on 1,000 tests a 96% pass rate is 40 failures. If 6% (60 tests) are flaky, those 40 failures could be entirely noise or entirely real, and the dashboard cannot tell you which. Rerunning the 40 failures three times each and seeing, say, 30 pass on rerun and 10 fail every time would leave 10 real suspects to investigate, which is a far smaller and more honest number than "4 percent failing".

Trade-offs and pitfalls

• I would not say "go" on a 96 percent pass rate, nor a flat "no" without conditions, which only stalls the team.
• If the failures triage as flaky or environment and rollback is proven fast, I would say go with the staged plan.
• If a single real failure is in the payment or data-loss path, it is a hard no regardless of the rest.
• Do not try to fix all flakiness before release; fix the flakiness that blocks the decision.

Direct answer

Keep one source of truth and one definition, and change only the zoom: team leads get detail they can act on this week, product managers get impact and trend tied to their plan, executives get one number, its direction and the decision needed. I will use defect escape rate (the share of a release's defects found only after release) as the worked metric.

Worked example data (illustrative)

Last four releases: escaped defects out of all defects found: R1 18 of 60 (30%), R2 15 of 50 (30%), R3 9 of 45 (20%), R4 6 of 40 (15%). In R4, the 6 escapes were checkout 4, search 1, profile 1.

What each audience sees

Principles

• Same numbers, different zoom: never change values between audiences, only how much detail sits under them.
• Every view answers "so what?": leads see what to fix, product managers see what it means for the plan, executives see what decision is needed.
• Show denominators for small samples: R4's 15% is 6 of 40; one more escape moves it by 2.5 points, so avoid over-precise claims.
• Choose forms deliberately: tables for action, line charts for trend, stacked bars for composition. Avoid pie charts and 3D, and never rely on red and green alone (add a text label so colour-blind readers get the status).
• Fewer metrics upward: an executive page with 20 metrics is not read; keep to three or four and link down for detail.

Pitfalls

Do not simplify to the point of hiding severity (15% of trivial defects versus 15% of critical ones), and do not present a single good-looking figure without the context that would let a non-technical reader challenge it.

Direct answer

Show it as a chain with a comparison at the end: quality problem, then customer-visible symptom (errors, support contacts), then behaviour (conversion, retention, churn meaning customers leaving). The cleanest evidence is a comparison between users who got the fixed build and users who did not. Be plain about what the data can and cannot prove: quality work correlates with better outcomes, and a controlled comparison makes the causal story credible.

Structured elaboration

What data to join

• Releases and deployments: release id, deploy time, flag or rollout cohort.
• Defects and incidents: introduced-in release, severity, affected feature, time to detect and restore, rollbacks.
• Product analytics events: user id, funnel step (for example checkout completed), which build the user saw.
• Support tickets and churn: contacts per 1,000 users and cancellations, per user id.
• Join at user level by build and feature exposure. Release-level joins have only a handful of rows and are dominated by other things.

A tiny sample of the joined table (the join key is user_id, plus the build the user was served):

Grouping this table by build and averaging checkout_completed gives the conversion rates compared below.

What to compute

1. Symptom rates: error rate and support contacts per 1,000 users, per build.
2. Outcome difference between cohorts: conversion (share of visitors who complete a purchase) or 30-day retention (share still active after 30 days) for users on each build.
3. A confidence interval: the range the true difference plausibly lies in, not just the point estimate.
4. Cost of quality: what escapes cost (engineer-hours on triage, hotfix and communication, plus support load) versus what prevention and testing cost. Example: 2 escapes a month at 30 engineer-hours each is 2 x 30 = 60 hours a month (an assumption to replace with your own tracked hours).

Worked example

A fix was rolled out to a randomly chosen half of users first, so half were still on the buggy checkout build and half on the fixed one. Save as rollout_compare.py:

import math

# standard error (se) = how much a measured rate would typically wobble from sampling luck alone
# Fix rolled out to a randomly chosen half of users first: half still on the buggy build, half on the fixed build
n_bug, conv_bug = 5000, 540
n_fix, conv_fix = 5000, 600
p_bug, p_fix = conv_bug / n_bug, conv_fix / n_fix
diff = p_fix - p_bug
se = math.sqrt(p_bug * (1 - p_bug) / n_bug + p_fix * (1 - p_fix) / n_fix)
print("conversion bug build %.1f%%, fixed build %.1f%%" % (100 * p_bug, 100 * p_fix))
print("difference %.2f points, 95%% CI %.2f to %.2f" % (100 * diff, 100 * (diff - 1.96 * se), 100 * (diff + 1.96 * se)))

Output:

conversion bug build 10.8%, fixed build 12.0%
difference 1.20 points, 95% CI -0.05 to 2.45

The interval is built as the difference plus or minus 1.96 standard errors. The standard error is how far a measured rate would typically wander by luck if you repeated the comparison, and 1.96 of them either side covers the middle 95 percent of those luck-driven outcomes, so the interval is the plausible range for the true difference. The fixed build converted 1.20 points better, but the 95 percent interval runs from about -0.05 to 2.45 points, so at 5,000 users per group the evidence is borderline, not conclusive. If 100,000 checkouts start a month, 1.20 points is about 1,200 orders (100,000 x 0.012), with a plausible range of roughly -50 to 2,450. The honest report is "likely helped, needs more traffic or a longer window", not "quality earned us 1,200 orders".

Trade-offs and pitfalls

Correlation versus causation

• The staged, randomized rollout above is the core method; the rest of this section explains why it beats the alternatives, and the last bullet is a fallback you will rarely need.
• Correlating releases with monthly churn is weak: releases differ in size, season and marketing. A confounder is a third factor that moves both the thing you changed and the outcome (for example a holiday sale that lifts conversion and also delays releases). Reverse causation means the arrow points the other way: for example, a team that sees engagement falling may rush out hurried fixes, so the churn drives the release activity rather than the releases driving the churn. (A busy season that pushes both releases and churn is the confounder case above, a different problem.)
• A randomized rollout (chance, not people, decides who gets the fix first) removes those confounders, because both groups face the same season and marketing.
• Without randomization, use matched cohorts (pair each exposed user with an unexposed one of similar plan, country and activity) or difference-in-differences (compare the before-and-after change in exposed users with the change in similar unexposed users) and describe results as "consistent with".
• Never expose users to a known defect just to run an experiment; use rollouts that happen anyway.

Trade-offs

• Argue the chain link by link: escapes to support contacts, contacts to incidents and rollbacks, incidents to churn. Each link is easier to defend than one leap to revenue.
• Report intervals and assumptions with every order or revenue figure.

Direct answer

Assume any number that becomes a target will be gamed, since Goodhart's law says a measure stops being reliable once it is used as a target. My guardrails make gaming visible and unrewarding: pair each metric with a counter-metric, show denominators, collect data automatically, define metrics in reviewed code, and use the numbers to guide conversations rather than to rank people. Then I keep measurement useful by favouring trends over targets and retiring metrics that stop informing decisions.

Worked example: gaming a pass rate

A suite has 500 tests, 25 are failing. The team "fixes" the dashboard by quarantining those 25 (moving them out of the blocking set and out of the pass-rate count). The dashboard now says 475 of 475 passed, 100%. The honest figure is 475 / 500 = 95%, with 25 known problems hidden. A guardrail that shows the quarantined count next to the pass rate, and expires quarantines, defeats this.

Terms used below: coverage is the share of code lines that tests execute. A mutation score comes from a tool that makes tiny deliberate breakages (mutants, for example flipping > to >=) and reports the share of mutants that at least one test catches, so it measures whether tests actually check behaviour. A flaky test passes and fails on identical code. CI (continuous integration) is the automated system that builds and tests every change. A quarantine list holds tests that still run but no longer block builds.

Worked example: counter-metric pairing (illustrative numbers)

A team is told to raise coverage from 70% to 90%. They add tests that call code but assert nothing. Coverage reaches 92%, but of 50 mutants the tool creates, the tests catch only 20 both before and after: mutation score 20 / 50 = 40%, unchanged. Shown alone, coverage says "much better"; shown with the mutation score, it says "no real improvement". The same logic pairs speed with escape rate: if release cycle time drops from 10 days to 5 while customer-found defects per release (escapes) rise from 3 to 9, the pair shows speed was bought with quality.

Guardrails, why each works, what it costs

Keeping the measurement useful

• Look at trends and anomalies, and ask "what happened?" instead of "who did this?".
• Involve the team in choosing metrics so they own them.
• Review quarterly and retire metrics that never change a decision.
• Do not over-guard: a dashboard so locked down that it stops being read is worse than a slightly gameable one.

Pitfalls

Guardrails only reduce gaming, they cannot stop a determined team. Culture matters more: if numbers are punished, people hide problems, and the best guardrail is being safe to report bad news.

Direct answer

Code coverage (the share of lines or branches, meaning the true and false sides of each decision, that tests execute) tells you what is definitely untested. It does not tell you what is well tested: a test that runs a line without checking anything counts the same as a strong one. So 90 percent is not proof of quality. Beside it I would report mutation score (the share of deliberately planted bugs that the tests catch), requirement and risk coverage, escape rate (the share of defects that reached customers), flaky rate (the share of tests that pass and fail on identical code) and the coverage of changed code. If you only have room for three, start with mutation score, escape rate and coverage of changed code; the rest are useful extras.

Structured elaboration

What it hides

• Executing code is not asserting on it: an assertion is the line in a test that checks a result (for example assert fee(60) == 0), and tests with no assertions still produce coverage.
• Boundaries and business rules: reaching an if says nothing about whether the edge value is right.
• What is missing: code never written for a requirement has no lines to cover.
• The uncovered 10 percent may be the riskiest part (error handling, payments).
• Generated code, trivial getters and tests written to hit the number can inflate it.

What to report beside coverage, and how to present it to non-technical stakeholders

Escape rate is defects found in production divided by all defects found for a release. Example: 12 defects found for a release, 2 of them in production, so 2 / 12 = 17 percent. Flaky rate is flaky tests divided by all tests; 30 flaky tests out of 1,000 is 3 percent.

Statement, branch and condition coverage are finer ways to count what code ran (a statement is one line, a branch is one side of an if, a condition is one true/false part inside an if). You do not need them for this question, only that "90 percent" usually means one of the first two. The point here is which numbers to place beside it.

Worked example

Save as mutation_demo.py. A mutant is a copy of the code with one small deliberate bug; a good suite "kills" it by failing. Mutation score is mutants killed divided by mutants created. The first suite runs both branches, so line and branch coverage are both 100 percent, yet it kills nothing.

# Original code and two hand-made "mutants" (small deliberate bugs)
def fee_original(total):
    if total >= 50:
        return 0
    return 5

def fee_mutant_a(total):      # wrong constant: free shipping now costs 5
    if total >= 50:
        return 5
    return 5

def fee_mutant_b(total):      # boundary bug: >= became >
    if total > 50:
        return 0
    return 5

def suite_no_asserts(fee):    # runs every line and both branches, checks nothing
    fee(60)
    fee(10)

def suite_basic_asserts(fee): # asserts the two obvious cases
    assert fee(60) == 0
    assert fee(10) == 5

def suite_with_boundary(fee): # adds the boundary value
    assert fee(60) == 0
    assert fee(10) == 5
    assert fee(50) == 0

def survives(suite, fee):     # True = the suite did NOT notice the bug
    try:
        suite(fee)
        return True
    except AssertionError:
        return False

mutants = {"A wrong constant": fee_mutant_a, "B boundary": fee_mutant_b}
suites = {"no asserts": suite_no_asserts,
          "basic asserts": suite_basic_asserts,
          "asserts + boundary": suite_with_boundary}
for sname, suite in suites.items():
    suite(fee_original)       # every suite passes on the real code
    killed = sum(not survives(suite, m) for m in mutants.values())
    print("%-20s original passes; mutants killed %d of %d -> mutation score %d%%"
          % (sname, killed, len(mutants), 100 * killed // len(mutants)))

Output:

no asserts           original passes; mutants killed 0 of 2 -> mutation score 0%
basic asserts        original passes; mutants killed 1 of 2 -> mutation score 50%
asserts + boundary   original passes; mutants killed 2 of 2 -> mutation score 100%

Same coverage, three very different levels of protection. Even the second suite, with real assertions, misses the boundary bug at exactly 50.

Trade-offs and pitfalls

• A coverage target invites tests written for the number, not the risk.
• Prefer the trend and coverage of changed code over one global figure.
• Escape rate is the outcome; mutation score is a check on the tests themselves. Mutation runs are slow on big codebases, so run them on changed or critical modules.