Quality Metrics and Test Reporting Questions

Direct answer

Assume any number that becomes a target will be gamed, since Goodhart's law says a measure stops being reliable once it is used as a target. My guardrails make gaming visible and unrewarding: pair each metric with a counter-metric, show denominators, collect data automatically, define metrics in reviewed code, and use the numbers to guide conversations rather than to rank people. Then I keep measurement useful by favouring trends over targets and retiring metrics that stop informing decisions.

Worked example: gaming a pass rate

A suite has 500 tests, 25 are failing. The team "fixes" the dashboard by quarantining those 25 (moving them out of the blocking set and out of the pass-rate count). The dashboard now says 475 of 475 passed, 100%. The honest figure is 475 / 500 = 95%, with 25 known problems hidden. A guardrail that shows the quarantined count next to the pass rate, and expires quarantines, defeats this.

Terms used below: coverage is the share of code lines that tests execute. A mutation score comes from a tool that makes tiny deliberate breakages (mutants, for example flipping > to >=) and reports the share of mutants that at least one test catches, so it measures whether tests actually check behaviour. A flaky test passes and fails on identical code. CI (continuous integration) is the automated system that builds and tests every change. A quarantine list holds tests that still run but no longer block builds.

Worked example: counter-metric pairing (illustrative numbers)

A team is told to raise coverage from 70% to 90%. They add tests that call code but assert nothing. Coverage reaches 92%, but of 50 mutants the tool creates, the tests catch only 20 both before and after: mutation score 20 / 50 = 40%, unchanged. Shown alone, coverage says "much better"; shown with the mutation score, it says "no real improvement". The same logic pairs speed with escape rate: if release cycle time drops from 10 days to 5 while customer-found defects per release (escapes) rise from 3 to 9, the pair shows speed was bought with quality.

Guardrails, why each works, what it costs

Keeping the measurement useful

• Look at trends and anomalies, and ask "what happened?" instead of "who did this?".
• Involve the team in choosing metrics so they own them.
• Review quarterly and retire metrics that never change a decision.
• Do not over-guard: a dashboard so locked down that it stops being read is worse than a slightly gameable one.

Pitfalls

Guardrails only reduce gaming, they cannot stop a determined team. Culture matters more: if numbers are punished, people hide problems, and the best guardrail is being safe to report bad news.

Direct answer

Defect density is the number of defects found in a piece of software divided by its size. The classic form is defects per KLOC (thousand lines of code). It is a comparison tool for spotting modules or releases that look unusually buggy, but lines of code is a poor ruler, so I would pair it with better denominators and never use it to compare across languages or teams.

Calculation with a small example

Density = defects / (lines of code / 1,000). Four modules:

modules = [
    ("checkout", 18, 12000),
    ("auth", 3, 1500),
    ("reporting", 5, 20000),
    ("generated_stub", 0, 0),
]

def density_per_kloc(defects, loc):
    if loc <= 0:
        return None  # no size means no meaningful density
    return defects / (loc / 1000)

for name, d, loc in modules:
    dens = density_per_kloc(d, loc)
    print(name, "n/a" if dens is None else round(dens, 2))

sized = [(d, loc) for _, d, loc in modules if loc > 0]
total_d = sum(d for d, _ in sized)
total_loc = sum(loc for _, loc in sized)
print("pooled:", round(density_per_kloc(total_d, total_loc), 2))
ratios = [density_per_kloc(d, loc) for d, loc in sized]
print("mean of ratios:", round(sum(ratios) / len(ratios), 2))

Output:

checkout 1.5
auth 2.0
reporting 0.25
generated_stub n/a
pooled: 0.78
mean of ratios: 1.25

• Zero size: return "not available" (None), not 0 and not a crash. A module with no lines has no density.
• Roll-ups: the overall figure is total defects over total size (26 / 33.5 KLOC, about 0.78), not the average of module densities (1.25), which over-weights small modules.
• Rounding: keep raw counts and round only for display.

Why per-KLOC can mislead

• Verbose languages and generated code inflate size and shrink density without any quality change; splitting one line into three also lowers it.
• Small modules swing wildly: auth has the highest density (2.0) on just 3 defects.
• Density reflects detection effort as much as quality: a heavily tested module finds more defects, and a zero-defect module may simply be untested.
• Units matter: defects per KLOC compares only within the same counting rules (are comments, blanks and generated code excluded?). Comparing across languages is the common misreading.

Better denominators

• Function points: a size measure based on features delivered (inputs, outputs, queries, data files), independent of language.
• Changed lines this release (code churn): normalises for how much was actually touched.
• Usage: defects or incidents per million requests or per active user, matching what customers experience.
• Effort or features shipped, as a rough cross-check.

Pair density with severity and pick one denominator per comparison, stated on the chart.

Direct answer

Code coverage (the share of lines or branches, meaning the true and false sides of each decision, that tests execute) tells you what is definitely untested. It does not tell you what is well tested: a test that runs a line without checking anything counts the same as a strong one. So 90 percent is not proof of quality. Beside it I would report mutation score (the share of deliberately planted bugs that the tests catch), requirement and risk coverage, escape rate (the share of defects that reached customers), flaky rate (the share of tests that pass and fail on identical code) and the coverage of changed code. If you only have room for three, start with mutation score, escape rate and coverage of changed code; the rest are useful extras.

Structured elaboration

What it hides

• Executing code is not asserting on it: an assertion is the line in a test that checks a result (for example assert fee(60) == 0), and tests with no assertions still produce coverage.
• Boundaries and business rules: reaching an if says nothing about whether the edge value is right.
• What is missing: code never written for a requirement has no lines to cover.
• The uncovered 10 percent may be the riskiest part (error handling, payments).
• Generated code, trivial getters and tests written to hit the number can inflate it.

What to report beside coverage, and how to present it to non-technical stakeholders

Escape rate is defects found in production divided by all defects found for a release. Example: 12 defects found for a release, 2 of them in production, so 2 / 12 = 17 percent. Flaky rate is flaky tests divided by all tests; 30 flaky tests out of 1,000 is 3 percent.

Statement, branch and condition coverage are finer ways to count what code ran (a statement is one line, a branch is one side of an if, a condition is one true/false part inside an if). You do not need them for this question, only that "90 percent" usually means one of the first two. The point here is which numbers to place beside it.

Worked example

Save as mutation_demo.py. A mutant is a copy of the code with one small deliberate bug; a good suite "kills" it by failing. Mutation score is mutants killed divided by mutants created. The first suite runs both branches, so line and branch coverage are both 100 percent, yet it kills nothing.

# Original code and two hand-made "mutants" (small deliberate bugs)
def fee_original(total):
    if total >= 50:
        return 0
    return 5

def fee_mutant_a(total):      # wrong constant: free shipping now costs 5
    if total >= 50:
        return 5
    return 5

def fee_mutant_b(total):      # boundary bug: >= became >
    if total > 50:
        return 0
    return 5

def suite_no_asserts(fee):    # runs every line and both branches, checks nothing
    fee(60)
    fee(10)

def suite_basic_asserts(fee): # asserts the two obvious cases
    assert fee(60) == 0
    assert fee(10) == 5

def suite_with_boundary(fee): # adds the boundary value
    assert fee(60) == 0
    assert fee(10) == 5
    assert fee(50) == 0

def survives(suite, fee):     # True = the suite did NOT notice the bug
    try:
        suite(fee)
        return True
    except AssertionError:
        return False

mutants = {"A wrong constant": fee_mutant_a, "B boundary": fee_mutant_b}
suites = {"no asserts": suite_no_asserts,
          "basic asserts": suite_basic_asserts,
          "asserts + boundary": suite_with_boundary}
for sname, suite in suites.items():
    suite(fee_original)       # every suite passes on the real code
    killed = sum(not survives(suite, m) for m in mutants.values())
    print("%-20s original passes; mutants killed %d of %d -> mutation score %d%%"
          % (sname, killed, len(mutants), 100 * killed // len(mutants)))

Output:

no asserts           original passes; mutants killed 0 of 2 -> mutation score 0%
basic asserts        original passes; mutants killed 1 of 2 -> mutation score 50%
asserts + boundary   original passes; mutants killed 2 of 2 -> mutation score 100%

Same coverage, three very different levels of protection. Even the second suite, with real assertions, misses the boundary bug at exactly 50.

Trade-offs and pitfalls

• A coverage target invites tests written for the number, not the risk.
• Prefer the trend and coverage of changed code over one global figure.
• Escape rate is the outcome; mutation score is a check on the tests themselves. Mutation runs are slow on big codebases, so run them on changed or critical modules.

Direct answer

Busy testing is measured by activity (tests run, cases written, tickets closed). Effective testing is measured by outcomes: are defects caught before customers see them, and is the signal fast and trustworthy enough that people act on it? I would track escapes, whether tests would actually notice a break, and the health of the pipeline, set targets as directions against my own baseline, and distrust any number that improves while customer experience does not.

What I would track (the core set is three items: escape rate, flaky rate and feedback time; the root-cause tag, mutation score, runtime signals and change failure rate are extras for teams that already have the basics)

• Defect escape rate, weighted by severity: defects found after release divided by all defects for the release, where each defect counts by its weight (for example Sev1 = 8, Sev2 = 4, Sev3 = 1; the weights are a team choice, illustrative here). The core outcome measure. Example: a release has 10 defects. Escaped: one Sev1 and one Sev3 (8 + 1 = 9 points). Caught before release: two Sev2 and six Sev3 (8 + 6 = 14 points). Weighted escape rate = 9 / (9 + 14) = 39%, while the unweighted rate is 2 / 10 = 20%. The weighted view shows that the one escape that mattered was serious.
• Escape root-cause tag: for each escaped defect, label whether a test existed and missed it, no test existed, or the environment could not reproduce it. This tells you what to fix.
• Mutation score (supplement to coverage): a mutation testing tool makes small deliberate changes to code (for example flipping a > into >=) and reruns the tests. If the tests still pass, that "mutant" survived, which means no test really checks that behaviour (an assertion is the line in a test that checks a result; survivors mean such lines are missing). Example: 200 mutants generated on the modules changed this month, 150 killed, so the score is 150 / 200 = 75%. The 50 survivors are a to-do list of missing assertions. Mutants that behave identically to the original code (equivalent mutants) inflate survivors, so review before chasing.
• Flaky test rate and feedback time (how long from push to result): a suite people do not trust or wait for is not effective.
• Runtime signals: production error rates and canary results (releasing to a small share of users first) show what tests missed, and feed the root-cause tag.
• Change failure rate: the share of deployments causing an incident or rollback, a delivery-level check. Example: 40 deployments, 3 needed a rollback, so 3 / 40 = 7.5%.

Coverage (the share of code lines that tests execute) is one input here, never the verdict.

Targets

Choose directional, baseline-relative targets you can defend: halve the flaky rate this quarter; no Sev1 escape (a top-severity defect such as an outage or data loss) in an area we claim is covered; mutation score on changed code trending up; feedback time under the budget developers agree they will wait for. Avoid a single global number such as "90% coverage".

What should make you distrust the numbers

• Pass rate or coverage rising while escapes stay flat (tests without assertions, deleted or quarantined tests).
• Escape rate falling while support tickets or incidents do not (defects unlogged, downgraded or reclassified).
• Mutation score high because only trivial mutants were generated.
• Everything improving right after a target is announced.

Improving reliability without slowing delivery

Run fast tests on every pull request (a proposed code change awaiting review) and slower suites after merge; run mutation testing only on changed files, nightly; quarantine flaky tests (move them out of the gate that blocks merges, still visible and running) with an owner and an expiry date rather than blanket retries. Pipeline health means the automated build-and-test system being fast and trusted, not just green.

Direct answer

A release-readiness setting is the moment a team decides whether a build is fit to ship, and the metrics that feed that decision. A quality metric is a measurement of the product, or of the process that builds it, that helps someone make a decision about quality. A meaningful one changes a decision and is hard to improve without genuinely improving quality. A vanity metric looks impressive, can only trend upward, and nobody would act differently if it moved.

How to tell them apart (four questions)

1. Decision test: if this went red, what would we do, and who does it? No answer means vanity.
2. Outcome link: does it connect to something users or the business feel (incidents, escapes, churn), or only to activity?
3. Gaming test: could a team move it a lot without improving the product?
4. Definition test: is the numerator, denominator and time window fixed, so two people compute the same value?

Release-readiness examples

How to validate that a metric is worth tracking

Back-test it (compare the metric's past values with how those releases actually turned out) before adopting it. Illustrative data for four past releases:

Pass rate stays between 97% and 99% while incidents range from 0 to 4, so it does not discriminate (it fails to separate good releases from bad ones). Open severe defects orders the releases almost the same way as incidents (ranked worst to best by open severe defects: R1 with 6, R3 with 4, R2 with 1, R4 with 0; ranked by incidents: R1 with 4, R3 with 3, R4 with 1, R2 with 0. Only R2 and R4 swap places), so it is worth keeping. Four points is a sanity check, not statistics, so keep watching it. Then pilot it for two or three releases, confirm the definition is written down, check whether it moved for a real reason when it moved, and retire it if it never changes a decision.

Pitfalls

A metric can be meaningful for one team and vanity for another. Do not add a metric because it is easy to collect, and do not set a target until you know how it behaves.