Quality Metrics and Test Reporting Questions

Direct answer

Six tiles on one screen (read by a product manager, QA and engineers): a release verdict, critical escapes, gating-test health, recovery speed, change failure rate and open defects. Each tile shows one number, a small trend line (sparkline) and a colour, and each has a guardrail (the line that turns it amber or red) and a drill-down for engineers. Coverage and raw test counts stay out of the top level; they live one click down as context.

Structured elaboration

The tiles

DORA's current name for the recovery metric is failed deployment recovery time (its older name was MTTR); the tile above uses incident-level detect and restore times, a close cousin. Unplanned deployments caused by production incidents are counted separately by DORA, as deployment rework rate.

How each shows its signal. Number first, sparkline second, colour third, and every tile states its window ("last 30 days") and last-refresh time, so a stale tile cannot pass for current.

One alert. A severity-1 defect opened against the release candidate (the build proposed for release) posts to the release channel and pages the QA lead (sends an urgent phone alert). Everything else stays passive on the screen so the alert keeps meaning something.

What one tile looks like

+------------------------------------+
| Critical escapes      (last 30 days)|
|   1        [sparkline: 0 0 1 0 0 1] |
|   RED: 1 newer than last release    |
| updated 09:05 | click: escape list  |
+------------------------------------+

Big number on top, sparkline (a tiny trend line with no axes) beside it, one line saying why it has its colour, then window, refresh time and the drill-down link at the bottom.

Worked example

Suppose today the screen reads: 1 critical escape found after the last release (red), first-attempt pass rate inside the team's normal band (green), flaky rate steady (green), median restore time inside target (green), change failure rate 7.5 percent, which is 3 failed deployments out of 40 (green against a previous median of 10 percent), and no open severity-1 defects (green). The verdict tile is red with one line: "Critical escape has no regression test." An engineer clicks the escapes tile, sees the defect, its owner and the missing test link, and the fix and test are added; the tile then turns green on the next refresh.

Trade-offs and pitfalls

• Fewer tiles beat more: product wants a verdict, QA wants test health, engineering wants causes, and six tiles let each find theirs.
• Never show pass rate without flaky rate beside it, or a retry-heavy suite looks healthy.
• The guardrail numbers here are starting points; calibrate each to your own baseline rather than copying them.

Direct answer

Defect detection effectiveness (DDE), also called defect removal efficiency (DRE), is the share of all defects for a release that were caught before release. It measures how good your pre-release testing was, but it is a poor sole basis for a release decision because it looks backward, ignores severity, and can be inflated by logging trivial bugs.

Formula

DDE = defects found before release / (defects found before release + defects found after release) x 100

Intuition: of every 100 bugs the release ever had, how many did testing stop?

Worked example

A sprint's release had 45 defects found in testing. In the 30 days after release, users and monitoring found 5 more that were traced to that release.

DDE = 45 / (45 + 5) = 45 / 50 = 90%

Data sources and the lag problem

• Pre-release defects: bug tracker entries linked to the release, plus failed test runs that led to a fix.
• Post-release defects: bug tracker entries and support tickets traced back to the release, matched via commits or fix-version fields.
• Lag: late-found defects keep arriving. At 30 days the number is 90%. If 3 more surface by day 60, then 45 / (45 + 8) = 45 / 53 = about 84.9%. The score for a release falls over time, so always state the window (and compare releases only at the same age).

Why it is a poor sole basis for a release decision

1. Backward-looking. It scores a release that has already happened; you cannot compute it before shipping.
2. Ignores severity. Catching 45 cosmetic bugs and missing one payment bug still yields 90%.
3. Depends on effort and definitions. Hunting harder for easy bugs raises it; recording duplicates or trivial findings inflates it. A release with 2 found and 0 escaped scores 100% while covering almost nothing.
4. Denominator is unknowable in real time. The bugs you have not found yet are not in it.
5. Says nothing about untested areas. No escapes may mean no users in that area yet.

What to use with it

Pair it with severity-weighted escapes, release-gate test status, and coverage of changed code. Use DDE as a trend across releases, not a pass/fail gate.

Direct answer

A healthy suite is one the team trusts, that gives fast feedback, that catches real defects, and that costs a sustainable amount to maintain. No single number shows all four, so I track a small set of metrics, one group per question, and for each I know how it is measured, which trend worries me, and the one action it triggers. (CI means continuous integration, the automated system that builds and tests every change. A flaky test is one that passes and fails on the same code. Quarantine means moving an unreliable test out of the gate that blocks merges. p95 is the value 95% of observations fall below, so it describes the slow tail rather than the average, and p50 is the median. Test rot is a suite decaying as the product changes and tests stop matching it. Escaped defects are bugs found after release. Wall-clock time is elapsed real time, not CPU time. To shard and parallelise is to split the suite into pieces that run at once on several machines. A runner is the machine that executes CI jobs. A triage rota is a rotating on-call duty to look at red builds. Hard-coded waits are fixed sleeps, such as waiting 5 seconds for a page, instead of waiting for a condition. If you are starting from nothing, begin with pass rate, flake rate and duration; the cost and deflection rows are organisation-level extras.)

The metrics, grouped by the question they answer

Reading the table by question: can I trust the suite (pass rate, flake rate, quarantine size and age); is it fast (duration and queue time, time to green); does it find bugs (defect escape rate); is it affordable (maintenance cost per test, manual-test deflection).

Targets, cadence and the organisation view

• I set targets against each team's own baseline first (for example, "no quarantined test older than a fixed number of days"), not a universal number borrowed from another company.
• The team reviews a one-page view weekly; an engineering manager reviews the trends monthly with the same definitions rolled up across teams. The organisation dashboard adds maintenance cost per test and manual-test deflection, because those are the numbers that show whether automation is paying for itself.

Worked example

A team has 1,200 tests. A quarter ago it spent 42 engineer-hours a month on test upkeep. Now it spends 60.

• Then: 42 h x 60 min / 1,200 tests = 2.1 minutes per test per month.
• Now: 60 h x 60 min / 1,200 tests = 3.0 minutes per test per month, about 43% more (60 / 42 = 1.43).
• Meanwhile pass rate stayed at 98% and quarantine grew from 10 to 45 tests. The pass rate alone says "fine"; the cost and quarantine metrics say the suite is quietly decaying. The action is to review the 45 quarantined tests, delete or fix them by their expiry date, and look at which tests take the most upkeep.

Defect escape rate traced: a release had 15 defects, 12 caught before release and 3 escaped, so 3 / (3 + 12) = 20%. If pass rate stayed 98% while that rate went from 20% to 30% over two releases, the suite is passing yet missing more.

Trade-offs and pitfalls

• Any single metric can be gamed (skipping tests lifts pass rate, deleting tests lifts everything), so I pair each with a counter-metric, a second number that exposes the cheat: pass rate with skipped and quarantined counts, duration with escape rate (a suite made fast by deleting tests will show escapes rising).
• Too many metrics means none is acted on. Each one must map to a decision; if it never changes a decision I drop it.
• Raw line coverage is deliberately absent: it says code was executed, not that behaviour was checked. I prefer coverage of critical user paths.

Direct answer

Defect escape rate is the share of a release's defects that got past your pre-release testing and were found afterwards. It is defects found after release divided by all defects found for that release. For this release: 70 (customers) + 20 (operations) = 90 escaped out of 120 total, so the escape rate is 90 / 120 = 75%. That is high: only a quarter of known defects were caught before release.

Calculation, step by step

• Caught before release by QA (quality assurance testing): 30
• Escaped (found after release): 70 by customers + 20 by operations = 90
• Total logged for the release: 30 + 90 = 120
• Escape rate: 90 / 120 = 0.75, or 75%
• Its mirror, the defect detection percentage (share caught before release): 30 / 120 = 25%
• As a ratio of escaped to caught: 90 / 30 = 3.0, so three defects escaped for every one caught

Escape versus leakage

Teams use the two words loosely, so state your definition. A common split: escape means the defect reached production (users or operations); leakage means a defect slipped past the stage that should have caught it (for example found in system test when a unit test should have caught it), which you can measure at every stage. Every escape is a leak, but not every leak escapes. With this release's 120 defects: the 90 escapes reached production, so the escape rate is 90 / 120 = 75%. Now suppose 10 of the 30 defects QA caught were found in system test although a unit test should have caught them (an illustrative split). Those 10 leaked past the unit-test stage, yet they did not escape. Leakage from the unit stage is therefore 10 defects (a third of the 30 caught before release), while the escape rate is unchanged at 75%. Some organisations use leakage as a synonym for escape, so confirm the local convention before comparing numbers.

What I would conclude about risk

The known risk is high, but I would qualify it before acting:

• Severity: 85 cosmetic escapes and 5 critical ones are very different stories, even though both make 90 escapes (85 + 5). Split the rate by severity.
• Who found them: 70 by customers is customer-visible pain; 20 by operations may point to configuration (settings differing between environments) or environment gaps that QA tests do not reach, so treat that group separately.
• Under-counting: customers do not report everything, so 75% is a floor for customer-visible issues.
• Window: fix the measurement window (for example the first 30 days after release) so releases compare fairly.

Where to focus testing

Do not "test more everywhere". Take the 90 escapes and group them by component (a module or part of the product, such as checkout or search), severity and root cause: no test existed, a test existed but missed it, or the environment differed. If, for instance, most severe escapes cluster in one component, add tests and review effort there first. For the 20 operations-found defects, look at environment parity (making the test environment match production in versions, data and settings), configuration testing (checking the settings that change between environments) and monitoring.

Pitfalls

The rate is gamed by not logging defects, closing them as "works as intended", or downgrading severity; cross-check with support tickets and incident counts.

Direct answer

Do not assume the product got worse or that the new tests are bad. First split the drop into the old tests and the new ones, then classify every failure as a real regression, a flaky test, an environment problem or a test-data problem, using reruns and clustering as evidence. Meanwhile keep the original suite as the gate and run the new UI batch as non-blocking, so the pipeline stays trusted.

Structured elaboration

How I investigate

1. Freeze the facts: which commit, which run, which runner. Rerun the same commit to see what is stable. A runner is the machine that executes the CI (continuous integration, the automated build-and-test pipeline) jobs.
2. Split failures by cohort: old suite versus new UI batch, and by first-failure time.
3. Rerun failures in isolation and on a clean runner: deterministic or intermittent?
4. Cluster by error signature, runner and time window. An error signature is the first line of the failure message with volatile bits (ids, timestamps) removed, so identical causes group together.
5. Bisect (search commits for the first bad one) if the old tests dropped.

Evidence that separates the causes

Keeping confidence in the pipeline meanwhile

• Keep the original suite as the gate; its 98 percent baseline is unchanged and still means something.
• Run the new UI batch as non-blocking with a named owner and an expiry date (say two weeks), and publish both pass rates separately every day.
• Track first-attempt pass rate; silent auto-retries hide flakiness and should never be the reported number.
• Admit a new test to the gate only after a run of consecutive green nights (the length is a team convention, for example 20).

Worked example

A quick sum shows why the split matters. Say there were 500 tests with 490 passing (98 percent), and after adding 100 UI tests there are 600 with 552 passing (92 percent).

• If the old tests are unchanged (490 still pass), the new tests passed 552 - 490 = 62 of 100, a 62 percent pass rate. That points at the new tests or their environment.
• If instead all 100 new tests passed, the old ones would be at 552 - 100 = 452 of 500, which is 90.4 percent: a real regression in the old suite.

The same headline number implies opposite investigations, so the cohort split comes first.

Classifying real failures (illustrative)

Non-blocking simply means the batch still runs and reports, but a red result does not stop a merge or release.

Trade-offs and pitfalls

• Blaming the tests reflexively: some failures are real, so triage each one.
• Deleting failing tests, or quarantining with no expiry, turns the quarantine into a graveyard.
• Rerunning until green creates false confidence.
• What would change my call: if the old suite has dropped too, this is no longer a new-test problem, and I would escalate it as a product or environment issue and block the release.