Quality Metrics and Test Reporting Questions

Overview
Below are 7 actionable QA metrics tailored for Product Managers and Engineering Managers. For each: calculation, data source, reporting cadence, and thresholds/actions are provided from a QA engineer perspective.

1. Escaped Defects

• Calculation: Count of production bugs / total bugs found in release
• Data source: Issue tracker (Jira) tagged "production" + release tag
• Frequency: Per release and weekly roll-up
• Thresholds/Actions: >5% → hotfix sprint + root-cause analysis (RCA); 1–5% → prioritize regression tests for impacted areas

2. Severity 1/2 Defect Rate

• Calculation: # Sev1+Sev2 defects per 1000 users or per release
• Data source: Issue tracker severity field + telemetry
• Frequency: Weekly + per-release
• Thresholds/Actions: Any Sev1 within release → immediate incident review; trend upward → allocate dev/QA pair to stabilization

3. Automation Pass Rate

• Calculation: Passed automated tests / total automated tests executed
• Data source: CI system test reports (Jenkins/GitHub Actions)
• Frequency: Per CI run and daily summary
• Thresholds/Actions: <90% → pause merge, investigate flaky tests; improving trend → increase automation coverage

4. Test Coverage (functional)

• Calculation: % of planned test cases executed for critical user journeys
• Data source: Test case management (TestRail) mapped to features
• Frequency: Per sprint and release
• Thresholds/Actions: <85% on critical flows → block release until coverage improved or risk accepted by PM

5. Test Cycle Time

• Calculation: Average time from test start to completion per test run
• Data source: Test management timestamps
• Frequency: Sprint-level
• Thresholds/Actions: >48 hrs average → identify bottlenecks (env provisioning, flaky tests) and optimize

6. Mean Time to Detect (MTTD)

• Calculation: Avg time from defect introduction (commit) to detection
• Data source: VCS commit timestamps + defect report timestamps
• Frequency: Monthly
• Thresholds/Actions: Rising MTTD → shift-left initiatives (earlier integration tests, more unit tests)

7. Flakiness Rate

• Calculation: # flaky failures / total test runs for automated suite
• Data source: CI historical test results with flakiness tagging
• Frequency: Daily/weekly
• Thresholds/Actions: >5% → quarantine flaky tests, require fixes before re-enabling; maintain flaky-test dashboard

Closing note: Present these metrics in a single dashboard view with filter by release/component, trend charts, and drilldowns to failing tests/defects so PMs and Eng Managers can both monitor risk and assign follow-up actions.

Definition (QA perspective)

• MTTD (Mean Time To Detect): average time from incident occurrence (or symptom start) to when QA/monitoring/ops first detect it. Measures detection effectiveness.
• MTTR (Mean Time To Fix): average time from detection to when the issue is resolved and verified by QA. Measures response and remediation speed.

How to calculate (formula)

MTTD = (sum of detection times) / (number of incidents)
MTTR = (sum of fix times) / (number of incidents)

Example calculation

• Detection times: 2h, 5h, 1h -> sum = 8h
• Fix times: 8h, 2h, 4h -> sum = 14h

MTTD = 8h / 3 = 2.67 hours
MTTR = 14h / 3 = 4.67 hours

How these guide QA improvements

• High MTTD → improve monitoring, alerting, better test coverage (especially end-to-end/production-like tests), synthetic checks.
• High MTTR → streamline triage, improve reproducible bug reports, add automated rollback/patch pipelines, create runbooks and automated test suites to verify fixes faster.
• Use both metrics to set SLAs, prioritize investments (detection vs remediation), and measure impact of process changes over time.

Definition (brief)
Defect Detection Rate (DDR) is the proportion of defects found during testing (before release) relative to total defects discovered in the sprint period (testing + post-release). It measures how effectively your QA activities surface bugs before they reach users.

Data sources needed

• Test runs / execution results (automated and manual) — defects raised during test execution
• Bug tracker — all defects logged with discovery date and origin (test vs production)
• Production incident logs (to count post-release defects)
• Optional: code commits / PRs tagged to defects (for root-cause classification)

Formula

DDR = (Defects found in testing during sprint) / (Defects found in testing during sprint + Defects found in production during sprint window)

Plain-English: percent of total discovered defects that were found before release.

Example
If testing uncovered 40 defects and 10 were later reported in production for that sprint window:
DDR = 40 / (40 + 10) = 0.8 = 80%

One limitation
DDR alone ignores defect severity and business impact. A high DDR can coexist with missed critical bugs in production; conversely many low-severity pre-release finds can inflate DDR. Use DDR alongside severity-weighted metrics and escape rate to inform release decisions.

Direct answer (QA perspective)
As a QA Engineer I’d compute these metrics consistently from the incident timeline fields you have: incident start, detection timestamp, remediation start, and resolution timestamp. I’ll state clear definitions, give formulas, then show a short 5-incident numeric example.

Definitions

• MTTD (Mean Time To Detect): average time from incident start to detection.
• MTTR (Mean Time To Recover): average time from remediation start to resolution (how long remediation took to recover service).
• Mean Time To Fix: average time from detection to resolution (how long it took to implement the fix after detecting).

Formulas

MTTD = (1/N) * sum_i (detection_i - incident_start_i)

MTTR = (1/N) * sum_i (resolution_i - remediation_start_i)

Mean Time To Fix = (1/N) * sum_i (resolution_i - detection_i)

Example (five incidents, durations in minutes)
Assume we compute per-incident intervals:

• Incident 1: detection lag 10, remediation->resolution 30, detection->resolution 40
• Incident 2: detection lag 5, remediation->resolution 20, detection->resolution 25
• Incident 3: detection lag 20, remediation->resolution 45, detection->resolution 60
• Incident 4: detection lag 15, remediation->resolution 10, detection->resolution 25
• Incident 5: detection lag 8, remediation->resolution 12, detection->resolution 20

Compute averages:

• MTTD = (10+5+20+15+8)/5 = 11.6 minutes
• MTTR = (30+20+45+10+12)/5 = 23.4 minutes
• Mean Time To Fix = (40+25+60+25+20)/5 = 34.0 minutes

Notes / QA best practices

• Use consistent clock/timezones and store ISO timestamps.
• Exclude non-production or duplicate incidents, and document outliers.
• Track distributions (median, p95) in addition to means to avoid skew from outliers.

Overview (90-day objective)
Establish a defensible baseline of product quality so we can track improvement, detect regressions, and prioritize engineering effort.

Week 1–2: Discovery & tooling

• Stakeholders: Engineering leads, Product Manager, Customer Support, SRE, Data Analyst.
• Inventory telemetry, test suites, CI, bug tracker. Agree definitions (e.g., what counts as P0).

Week 3–6: First metrics to collect (priority & justification)

1. Defect escape rate (bugs found in production / total bugs) — high priority: shows real user impact and release quality.
2. Mean Time to Detect (MTTD) & Mean Time to Resolve (MTTR) — ops/response measurables for incident lifecycle.
3. Automated test coverage (functional + end-to-end %) — measures regression safety and automation investment.
4. Flaky test rate (failed then passed without code change) — protects CI trustworthiness.
5. Customer-reported incidents / NPS-impacting bugs per week — ties QA to user experience and product metrics.

Week 6–10: Baseline & validation

• Pull historical logs where available; for missing data, run 4 weeks of focused collection and retroactively tag recent releases.
• Use CI runs, issue tracker (labels), production logs, and support tickets. Document collection queries and dashboards (Grafana/Looker).

Week 11–12: Reporting, SLAs & next steps

• Share dashboards with stakeholders, set initial targets (e.g., reduce escape rate by 30% in next quarter), and establish cadence: weekly QA triage, monthly quality review.
• Plan improvements: add targeted automation for high-risk flows, flakiness reduction sprints, and alerting for MTTD.

This plan focuses first on production impact and CI health so QA efforts align with user experience and engineering priorities.