Quality Metrics and Test Reporting Questions

Maintainability Scorecard for Microservices (for SRE onboarding reviews)

Dimensions (each scored 0–4):

1. Ownership & On-call (0: none, 4: named owners + runbook + on-call rotation)
2. CI/CD & Deployment Automation (0: manual, 4: fully automated with safe rollbacks)
3. Test Coverage & Quality (0: none, 4: unit+integration+e2e + flaky-test metrics)
4. Observability (0: none, 4: structured logs, traces, metrics, dashboards)
5. Alerting & SLOs (0: none, 4: defined SLOs + actionable alerts + error budget)
6. Documentation (0: none, 4: architecture, runbooks, setup, troubleshooting)
7. Security & Compliance (0: none, 4: scanned deps, secrets management, least privilege)
8. Scalability & Resilience (0: no resilience, 4: autoscaling, circuit breakers, chaos-tested)
9. Config & Secret Management (0: in-code configs, 4: centralized, versioned, secure)
10. Dependency & Build Hygiene (0: outdated/unclear deps, 4: pinned, reproducible builds)

Scoring rubric:

• 0: Absent / high risk
• 1: Minimal / manual work, known gaps
• 2: Basic coverage, several gaps
• 3: Good, minor polish needed
• 4: Exemplary / production-ready

Weights: Assign higher weight to Observability, SLOs, Ownership, CI/CD (e.g., 1.5x). Compute weighted percent.

Thresholds:

• Green: >=80% — approve onboarding
• Yellow: 60–79% — onboarding with remediation plan
• Red: <60% — blocked until critical issues fixed

Action plan for <80% (especially Red):

1. Classify findings into Critical (production risk), High, Medium, Low within 48h.
2. Immediate fixes (48–72h): restore alerts, assign owners, revert risky changes.
3. 30-day remediation sprint: prioritized tickets, owner-assigned, weekly checkpoints with SRE.
4. Provide templates: runbook, dashboard, CI pipeline example, test harness.
5. Follow-up review at 30 days; escalate to engineering leadership if unresolved.
6. Automate future checks: integrate scorecard into PR gates/CI and periodic audits.

This scorecard balances risk, operability, and developer velocity; use it as a living checklist tailored to service criticality.

Approach: aggregate test runs over the last 30 days in a single pass, compute counts, failures, failure rate and average duration, then filter with HAVING for failure_rate > 5% and avg_duration > 2s. Use Postgres boolean-to-int cast for an efficient conditional sum.

SELECT
  test_name,
  COUNT(*)                     AS runs,
  SUM((status = 'fail')::int)  AS failures,
  SUM((status = 'fail')::int)::double precision / NULLIF(COUNT(*),0) AS failure_rate,
  AVG(duration_seconds)        AS avg_duration
FROM test_runs
WHERE started_at >= now() - INTERVAL '30 days'
GROUP BY test_name
HAVING
  (SUM((status = 'fail')::int)::double precision / NULLIF(COUNT(*),0)) > 0.05
  AND AVG(duration_seconds) > 2;

Key points / reasoning:

• The WHERE clause restricts to the 30-day window so aggregation only scans recent rows.
• SUM((status='fail')::int) is a single-pass, efficient conditional count (avoids multiple scans).
• NULLIF(COUNT(*),0) prevents division-by-zero; HAVING ensures runs>0 implicitly.
• HAVING applies the failure_rate and avg_duration filters after grouping (required for aggregates).

Performance tips:

• Add an index on (started_at) or a composite index on (started_at, test_name) if queries are frequent.
• If cardinality is high, consider partial indexes for recent data or a materialized view refreshed periodically for dashboards/alerts.