Consultative Discovery and Requirements Gathering Questions

To propose SLOs I’d ask targeted clarifying questions across consumers, impact, measurement, and governance:

Consumers & usage

• Who are the consumers (teams, automation, dashboards, alerts)? Which are latency-sensitive vs batch?
• Is any consumer revenue- or safety-critical?
• Are read patterns uniform or spiky? Any hot endpoints or filters?

Business criticality & impact

• What business outcomes depend on this API (SLAs, alerts, customer-facing dashboards)?
• What is acceptable user impact during partial degradation?
• What is the cost of false negatives/positives from delayed metrics?

SLIs / SLO targets

• Which SLIs do you want: availability (5xx), error rate (4xx+5xx), latency? Any correctness/integrity SLIs?
• Target percentiles: do consumers need p50/p90/p95/p99/p999? For dashboards p95/p99; for alerts/pipelines p99/p999.
• Suggested initial targets: e.g., Availability 99.9% (monthly) for non-critical internal, 99.99% if used for alerting/CI.

Error budget & policy

• What cadence for error budget burn reviews? Who owns mitigation vs feature freeze?
• What mitigation actions when budget burns (postmortem, rollbacks, throttling)?
• Allowed maintenance windows and planned downtime policy?

Monitoring, instrumentation & retention

• What metrics are emitted (request count, latencies histogram, success/failure, throttled)?
• Logging level, traces required for debugging, and correlation IDs?
• Retention needs for metrics/traces (30d/90d/1y) for capacity planning and RCA?

Alerting & dashboards

• Which symptoms should trigger paging vs Slack alerts? Thresholds and runbooks?
• Who must be paged (oncall team, API owner, downstream owner)?

Operational constraints & signoffs

• Who are stakeholders for SLO signoff (API owner, consuming teams, product/ops leads)?
• Who approves error budget policy and maintenance windows?
• Are there regulatory, compliance, or data residency constraints?

Data volume & scaling

• Is 50k/day steady or projected growth? Expected QPS peaks and retention of historical queries for replay?

This information lets us pick concrete SLOs (e.g., p95 latency <200ms, p99 <500ms; availability 99.9%), define error-budget actions, monitoring schema, retention, and a stakeholder signoff matrix.

Situation: I joined a team that handed me a one-line ticket: "Make our system reliable." The service had frequent user-facing errors, noisy alerts, and no SLOs; stakeholders expected improvement but gave no specifics.

Task: Define a concrete plan to improve reliability in 8 weeks and deliver measurable gains without disrupting feature delivery.

Action:

• Clarifying questions I asked:
  • What user journeys/endpoints matter most? (Which SLAs exist?)
  • Current availability, incident cadence, and MTTR baselines?
  • Who are the stakeholders and what risk is acceptable?
  • What tooling and telemetry are available (metrics, logs, tracing)?
  • Are there upcoming releases or planned changes?
• Assumptions I stated when info was missing:
  • Assume top 3 endpoints cover 80% of user impact.
  • Assume existing monitoring captures errors but lacks latency SLO.
  • Assume team can allocate 20% sprint capacity to reliability work.
• Discovery → acceptance criteria / experiment:
  • Ran a 2-week discovery: mapped call paths, identified top failure modes, and measured baseline: 99.2% availability, mean MTTR 72 minutes, error budget burn of 35%/week.
  • Converted findings into concrete SLOs (99.9% availability for checkout, 95th latency <300ms), alert thresholds, and three experiments:
    1. Implement circuit breakers and retries for dependency failures (canary rollout).
    2. Add targeted dashboards + SLO-based alerts to reduce noise.
    3. Automate a rollback and runbook-trigger for the top two incident types.
  • Acceptance criteria: SLOs defined and onboarded into alerting; canary shows <1% degradation; MTTR <30 minutes within 4 weeks.
• Execution: I implemented the circuit breaker library, added SLO-driven alerts, wrote runbooks, and shipped a canary to 10% traffic then ramped.

Result:

• Within 6 weeks availability for checkout improved from 99.2% to 99.94%.
• MTTR dropped from 72 to 22 minutes.
• Weekly error budget burn fell from 35% to 5%.
• Alert noise reduced by 60% (fewer paging incidents); engineers reclaimed ~10% weekly capacity.
  This experience reinforced that clear questions, explicit assumptions, and short experiments with measurable acceptance criteria turn vague mandates into reliable outcomes.

Situation: Our managed DB vendor reported intermittent degraded performance for one week. Their SLA allowed occasional slowdowns, but our user-facing latency and batch-job backlogs were rising, and product owners were escalating.

Task: As the SRE lead, I needed to (1) quantify the business impact, (2) collect authoritative evidence, (3) negotiate remediation with the vendor, and (4) scope and cost a contingency (failover or migration) with timelines so stakeholders could decide.

Action:

• Clarify business impact
  • Convened a rapid sync with product, support, and finance to map user-facing symptoms to business metrics: API P95 latency increased from 220ms→480ms, error budget burn rate jumped to 60%/week, nightly ETL windows extended by 45 minutes causing a 12% delay in reporting revenue.
  • Classified impact tiers: critical (checkout, writes), degraded (analytics), acceptable (non-peak reads).
• Gather evidence
  • Correlated our telemetry (Prometheus, APM traces) with vendor metrics and logs. Captured time-synced traces showing DB query durations spiking from median 20ms→180ms for key transactions.
  • Exported sample slow queries, connection pool stats, and a 72-hour histogram of latency + SLO breach incidents. Packaged as an incident dossier with timestamps, request IDs, and business impact summaries.
• Negotiate remediation
  • Presented dossier in a formal SLA exception call. Requested immediate mitigation actions: priority escalation, temporary resource bump (I/O/CPU), and a root-cause timeline within 24–48 hours.
  • Proposed concrete remediation KPIs: reduce P95 DB latency to <250ms within 48h, and commit to compensatory credits if missed for each day beyond SLA tolerance.
  • Built a fallback timeline: if no material improvement in 48h, we would execute contingency plan.
• Scope contingency (failover/migration)
  • Identified two options:
    1. Provider-managed failover (cross-AZ or read-replica promotion): estimated 2–8 hours cutover, operational cost +10–15% monthly, minimal data movement risk. Runbook drafted, tested in staging (30–60m rehearsal).
    2. Full migration to alternative managed DB (cloud vendor or self-managed cluster): estimated 3–6 weeks engineering work, ~$40k one-time migration engineering + ~20% ongoing ops delta (monitoring, runbooks). Risk: schema incompatibilities, downtime window (planned 30–120 min for final sync with CDC).
  • Produced cost-benefit matrix: business loss per hour of degraded service (~$12k/hr), remediation probability, and risk score. Recommended immediate attempt with vendor escalation + simultaneous parallel prep on Option 1 (promote replica) so failover can execute within 6 hours if needed; start Option 2 planning only if vendor cannot commit to long-term reliability.
  • Secured stakeholder sign-off for up to 48 hours vendor remediation; preapproved budget for emergency failover and a migration feasibility spike (1 week, ~$8k) to finalize migration plan if needed.

Result: The vendor agreed to priority remediation and temporary resource scaling; P95 latency fell to acceptable levels in 36 hours. Because we had the contingency runbook and a tested failover rehearsal, we reduced decision time and had clear cost/timeline trade-offs for leadership. The incident produced an updated SLO, stricter vendor KPIs, and a standing contingency fund and tested failover procedure.

Situation: We needed a company-wide SLO/error-budget policy across many services with different owners, SLAs, and maturity levels. My goal was to protect customers while preserving team autonomy.

Clarifying questions I asked:

• Who are our primary customer journeys and their availability/performance expectations?
• Which services are customer-facing vs internal, and what are their business priorities (revenue, compliance, experimentation)?
• What current telemetry exists (latency, error rates, availability) and how reliable is it?
• What timescales matter (per-minute, hourly, daily, weekly)?
• What are SLO consumers (support, legal, execs) and required reporting cadence?
• What exceptions are allowed (maintenance windows, planned feature flags, degraded modes)?

Negotiation and design steps I took:

1. Group services into tiers (critical, important, best-effort) based on customer impact and business priority — negotiate tier placement with owners using data and business metrics.
2. Propose measurable SLO templates per tier (e.g., critical: 99.99% availability monthly + p95 latency), with clear signal definitions (what metric, label filters, cardinality limits).
3. Define error budget policy: how burn is measured, alert thresholds (soft warning at 25% burn, hard at 50%), and automated vs human actions.
4. Create exception and onboarding process: teams can request temporary SLO adjustments with a justification, rollback plan, and stakeholder sign-off.
5. Enforcement mechanisms that preserve autonomy:
   • Require standardized SLO telemetry and dashboards but allow teams to implement internal mitigations.
   • Automate guardrails (rate limiters, circuit breakers, CI gates) for hard budget breaches; soft breaches trigger collaboration (runbook, monthly review).
6. Operational processes:
   • Monthly error-budget reviews with SRE, product, and service owners.
   • Playbooks for soft/hard burns, including rollback and feature holds.
   • Post-incident reviews tied to SLO breaches and improvement tickets.
7. Pilot and iterate: run policy on a few representative services, collect feedback, and adjust templates, thresholds, and tooling.

Result / rationale: This approach balances customer protection with team autonomy by using data-driven tiers, standard telemetry, clear escalation paths, and a lightweight exception process. It enforces critical protections with automation while keeping day-to-day remediation and design ownership with service teams.

Situation: Service X shows a rising 5XX error rate; the owner denies code changes caused it. Below are the clarifying and investigative questions and the acceptance criteria I would use to identify cross-service causes, determine ownership, and enable tracked remediation.

Clarifying questions (scope & symptoms)

• When exactly did the increase start? (timestamp, deploys, config changes)
• Is the error rate steady, spiky, or correlated with traffic patterns?
• Which endpoints and HTTP status codes are increasing? (500 vs 502 vs 503 vs 504)
• Are errors client-visible or internal service-to-service errors?
• Do logs/traces show consistent stack traces or varying failures?
• Any recent infra changes (load balancer, API gateway, DNS, proxy, TLS certs, infra autoscaling, service mesh, rate-limit rules)?

Investigative questions (cross-service / dependency focus)

• What downstream services or databases does Service X call for the failing requests? Are their latencies or error rates increasing?
• Do traces show timeouts or errors originating after calling dependency Y? Attach example traces.
• Have we changed contractual interface (schema, headers, auth tokens) between services recently?
• Are there any network-level errors (connection resets, DNS failures, 5xx from upstream proxies)?
• Is there a correlated spike in resource metrics (CPU, memory, file descriptors, connection pool exhaustion) on Service X or its dependencies?
• What changed in the client side (load generator, traffic routing, CDN) around the incident window?

Ownership and boundary questions

• Which teams own each component in the call chain (ingress, gateway, service X, dependency A/B, DB, infra)?
• Which team owns retries/timeouts/circuit-breaker behavior for each hop?
• What SLAs/SLOs exist for each component and where does the error budget sit?
• Who is the canonical owner of the client-facing error surface vs internal propagation behavior?

Data to collect and share

• Top N failing request traces, sample logs, request payloads, and timestamps
• Metric dashboards: 5xx rate, p50/p95 latency, CPU/memory, thread pools, connection stats for service and dependencies
• Recent deploy/change list (CI/CD, infra-as-code), config diffs, feature flags
• Network/infra events (deployments, scaling, network maintenance)

Immediate investigative actions I’d propose

• Pinpoint a reproducible failing request and run distributed trace to find first non-OK response
• Temporarily increase logging (correlate trace IDs) for affected endpoints
• Roll back or shadow-traffic suspect deploys, throttle traffic to isolate
• Run health checks against dependencies directly to validate behavior

Acceptance criteria for remediation (clear, testable, and assignable)

• Root cause identified and documented: "Failure originates in component Y due to Z" with evidence (trace, log, metric).
• Ownership assignment: named owner/team for the root cause and for any follow-up fixes across components.
• Fix implemented and code/config change reviewed, or compensating mitigation applied (timeout/retry config, circuit-breaker, capacity increase).
• Verification tests: synthetic tests and production canaries show 5xx rate returned to baseline (e.g., 5xx < X% over 30 min and p95 latency < threshold).
• Post-deploy monitoring: automated alert for recurrence and dashboard updated with baseline and SLO impact.
• Postmortem with follow-ups: action items assigned, timelines, and closing criteria.

Example acceptance statement:

• "After patch/deploy, 5xx rate for /api/payments stabilizes to <0.1% for 60 minutes under normal traffic, traces show no calls returning 5xx from dependency A, and the owning team has a rollback plan and playbook documented."

This set of questions and criteria lets you triage whether the failure is in Service X or elsewhere, assign clear ownership, and create measurable acceptance criteria so remediation work can be tracked to completion.