Decision Making Under Uncertainty Questions

Focuses on the frameworks, heuristics, and judgment used to make timely, defensible choices when information is incomplete, conflicting, or still evolving, in any domain. Covers diagnosing what is genuinely unknown before deciding, setting explicit decision criteria and thresholds, weighing probabilities against impact (expected value and cost benefit thinking), and defining upfront triggers for reversing course, escalating, or waiting for more evidence. Also covers calibrating risk tolerance to the stakes involved, choosing between a small test or pilot versus committing directly to a decision, communicating uncertainty and trade offs to stakeholders in plain terms, and how senior candidates fold organizational constraints (budget, time, politics, precedent) into a call when the fully right answer cannot be known in advance. The underlying judgment applies to any high-stakes decision made with partial information: a hiring call with an incomplete reference check, a budget reallocation with uncertain ROI, a legal or compliance risk judgment, a vendor or partner selection, a go/no-go on a product bet, or a technical rollout. No single domain should dominate the framing.

MediumTechnical

52 practiced

You must set an initial SLO for a new public REST API that has no historical telemetry. Describe a practical process to derive a defensible starting SLO, including what business and technical inputs you'd use, how to set the error budget, instrumentation priorities, and a plan to iterate the SLO as data arrives.

Sample Answer

Start by clarifying scope and stakeholder constraints- Who: product manager, business ops, support, devs.- What: define “successful API call” (HTTP 2xx? business-level success like processed transaction).- Constraints: regulatory SLAs, marketing promises, acceptable cost.

Derive a defensible initial SLO (practical process)1. Map business impact to user-facing intent: categorize endpoints by criticality (payment, auth = high; analytics = low).2. Use industry baselines and comparable services: public APIs typically target 99.9% for critical, 99.5% for important, 99% for best-effort endpoints as a starting point.3. Translate to measurable service-level indicators (SLIs): request success rate, p95 latency, and availability (up/down).4. Choose conservative starting points: e.g., success-rate SLO = 99.9% for critical endpoints; p95 latency SLO = 300ms.

Setting the error budget- Error budget = 1 - SLO (e.g., 0.1% monthly = ~43m/month of allowed failures).- Allocate budget by endpoint weight (traffic or business impact).- Define policy for burn: automated throttling/rollbacks at 50% burn, incident review at 75%, full freeze and remediation at 90%.

Instrumentation priorities (minimum viable observability)1. Request-level telemetry: timestamp, endpoint, status code, response time, user/client id.2. Business-level success/failure flag (did the API achieve the business outcome).3. Traffic dimensions: region, client version, auth tier.4. Error traces and structured logs with correlation IDs.5. Synthetic probes for baseline availability.Instrument first for SLIs; ensure sampling for traces but full counters for SLI metrics.

Plan to iterate as data arrives- First 30 days: collect telemetry, run synthetic checks, and compare incidents to budget.- Weekly review: validate SLI correctness, adjust instrumentation gaps.- 30–90 days: analyze distribution, customer impact, and cost. If observed reliability is much better than SLO, consider tightening; if worse, loosen only after business discussion and remediation plan.- Formalize cadence: monthly SLO review with stakeholders and post-incident learnings; lock changes behind a change control and communicate to customers.

Why this is defensible- Ties SLOs to business impact, uses industry priors, explicitly defines measurement, and builds a feedback loop so the SLO becomes data-driven quickly.

HardSystem Design

39 practiced

Design an automated rollback orchestration system that makes probabilistic rollback decisions based on incomplete telemetry. Include the high-level architecture (telemetry ingestion, decision engine, circuit-breaker, human-in-loop UI), the decision algorithm approach (e.g., thresholding, Bayesian reasoning), what SLIs must be present to make safe automated rollbacks, and known failure modes and mitigations.

Sample Answer

Requirements & constraints:- Safe automated rollback with incomplete/noisy telemetry, low false-positive rollback rate, quick mitigation when real regressions occur, human oversight and auditability, works across canary/gradual rollouts.

High-level architecture:- Telemetry ingestion: Instrumentation agents → metrics/traces/logs collectors (Prometheus, OTLP, Kafka). Add a stream processor (Flink/Beam) to normalize, aggregate, and compute short-window SLIs.- Feature store: time-series feature snapshots and baseline (historical, control cohort).- Decision engine: probabilistic module (Bayesian + sequential tests) that consumes SLI deltas, priors, and outputs rollback probability score.- Orchestrator & circuit-breaker: enforces rollback actions via deployment APIs (K8s, Spinnaker) with rate limits, cooldowns, and kill-switch.- Human-in-loop UI: dashboard showing probability, contributing signals, playbook suggestions, one-click manual rollback/approve, audit trail, simulation mode.- Observability & logging: immutable audit logs, alerting, runbooks, canary tracing.

Decision algorithm:- Use Bayesian sequential hypothesis testing (e.g., Sequential Probability Ratio Test with Bayesian priors). Model: H0 = no regression; H1 = regression causing SLI degradation > delta. Maintain posterior P(H1|data) updated as data arrives; compute expected cost of false rollback vs cost of delayed rollback (loss function). Trigger automated rollback when P(H1|data) > p_auto (e.g., 0.995) AND supporting conditions met across multiple SLIs and cohorts.- Complement with rule-based hard thresholds and minimum sample-size windows to avoid reacting to transient spikes.- Use ensembled evidence: weighted contributions from error-rate increase, latency shifts (p95/p99), saturation signals, user-business metrics. Weighting adapts via Bayesian updating or periodic re-training on labeled incidents.

Required SLIs (minimum set for safe decisions):- Request success/error rate (by endpoint and region)- Latency percentiles: p50, p95, p99- Saturation/infra signals: CPU, memory, thread-pool exhaustion, queue length- Traffic/throughput and peak shifts- Business KPI: e.g., checkout completion rate, retention-sensitive events- Canary vs control delta (relative comparison)- Health checks and dependency errors- Telemetry freshness/coverage indicator (to detect incomplete data)

Failure modes & mitigations:- Noisy telemetry / transient spikes → mitigation: require persistence for multiple windows, minimum sample size, smoothing, and ensemble confirmation across SLIs.- Telemetry lag or loss → mitigation: telemetry freshness SLI; if stale, disallow automated rollback; fall back to human review.- Correlated external incidents (e.g., downstream outage) causing false rollback → mitigation: dependency-aware filters; include dependency error signals so decisions attribute root cause; require matching dependency patterns.- Adversarial/buggy metrics or instrumentation regressions → mitigation: cross-validate with traces/logs; synthetic transactions; integrity checks; compare canary vs control.- Overly aggressive priors causing excessive rollbacks → mitigation: conservative priors, cost-weighting, adjustable policy per-service error budget, shadow testing of policies.- Orchestrator failures or thrashing rollbacks/roll-forwards → mitigation: circuit-breaker with exponential backoff, cooldown windows, max rollback rate, and human approval after N automated actions within timeframe.- State drift / config bugs causing repeated rollbacks → mitigation: root-cause automation to pin faulty change, isolate via feature flags, and block future deployments until fix.

Operational practices:- Tune p_auto per service using past incidents and error budgets; simulate policies on historical incident data (replay) before enabling.- Start with conservative automations: only for high-impact, easily-revertible changes (configs, feature flags); expand scope after proven safety.- Maintain runbooks and post-incident reviews to update priors/weights and improve models.- Provide “explainability” in UI: show which SLIs contributed and show counterfactual (what would have happened if not rolled back).

Trade-offs:- Bayesian sequential tests balance detection speed vs false positives; rule thresholds are simple but brittle. Ensemble + human-in-loop gives best pragmatic safety.- More automation reduces MTTD/MTTR but increases blast risk; use phased enablement and strict circuit-breaker policies.

This design prioritizes safety by requiring multi-SLI confirmation, telemetry freshness, conservative probabilistic thresholds, and mandatory human oversight for ambiguous cases while enabling fast automated mitigation for clear, high-confidence regressions.

EasyTechnical

43 practiced

Define SLI, SLO, SLA, and error budget in the context of a production service. You are on-call and notice the p95 request latency SLI is degrading but the service still has remaining error budget. Describe, in concrete steps, how you would use SLIs/SLOs/error budget to make a fast operational decision, what short-term actions you might take, and how you would communicate the situation to stakeholders.

Sample Answer

SLI (Service Level Indicator): a measurable metric representing user-perceived service quality (e.g., p95 request latency, error rate, availability).

SLO (Service Level Objective): the target for an SLI over a time window (e.g., p95 latency < 200ms 99% of the month). It defines acceptable reliability.

SLA (Service Level Agreement): a contractual promise to customers tied to penalties if SLOs are missed.

Error budget: 1 − SLO (the allowed amount of failure). It quantifies how much degradation you can tolerate before taking more aggressive remediation.

Concrete operational decision steps (fast, SRE-friendly):1. Confirm metrics: validate the p95 spike is real (dashboard, query raw metrics, check ingest delays, correlate with deployments, infra alerts).2. Check error budget: if budget remains, classify severity as degraded but within tolerance — lower urgency; if near/exhausted, escalate to incident.3. Triage root causes: look at recent deploys, autoscaling events, backend downstream latency, DB slow queries, traffic surge, GC, noisy neighbor.4. Short-term mitigations (pick minimal-impact, reversible actions): - Roll back recent deploy if correlated. - Increase capacity: scale up/scale out or add replicas. - Apply rate-limiting or shed non-critical traffic/features (degrade image processing, batch jobs). - Adjust timeouts/retry policies or enable circuit breakers to avoid cascading failures. - Clear unhealthy instances (restart) or adjust config (thread pools, connection limits). Always prefer actions with quick rollback and low risk.

Monitoring and follow-through:- Watch SLIs and related indicators for improvement within a short window (e.g., 5–15 mins).- If no improvement, escalate to full incident process and use error budget exhaustion as justification for interrupting dev work.

Communication (concise, concrete):- To internal stakeholders (on-call, engineering lead, SRE manager): “p95 latency rose from 120ms→320ms starting 09:12. Error budget remaining: 60% this month. I’ve validated metric ingestion and correlated spike with deployment abc at 09:10. Immediate action: scaled service +2 replicas and rolled back deploy. Monitoring for 10 minutes. ETA to confirm: 15 minutes. If latency not improving, will open Sev2 and notify product.”- To customers (if public-facing/status): short status update: “We are experiencing increased request latency for some users. No availability impact; degraded performance for a subset. Engineers are investigating and applying mitigations. Will provide updates every 30 minutes.”- After resolution: post-incident note with timeline, root cause, actions taken, whether error budget was consumed, and follow-ups (fixes, SLO tuning, alerts/runbook updates).

Key rationale: use SLO/error budget to set urgency—if budget remains, prefer reversible mitigations and monitoring; if budget is near/exhausted, treat as higher-priority incident and mobilize broader response. Always communicate status, impact, and next steps clearly and quantitatively.

HardTechnical

47 practiced

Leadership requests a one-time migration to multi-cloud to improve resilience. Costs increase and legal/data-sovereignty requirements are unclear. Construct a structured trade-off analysis: estimate expected outages avoided (use historical outage frequency), migration risk profile (probability of migration causing incidents), incremental running costs, compliance/legal constraints, and propose a phased roadmap with decision gates and rollback criteria.

Sample Answer

Framework: quantify benefits, risks, costs, constraints, then propose a phased roadmap with gates and rollback.

1) Expected outages avoided (benefit)- Input: historical production outage frequency = 4 incidents/year, avg outage duration = 2 hours, avg customer-impacting user-minutes lost = 120k, business cost = $5k/hour.- Assumption: single-cloud correlated failures cause 50% of incidents (2/yr); multi-cloud reduces correlated failure risk by 80% for those events.- Expected avoided incidents/year = 2 * 0.8 = 1.6 incidents.- Expected avoided downtime = 1.6 * 2 = 3.2 hours → avoided cost ≈ 3.2 * $5k = $16k/year. Operational benefit: improved SLO attainment and reduced risk to error budget.

2) Migration risk profile (one-time risk)- Categories & probabilities (estimate): - Planning/config errors causing incidents: 10% - Data inconsistency/replication lag: 5% - Cutover rollback causing downtime: 8% - Total chance of migration-caused incident = ~20% (severity distribution: 70% minor <30m, 25% moderate 30–180m, 5% major >3h)- Mitigations: dry-run, canary traffic, automated rollback, runbooks, chaos testing, DB replication verification, dedicated incident rota.

3) Incremental running costs (ongoing)- Current annual cloud spend: $1,200,000- Multi-cloud incremental: cross-region/network egress, duplicate standby capacity, management overhead ≈ 8–12% → $96k–$144k/year.- Additional tooling/observability and staff training: one-time $120k, recurring $30k/year.- Net first-year incremental ~ $216k–$264k; subsequent years ~$126k–$174k.

4) Compliance / legal constraints- Unknowns: data-sovereignty regs per region, contractual vendor clauses, encryption/keys residency.- Immediate actions: legal & compliance quick audit; map data types to sovereignty needs; flag datasets requiring single-region residency.- Constraint outcomes: if key data must remain in original region, design controls (localized storage + cross-cloud stateless services) or exclude those workloads from migration.

5) Trade-off summary- Benefit ($16k/year avoided outage) vs cost (~$150k+ incremental/year) shows pure outage-avoidance ROI is negative. BUT add qualitative benefits: reduced vendor lock-in, negotiating leverage, regulatory resilience, and long-tail catastrophic protection (low-frequency high-impact events) not fully captured in average cost.- If regulatory constraints force complex separation, cost rises and migration scope must shrink.

6) Phased roadmap with decision gates & rollbackPhase 0 — Assess (0–2 months)- Tasks: compliance audit, workload classification (stateful vs stateless), infra readiness, run cost model.- Gate A: Go/no-go if legal blockers unresolved OR >25% of critical data cannot be replicated.

Phase 1 — Pilot non-critical services (3–6 months)- Deploy secondary replicas in second cloud for 10% traffic (stateless APIs, batch jobs).- Implement CI/CD, observability across clouds, failover runbooks.- Gate B metrics (2-week observation): replication lag <100ms for test data, error rate delta <1.5x, rollback verified.- If fail: immediate rollback to single-cloud routing, postmortem, fix gaps.

Phase 2 — Canary critical subsystems (6–10 months)- Expand to read-replicas for databases, DR drills, cross-cloud load sharing at low traffic.- Gate C: Successful DR run <RTO target, SLOs met, cost delta within budget threshold.- If fail: freeze expansion, remediate, or cut back to pilot set.

Phase 3 — Controlled cutover / steady state (10–14 months)- Gradually shift traffic, run dual-write only where safe, enforce compliance controls.- Gate D: Executive decision based on cumulative metrics: legal sign-off, ops confidence, cost vs risk acceptance.- Rollback criteria at every step: automated traffic diversion scripts, DNS/TCP routing reversion, DB write-disable on secondary, rollback SLA: restore previous state within agreed RTO (e.g., 2 hours) or abort and switch to emergency single-cloud incident plan.

7) Recommendations- Do compliance audit first. If legal shows major constraints, consider hybrid approach (critical data single-cloud, stateless services multi-cloud).- Pilot small, measure real costs/risks, and include catastrophe scenarios in benefit model (tail-risk insurance value).- Budget one-time migration buffer and maintain strict decision gates tied to measurable reliability and compliance metrics.

This plan balances quantitative ROI, migration risk, ongoing cost, and legal uncertainty while giving clear gates and rollbacks to protect availability.

EasyTechnical

41 practiced

Explain what a canary deployment is and, in the context of an uncertain production environment, describe scenarios where you would prefer a canary over a blue-green deployment and vice versa. Highlight operational constraints (traffic routing, observability, rollout velocity) that influence the choice.

Sample Answer

A canary deployment gradually shifts a small portion of user traffic to a new version while the rest continues to run the stable version. This lets you validate real-world behavior (errors, latency, resource use) on production traffic and automatically or manually halt/rollback if problems appear. Blue‑green deploys by running two identical environments (blue=live, green=new) and switching all traffic atomically to green once it’s deemed healthy.

When to prefer canary (examples):- Uncertain runtime behavior under real traffic (e.g., subtle latency regressions, memory leaks): canaries reveal these early with minimal blast radius.- Complex stateful services where database migrations are incremental and risk needs staged exposure.- Limited capacity to run full duplicate environments (cost or infra constraints): canary uses current fleet and traffic steering.

When to prefer blue‑green (examples):- Need near-zero cutover downtime and you can provision a full parallel environment (e.g., batch processing or critical transactional service).- Release requires coordinated config changes where single-step switch reduces inconsistent states.- Regulatory or testing requirements demand full environment validation before serving any users.

Operational constraints that influence choice:- Traffic routing: canaries need fine-grained routing (weight-based LB, Istio, feature flags). Blue‑green requires DNS/consul/LB swap ability and session-affinity handling.- Observability: canaries demand strong metrics, tracing, and automated canary analysis to detect subtle regressions; without these, canaries are risky. Blue‑green relies more on pre-cutover smoke tests and end-to-end checks.- Rollout velocity: canaries are slower but safer (iterative ramps). Blue‑green enables fast rollback by switching back, but provisioning/validation time can slow overall cadence.

As an SRE I choose canary when I can route traffic and trust observability for automated gating; choose blue‑green when I need atomic cutover and can afford duplicate infra and comprehensive pre-switch validation.

Unlock Full Question Bank

Get access to hundreds of Decision Making Under Uncertainty interview questions and detailed answers.

Join thousands of developers preparing for their dream job.