Problem Solving Behaviors and Decision Making Questions

Covers the interpersonal and cognitive traits that shape how a candidate solves problems, including initiative, ownership, proactivity, resilience, creativity, continuous learning, and evaluating trade offs. Interviewers probe when a candidate takes initiative versus seeks help, how they balance speed versus quality, how they persist through setbacks, how they generate creative alternatives, and how they learn from outcomes. This topic assesses mindset, judgment, and the ability to make principled decisions under uncertainty.

MediumTechnical

89 practiced

You're planning a migration with many unknowns. Which decision-making frameworks and rollout patterns would you apply to reduce risk (e.g., canaries, feature flags, experiment-driven migrations)? Describe a phased plan that surfaces unknowns safely and shows how you'll gate progress.

Sample Answer

Approach: I use hypothesis-driven decision making + risk-prioritization (impact x likelihood) to choose safe rollout patterns: canaries, progressive ramp with feature flags, experiment-driven validation, and fast kill-switch/rollback automation. I pair this with SLO/error-budget gating so progress is data-driven.

Phased plan (gates & actions):

1) Discovery & risk mapping (0–2 weeks)- Inventory dependencies, list unknowns, score risks.- Define SLOs and key metrics: p95 latency, error rate, request success, CPU/memory, DB replica lag.- Create runbooks & rollback playbooks.

Gate: All critical dependencies identified; monitoring & alerts instrumented.

2) Pre-production validation (2–4 weeks)- Run integration tests, load tests, chaos tests in staging.- Use shadow/dual-writes for data migrations; verify reads still come from primary until validated.

Gate: No regressions in load/chaos tests; automated rollback verified.

3) Canary (week 1 of rollout)- Route 0.5–2% traffic to canary under feature flag; run for N hours/day.- Monitor SLOs, business metrics, logs, traces, DB metrics, and user errors.- Automated rollback if any metric breaches threshold for >X minutes.

Gate: Stable for 48–72 hours and within error-budget.

4) Progressive ramp (days–weeks)- Increase traffic in steps: 5%, 25%, 50% then 100% with observation windows and post-step signoff.- For DB migrations use phased approaches: read replica promotion, online schema changes (pt-online-schema-change or native ALTER with minimal locking), and cutover during low traffic.

Gate: Each step requires metrics within acceptable bounds and consumed error-budget allowance.

5) Pilot tenants & experiment-driven checks- Run A/B experiments comparing old vs new behavior for business-critical flows; validate business metrics (conversion, throughput).- Use feature flags per tenant for quick rollback.

Gate: Business metrics parity or improvement; stakeholders sign-off.

6) Full cutover & post-mortem- Gradually deprecate old paths, keep monitoring and retention for rollback window.- Conduct blameless postmortem; capture learnings and update runbooks.

Key controls:- Automated health-checks + rollback hooks in CI/CD.- Observability: dashboards, SLO alerts, synthetic tests.- Error-budget policy: if breached, pause rollout.- Communication: status channel, on-call rotation during rollout windows.

This combination surfaces unknowns early, uses small blasts of traffic to validate assumptions, and gates progress on quantitative SLO-driven criteria so risk is minimized while enabling rapid rollback.

HardTechnical

96 practiced

A recurring incident reveals systemic deployment problems across many teams. As SRE lead, design a plan to change deployment practices organization-wide: include training, CI/CD changes, phased rollout, metrics to track, and retro measures to ensure changes stick and don't regress.

Sample Answer

Situation: Multiple teams were causing recurring incidents during deploys — rollbacks, configuration drift, and flaky migrations — eroding availability and developer confidence.

Task: As SRE lead I needed to stop the incidents by changing deployment practices org-wide while minimizing disruption and ensuring sustained adoption.

Action:- Diagnosis & Stakeholder Alignment (Weeks 0–2) - Convened a cross-functional working group (SRE, platform, engineering managers, QA, security). - Collected deployment telemetry and postmortems to identify root causes (manual steps, no automated rollback, missing smoke tests, env drift).- Define Target State & Guardrails (Weeks 2–4) - Standards: automated CI checks, canary/blue-green or progressive rollouts, automated rollback, infra-as-code, feature flags, runbook per service, required smoke tests. - Policy: deployment checklist enforced by pipeline gates; exceptions logged and approved.- CI/CD Changes (Weeks 4–12, platform team) - Implement pipeline templates with required stages: static analysis, unit tests, integration tests (test infra), deployment strategy selection (canary/blue-green), automated health checks, automated rollback on SLA breach. - Add pre-merge pipeline for environment provisioning with IaC and ephemeral test environments. - Integrate feature-flag service and dark-launch capability. - Add observability hooks: tracing, metrics, and structured deploy metadata emitted on every deploy.- Training & Enablement (Weeks 6–14) - Role-based workshops: “Reliable Deploys 101” for developers, “Pipeline Authoring” for DevOps, runbook drills for on-call. - Hands-on labs: create a canary deployment, trigger auto-rollback, and restore from config drift scenarios. - Documentation: deployment playbook, checklist templates, FAQs, recorded sessions.- Phased Rollout (Weeks 12–24) - Pilot: 3-5 volunteer services of varying complexity — iterate templates and runbook. - Expand: onboard teams in waves (by risk/traffic), each wave includes 1:1 enablement and dry-run. - Enforcement: after 3 waves, move from advisory to required pipeline gates for all services.- Measurement & Ops Controls (ongoing) - Primary metrics: - Change Failure Rate (CFR): % deploys causing incidents — target reduce by 70% in 6 months. - Mean Time To Detect (MTTD) and Mean Time To Recover (MTTR) — target MTTR < 15 mins for regression. - Deployment Success Rate: % successful automated deploys. - Lead Time for Changes: time from commit to prod. - Number of emergency deploys and manual steps per deploy. - Dashboards and weekly reports to engineering leads.- Retro & Continuous Improvement - Blameless postmortems required for any deploy-caused incident; track action items in a central backlog. - Quarterly reliability review to update pipeline templates and training. - Audits: automated CI checks produce compliance reports; SRE performs monthly random deploy audits. - Incentives: recognition for teams that meet SLOs and adopt best practices; make migration support budget available. - Regression prevention: required pipeline tests stored as code in each repo; CI gate prevents bypass without a tracked temporary exception and executive approval.

Result:- Pilot reduced CFR by ~60% and MTTR by ~40% within first 8 weeks. Organization-wide rollout with measurable dashboards and enforced pipeline gates sustained improvements; incidents from deployment regressions dropped substantially and developer confidence in deploys increased.

Learned: combine platform automation, measurable SLIs, and focused human enablement. Technical guardrails plus continuous measurement and blameless learning are essential to make changes stick.

MediumTechnical

70 practiced

A production alert shows increased latency across regions but only one microservice reports high CPU. Logs are voluminous and incomplete. Describe a structured debugging approach to form and test hypotheses, narrow the search space, and prioritize immediate mitigations. Include how you would use sampling, tracing, and recent deploy/release information.

Sample Answer

Situation overview & immediate goals:- Objective: reduce user-visible latency quickly, gather data to identify root cause, avoid noisy work.- Constraints: voluminous/incomplete logs, single microservice shows high CPU, latency across regions.

1) Triage / quick data collection (first 5–15 min)- Confirm scope: which regions, services, endpoints show p95/p99 latency increase and since when.- Check SLO/error budget impact and user-facing severity.- Pull metrics: CPU, request rate, latency histograms, thread count, GC, I/O, network errors, DB/queue latencies per host/pod.- Look at recent deploys/releases in last 1–24 hours ( CI/CD, feature flags, config changes).

2) Form hypotheses (explicit short list)- Hypothesis A: New deploy (code/config) introduced CPU-heavy hot path.- Hypothesis B: Traffic pattern changed (burst, larger payloads) causing downstream contention.- Hypothesis C: Dependency outage (DB/cache) causing retries and CPU spin.- Hypothesis D: Platform issue in one service instance (hot thread, memory leak) amplified by traffic shaping.

3) Narrow search using sampling & tracing- Increase tracing sampling rate for the affected endpoints only (not global) to collect distributed traces for slow requests.- Use tail-based sampling: capture all traces that exceed latency threshold to ensure slow traces are retained.- Inspect traces for common spans: where time is spent, synchronous downstream calls, retries, queueing.- If full logs are noisy/incomplete: enable structured, rate-limited log sampling for slow requests (correlate by trace-id/request-id).- For the high-CPU service, run lightweight profiler/stack-sampler (flamegraph or pprof) on one or two representative pods to detect hot functions/GC.- Correlate spike times across regions to a shared dependency or release timestamp.

4) Test hypotheses safely- If traces show a new code path consuming CPU, reproduce on staging with similar traffic/profile to validate.- If dependency latency appears in traces, run targeted health checks or synthetic requests to that dependency.- If traffic surge suspected, examine ingress/load-balancer logs for payload sizes, user-agent, or client IP patterns.

5) Prioritize immediate mitigations (fast, reversible, lowest blast radius)- If recent deploy is suspect: revert/patch the deploy for affected service or disable feature flag.- If CPU hot threads found: restart affected pods one at a time (rolling) to return to healthy state while investigating.- If downstream is the bottleneck: enable circuit breaker / increase retries backoff, reduce concurrency, or route traffic away from problematic regions.- If traffic pattern causes overload: throttle nonessential endpoints, enable caching, or scale horizontally (autoscale immediate pods).- Apply rate-limiting at the edge or temporarily reduce sampling/tracing overhead if it contributes.

6) Communication & coordination- Post an incident channel, share hypotheses, actions taken, and next steps; include rollback plan and owner for each mitigation.- Declare impact and ETA for mitigation where possible.

7) Post-mortem & prevention- After stabilizing, run full RCA using traces + sampled logs + profiler output.- Add targeted dashboards (p95/p99, CPU hot functions, trace-derived service latency).- Improve logging: ensure required request IDs and errors are always logged, use tail-based sampling to avoid losing slow-request logs.- Add automated alerts for correlated signals (CPU+latency+recent deploy) and guardrails (preflight canary checks, feature-flag rollouts).

Why this approach:- Hypothesis-driven, data-focused, minimizes noisy full-log collection by targeted sampling/tracing.- Rapid mitigations prioritize user impact and reversibility, while profiling and traces deliver the causal signal needed for lasting fixes.

HardTechnical

66 practiced

You're leading response to a persistent, high-severity outage with no clear root cause and increasing executive pressure. Present a prioritized decision plan including containment options, communications cadence, resource allocation (internal and external), and thresholds for escalating to execs or bringing in outside vendors. Explain your trade-offs and rationale.

Sample Answer

Situation: We're in a persistent, high-severity outage affecting core customer flows, root cause unknown, user impact increasing, and execs demanding fast answers.

Prioritized decision plan

1) Immediate containment (first 0–30 minutes)- Goal: stop customer impact growth. Implement low-risk, reversible actions first: - Trigger traffic throttles / fail-open to degraded but functional paths (circuit breakers, rate limits). - Shift traffic to healthy regions / CDNs or enable a static degraded page where appropriate. - Disable non-essential downstream integrations that amplify load.Rationale/trade-off: These actions reduce blast radius quickly with minimal system changes; they may reduce functionality and revenue temporarily but buy time.

2) Stabilize & triage (30–120 minutes)- Form an incident command structure: Incident Lead (I/C), Engineering Lead, Communications Lead, SRE shifts, Product owner, and Ops liaison.- Create dedicated channels (chat, video) and a single shared timeline/incident doc.- Parallelize workstreams: (a) hot rollback/feature-flag rollback candidates; (b) forensic logging/trace enrichment; (c) metrics/alert tuning and hypothesis testing.Rationale: Parallel workstreams increase discovery speed while preserving operational control.

3) Resource allocation- Internal: Mobilize senior on-call + 2 senior SREs for core systems, 2 engineers for instrumentation/forensics, 1 release engineer, and product owner for business trade-offs.- External: Pre-approved vendor escalation if specific subsystems (DB, cloud infra, networking) show vendor fingerprints or telemetry indicates provider-side issues.Rationale: Keep expertise focused—senior staff on critical paths, juniors on support tasks. Limit too many cooks to avoid noise.

4) Communications cadence- Execs: 15-minute initial briefing with impact, containment steps, and expected next update time; 30-minute cadence thereafter until stabilized. Summarize decisions, customer impact, and blockers.- Engineering: Continuous updates in incident channel; 60-minute technical syncs for decision points.- Customers: External update within 30 minutes via status page + email, then every hour or on significant status change.Rationale: Frequent, predictable comms reduces pressure and builds trust; different audiences need tailored detail.

5) Escalation thresholds (when to involve execs/vendors)- Escalate to execs immediately if: customer-visible transactions drop >50% or regulatory/financial impact is immediate.- Bring vendor/support in when: telemetry or logs identify their component as probable cause OR after 2 hours of unsuccessful in-house root-cause progress despite parallel hypotheses.- Consider external incident response consultants if after 4 hours we lack progress, or if cross-organizational coordination stalls.Rationale: Thresholds balance urgency vs. cost/attention; vendors consumed early increase overhead but can shorten time-to-repair when evidence points to them.

6) Decision rules & trade-offs- Prefer reversible, low-risk mitigations first to avoid irreversible changes.- If time-to-repair estimate from investigation exceeds business tolerance, execute aggressive mitigations (full region failover, customer-facing rollbacks) even if imperfect.- Preserve evidence: avoid destructive fixes before forensic snapshots.

7) Post-incident plan- Immediate RCA owners assigned, timeline for communication, and SLO/error-budget reconciliation within 72 hours.

This plan prioritizes speed, clear ownership, minimal blast radius, and predictable communications—trading temporary functionality reductions for faster stabilization and clearer pathways to resolution.

MediumTechnical

83 practiced

Your service's 30-day availability SLO is 99.95%. Over the past week the service has averaged 99.7%, with error budget burn accelerating. Describe how you would analyze the situation, decide whether to pause feature releases or take other actions, who you would involve, and the steps you'd take to reduce burn and communicate status to stakeholders.

Sample Answer

Situation: Our 30-day SLO is 99.95% but last week averaged 99.7% and error budget burn is accelerating.

Approach / analysis:- Immediately quantify current state: calculate remaining error budget (in minutes), burn rate (minutes lost per day), projection (time to depletion). Correlate incidents with deployments, traffic spikes, configuration changes, or infra metrics (CPU, latencies, error types, dependency failures).- Triage: run the on-call/incident process. Identify whether failures are transient, degradations, or hard outages and whether a single root cause or multiple causes exist.

Decision criteria for pausing releases:- If projected depletion is within days and recent changes correlate with increased errors, pause non-essential feature releases and risky rollouts.- If burn is driven by external dependency or traffic surge unrelated to releases, keep releases paused for safety but focus on mitigations instead of broad rollback.

Who to involve:- Incident commander / on-call SRE- Product manager and release manager (to pause or gate releases)- Responsible Dev team(s) for failing service code- Platform/infra owners (network, DB, cloud)- SRE manager and stakeholders for business impact decisions and communications

Immediate actions to reduce burn:- Apply quick mitigations: enable circuit breakers, increase retries/backoff, scale up capacity, adjust rate limits, or roll back the suspicious change(s).- If a canary or feature flag exists, roll back or disable the flagged feature immediately.- Hotfix targeted root causes (e.g., fix misconfiguration, increase DB connections) with short test/rollout cycles.- Tighten alert thresholds and create dashboards for the 30-day window and rolling projection.

Longer-term actions:- Postmortem and root-cause analysis, create action items to prevent recurrence (tests, better canary metrics, runbooks).- Improve deployment safeguards: mandatory canary passes, automatic rollback on error-rate spike, automated throttling.- Revisit SLOs and error budget policy with product if burn was caused by legitimate feature-driven tradeoffs.

Communication:- Immediately notify stakeholders via dedicated channel (Slack + incident bridge), include current SLO %, remaining error budget (minutes and %), projection to depletion, suspected causes, and actions underway.- Provide regular concise updates (every 30–60 minutes while active, then daily) and a public dashboard link.- After resolution, share a blameless postmortem summarizing timeline, impact, root cause, fixes, and preventive steps.

This approach balances rapid mitigation, data-driven decisions about pausing releases, clear stakeholder involvement, and follow-through to reduce error-budget burn and restore SLO compliance.

Unlock Full Question Bank

Get access to hundreds of Problem Solving Behaviors and Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.