Problem Solving and Scenario Analysis Questions

Candidates are expected to demonstrate a systematic, structured approach to analyzing and resolving complex scenarios relevant to their field. This includes clarifying the problem statement, eliciting requirements, constraints, and assumptions, and identifying missing information or ambiguous areas. Candidates should decompose complex problems into logical components, prioritize tasks or evidence, generate multiple solution options, and perform trade-off evaluation that balances impact, feasibility, cost, and risk. Core skills assessed include root cause analysis, structured diagnosis of an incident or issue, and reasoning through realistic scenarios drawn from the candidate's own domain (for example, a technical migration, a process breakdown, a customer escalation, a resourcing conflict, or a policy decision). Candidates should define how they would validate a proposed solution (test cases, acceptance criteria, or success metrics), describe how they would monitor or verify the outcome after implementation, and identify opportunities for improvement, risk mitigation, or automation where applicable. Clear communication of the recommended approach, the expected outcomes, and the rationale behind trade-offs made is essential.

HardTechnical

48 practiced

Design a distributed locking mechanism suitable for coordinating schema migrations across a fleet of deployers. Compare using Redis SETNX with TTL, database advisory locks, and consensus-based approaches (e.g., etcd), focusing on correctness under leader failover and clock skew.

Sample Answer

Requirements:- Single owner lock for schema migration across deployers- Safety: at most one active migrator- Liveness: progress despite failures- Tolerate leader failover and clock skew- Observable ownership and safe handoff

Summary recommendation:Use a consensus-backed lock (etcd/consul/raft) as primary. If unavailable, fall back to DB advisory locks for best-effort; avoid plain Redis SETNX as the only safety mechanism unless extended.

Comparison:

1) Redis SETNX + TTL- How it works: SETNX key with owner id and TTL; renew by refreshing TTL.- Pros: simple, low latency, widely available.- Cons: unsafe under clock skew & network partitions. If owner is paused (GC, long pause) and TTL expires, another node may acquire lock while original still running -> split-brain migrations. Renewals race with expiry; requires careful leader liveness detection (e.g., fencing tokens).- Correctness: not strictly safe under failover unless you implement a fencing mechanism (monotonic token/version) and the resource (migration runner) checks token before applying non-idempotent steps.

2) Database advisory locks- How it works: DB-native lock (e.g., Postgres pg_try_advisory_lock). Lock tied to DB connection/session.- Pros: Stronger safety: lock releases on connection drop or crash; less reliant on clocks. Good if migrations already target that DB and DB is highly available.- Cons: If migration touches multiple DBs or schema owners, centralized DB may be single point of failure. Requires careful handling during DB failover; connection-based semantics mean failover can drop locks unexpectedly or leave unclear ownership during replication switchover.- Correctness: Safer than Redis TTL for failover and clock skew because OS/DB manage session liveness. Still depends on DB HA behavior.

3) Consensus-based (etcd/raft/consul)- How it works: Use leader election or a lock implementation backed by consensus; lock persists as long as holder maintains session lease.- Pros: Strong correctness guarantees: linearizability, safety under leader failover, and leases are managed by cluster quorum so accidental dual ownership is prevented. Leases are monotonic and less sensitive to local clock skew because server-side lease expiry uses cluster time.- Cons: More operational complexity. Higher write latency. Must ensure client properly renews lease; handle network partitions gracefully (if minority cannot renew, lock lost).- Correctness: Best choice for strict safety and liveness trade-offs. Provides fencing tokens (revision numbers) so migrations can check ownership.

Design details / best practices:- Use consensus lock as primary. Store a fencing token (monotonic revision) with lock; include token in migration run and check in any external steps (idempotency + token check).- Make migrations idempotent and resumable; record migration state and owning token in a durable store.- Implement backoff retry and clear observability (who holds lock, lease TTL, metrics, alerts).- For environments where etcd isn't available, prefer DB advisory locks over Redis TTL. If using Redis, add compare-and-set semantics and a server-side verification step (e.g., put owner+revision in a migration table and require verification before destructive changes).- Test failover scenarios: client pause, network partition, DB failover, clock skew. Include chaos tests.

Final verdict: consensus-backed locks (etcd) + fencing tokens + idempotent migrations = most robust. Database advisory locks are acceptable if migrations are DB-local and DB HA is trusted. Redis SETNX+TTL is least safe unless augmented with strong fencing and careful engineering.

EasyTechnical

46 practiced

List logging best practices for production services with respect to structure, levels, sensitive data handling, and correlation. Provide an example structured log line (JSON) for an API request including fields that help with debugging latency and errors.

Sample Answer

Structure your logs and use levels consistently- Use structured logs (JSON) for machine parsing, with a stable schema (timestamp, level, service, env, msg, ctx).- Log at appropriate levels: DEBUG for dev/traces, INFO for high-level events, WARN for expected but concerning issues, ERROR for failures, FATAL for unrecoverable states.Sensitive data handling- Never log raw PII/credentials. Mask/omit fields (e.g., user_email -> user_email_masked).- Use tokenization/encryption for searchable identifiers if required by compliance.- Record decisions for redaction in code and docs; add a redaction flag in logs when data is omitted.Correlation and observability- Emit request_id/correlation_id and trace_id on all logs to join traces, metrics, and logs across services.- Include span identifiers and parent ids when available.- Ensure logs include service, instance, region, and deployment/tag for fast filtering.

Other best practices- Include structured latency fields: total_ms, db_ms, downstream_ms.- Avoid high-cardinality free-form fields; put large payloads in object storage and log a reference.- Rate-limit noisy logs and use sampling for DEBUG.- Ensure clocks are synced (NTP) and timestamps are ISO8601 UTC.

Example structured log for an API request:

json

{
  "timestamp": "2025-12-06T15:42:31.123Z",
  "level": "INFO",
  "service": "orders-api",
  "env": "prod",
  "instance_id": "orders-api-54c9f7d8",
  "deployment": "v1.17.3",
  "request_id": "req-8f3a2b6c",
  "trace_id": "trace-4a7e9d2b",
  "span_id": "span-01",
  "user_id_hash": "sha256:3f5e...c9",
  "user_email_masked": "j***@example.com",
  "method": "POST",
  "path": "/v1/orders",
  "status": 500,
  "error": "payment_gateway_timeout",
  "latency_ms": {
    "total": 1240,
    "handler": 1200,
    "db": 30,
    "downstream_payment_ms": 1000
  },
  "response_size_bytes": 512,
  "redaction": ["payment_card_number"],
  "sampled": true,
  "msg": "order creation failed due to downstream timeout"
}

HardSystem Design

33 practiced

A DNS provider outage is causing failures to route traffic to your service. Design a DNS failover and multi-provider strategy that minimizes failover time and avoids DNS caching issues, including TTL choices, health checks, and fallback routing techniques.

Sample Answer

Requirements:- Failover time < 60s for DNS provider outage; preserve client reachability globally.- Minimize impact from DNS caching and TTLs; avoid single-provider dependency.- Automated health checks and failover; ability to roll back.- Support geo-routing and consistency with SLOs.

High-level design:- Use two (or more) independent authoritative DNS providers (primary and secondary) with synchronized zone data via automation (GitOps + signed zone files / API sync). Use providers that support DNS Failover APIs and health-check integrations.- Front DNS with short TTLs at multiple layers and use anycast routing for endpoints.

Key components and configuration:1. TTL strategy- Set TTL = 30s for failover-sensitive A/AAAA and ALIAS/ANAME records. Use slightly higher (60–300s) for less critical records.- Use provider-side low TTLs; also publish SOA minimum/refresh tuned to providers.

2. Health checks- Implement active multi-region health probes (SRE-owned) that check HTTP(S)/TCP and synthetic transactions from multiple POPs. Feed results to both providers via authenticated APIs.- Providers should have independent health-checks too; combine via quorum (require N out of M probes).

3. Failover logic & routing- On probe failure, update DNS to remove unhealthy endpoints and add healthy endpoints (weighted/priority records). Use provider APIs for instant changes.- Use provider-side features: failover records, weighted policies, and GeoDNS to route to nearest healthy region.- Maintain an async orchestration service that pushes updates to all providers atomically and records versioned changes. If primary provider API is down, orchestration pushes to secondary.

4. Cache-busting and minimizing TTL effects- Use short TTLs to reduce caching window, but accept some residual cache. To reduce user-visible downtime: - Combine DNS changes with connection-level redirection: keep application endpoints reachable via fronting CDN/anycast IPs and use health-based routing at L4 where possible (e.g., BGP/anycast). - Use CNAME/ALIAS for service.example.com pointing to provider-managed failover targets; when switching, change the final target. - Implement Keep-alive: serve HTTP 200 from degraded region with 302 to healthy region if session can be resumed.

5. BGP/Anycast and secondary fallback- For critical low-latency failover, run anycasted ingress (CDN or your anycast fleets) advertised via multiple transit providers; DNS directs to anycast IPs, so DNS failure doesn't block reachability.- As last resort, pre-publish a provider-independent emergency NS entry (low TTL) hosted across cloud providers and in IaC for fast activation.

Trade-offs and notes:- Short TTLs increase DNS query load—use caching resolvers and rate-limit updates; ensure providers can handle update/QPS.- Residual caching unavoidable; complement DNS with network-level solutions (CDN/anycast) to minimize impact.- Test via DR drills, chaos engineering, and runbooks to ensure orchestration works and rollbacks are safe.

Monitoring & runbooks:- Alert on probe divergence between SRE probes and provider health.- Automate failover but require manual approval for wide-scope changes; keep playbooks for provider API failures, emergency NS switch, and rollback steps.

This approach balances rapid DNS-based failover with network-level resiliency to minimize user impact from DNS provider outages.

EasyTechnical

45 practiced

Define idempotency in the context of APIs and SRE. Give two concrete examples of how idempotency can prevent operational problems during retries, deployment rollbacks, or network partitions. Explain how you would test idempotency in an automated fashion.

Sample Answer

Idempotency (APIs/SRE): an operation is idempotent if performing it multiple times has the same effect as performing it once. For APIs and SRE this means retries, duplicate deliveries, or repeated client calls should not produce duplicate side-effects (e.g., double charging, duplicated DB rows), even when requests are retried due to timeouts, network partitions, or during rollbacks.

Example 1 — Payment/Charge API:- Problem prevented: If a client retries a "charge" after a network timeout, they shouldn’t be billed twice.- Implementation: accept a client-generated Idempotency-Key; persist a mapping (Idempotency-Key → outcome/result, timestamp). On duplicate key, return the stored response without re-executing payment logic. This avoids duplicate charges during retries or when the payment gateway times out.

Example 2 — Resource creation in distributed DB:- Problem prevented: During deployment rollback or duplicate messages, creating the same resource twice can corrupt state.- Implementation: design endpoints as upserts or use deduplication via a unique business key + a conditional write (INSERT ... ON CONFLICT DO NOTHING or compare-and-set). Combined with idempotency keys you guarantee one logical create.

Automated testing approach:1. Unit/integration tests: use deterministic idempotency keys and run the identical request N times; assert only one side-effect (one DB row, one external charge call). Verify returned responses match the first successful one.2. Property/contract tests: fuzz varying ordering, delays, and duplicate delivery semantics to ensure invariants hold.3. End-to-end CI tests: simulate network failures and retries (wiremock or test doubles for external services) and assert metrics (e.g., external-calls count) remain 1 per logical operation.4. Chaos/blackbox: inject network partitions or kill services while retrying requests; validate state convergence and idempotency window expiry behavior.5. Monitoring assertions: in tests assert idempotency cache eviction, TTLs, and that repeated keys after expiry are handled correctly.

Key considerations: define TTL for stored outcomes, ensure idempotency storage is highly-available and performant, and log/metric every dedup event for observability.

HardTechnical

45 practiced

You must prioritize SLOs and error budgets across dependent teams where one team owning service X consumes many other internal services. Propose a cross-team governance model for SLO ownership, error budget transfer/borrowing, and decision-making during outages, emphasizing incentives and operational safety.

Sample Answer

Situation: At a previous company our customer-facing service X depended on five internal services. Each team had their own SLOs but no clear governance; when X’s customer SLOs degraded, teams argued over blame and refused to throttle features, consuming X’s error budget.

Task: As the SRE lead, I needed a cross-team governance model that balanced ownership, enabled safe error-budget transfers/borrowing, and produced clear outage decisions without disincentivizing teams.

Action:- Clarified requirements: customer-facing SLOs for X, downstream SLOs, traffic/cost constraints, and acceptable escalation latencies.- Established roles: Service Owner (each team), Consumer Owner (X’s product owner), and SLO Steward (SRE-owned neutral arbiter).- Created an SLO contract: for each dependency, define SLI, SLO, error-budget cadence, and a dependency impact matrix (how downstream degradation maps to customer-visible error rate).- Error-budget policy: - Borrowing: short-term borrowing allowed for up to 20% of downstream daily error budget with automatic time-limited guardrails (auto-throttle if borrowed budget > threshold). Borrowed budget must be repaid within a week via reduced non-essential traffic or feature flags. - Transfer: permanent transfers require written approval by both Service Owner and Consumer Owner, SRE Steward sign-off, and a mitigation plan.- Outage decision flow: - Automated detection maps SLIs to impact matrix and alerts the SRE Steward and both Owners. - If customer SLO risk > critical threshold, SRE Steward invokes protective actions (traffic shaping at X, circuit-breakers, feature toggles), prioritizing customer experience. - Post-incident, run a blameless postmortem with quantitative root-cause, repayment plan, and SLA contract updates.- Incentives & safety: - Tie part of team OKRs/quarterly metrics to collaborative reliability goals (error-budget health for shared dependencies). - Maintain a shared reliability dashboard and weekly “budget health” sync. - Provide runbook templates, automated tooling for borrowing/repayment, and a small stability fund (on-call support hours) to help teams remediate quickly.

Result: This model removed ambiguity, reduced cross-team friction, lowered time-to-protect during incidents, and cut customer-impacting incidents by ~35% in the first two quarters. Teams reported clearer incentives to fix root causes instead of shifting load, and repayment policies ensured long-term operational safety.

This approach balances clear ownership, automated protection, measurable incentives, and governance that keeps customer SLOs paramount while enabling pragmatic short-term flexibility.

Unlock Full Question Bank

Get access to hundreds of Problem Solving and Scenario Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.