Diagnosing Production Infrastructure and Reliability Challenges Questions

Diagnosing production infrastructure and reliability problems and proposing a fix with trade-off analysis, rollout strategy, and monitoring or validation metrics. Covers spotting issues such as scaling limits, deployment complexity, observability gaps, technical debt, and single points of failure, whether working from internal telemetry and on-call context (dashboards, logs, traces, incident timelines, postmortems) or from external public signals about a company's systems (engineering blog posts, job postings, GitHub repos, status pages) for research or case-study style assessments. Includes proposing remediation, articulating trade-offs, defining SLIs and SLOs, planning a safe rollout, and specifying how you would validate the fix once it ships.

EasyTechnical

27 practiced

A public code search indicates a company uses synchronous RPC calls between many services with few timeouts configured. Describe the operational risks of this design and present three concrete changes (with pros/cons) you would implement to improve resilience without a major rewrite.

Sample Answer

Operational risks- Cascading failures: synchronous RPCs with long/no timeouts allow one slow service to block callers, which propagates across call chains and can exhaust threads/connections.- Resource exhaustion: hanging connections consume CPU, threads, sockets, connection-pool slots, and memory, causing system-wide degradation.- Increased latency and reduced availability: user requests wait on multiple slow dependencies, raising P99 latency and error rates.- Hard-to-debug outages: lack of timeouts/visibility makes failure modes obscure and recovery slow.

Three concrete changes (no major rewrite)

1) Enforce short, sensible client-side timeouts per call- What: Add per-RPC timeout configuration (e.g., 500ms–2s depending on RPC criticality) and fail fast.- Pros: Prevents blocking resources, reduces blast radius, enables graceful degradation. Easy to roll out incrementally per service.- Cons: Requires tuning to avoid false positives; may surface more retries/errors initially.

2) Add circuit breakers with metrics and backoff- What: Wrap RPCs in circuit breakers (open on error rate/latency thresholds) with exponential backoff and health checks.- Pros: Stops repeated failing calls from overwhelming services, gives downstream time to recover, provides observability signals.- Cons: Adds stateful client logic; needs careful threshold selection; can cause temporary increased error rates when tripped.

3) Introduce bulkheads and connection-pool limits + priority for critical paths- What: Partition resources (thread pools, connection pools) per downstream dependency or request type and cap concurrent calls.- Pros: Prevents one noisy dependency from starving others, improves isolation and predictability.- Cons: Requires config changes and monitoring; wrong sizing can underutilize resources.

Implementation notes- Roll out gradually: instrument current RPCs for latency/error, deploy on a few services, monitor metrics (error rate, retries, circuit open time, pool saturation).- Add dashboards and alerts for new signals, and document timeout/circuit policies.

MediumSystem Design

21 practiced

A company publishes a status page showing frequent short-lived spikes in tail latency across a multi-service checkout flow. Propose a 60-day remediation plan focusing on the service/application layer: include diagnosis steps, short-term mitigations, longer-term architectural changes, and how you'd measure success.

Sample Answer

Requirements & constraints:- Multi-service checkout, SLOs: e.g., 99th percentile latency < 300ms, spikes are short-lived and frequent.- No infra changes assumed (focus on service/application layer).- 60-day timeline, iterative measurable outcomes.

Day 0–7: Triage & observability improvements- Instrument end-to-end distributed tracing (e.g., OpenTelemetry): add span annotations for payment, inventory, auth, DB calls, external APIs.- Ensure high-cardinality tags: user region, checkout step, request size, feature flags.- Add per-service 1ms-resolution latency histograms and increase sampling for p99 windows during incidents.- Run a short load test to reproduce spikes.

Day 8–21: Diagnosis & root-cause hunting- Use traces to find offending spans and timing correlations (e.g., backlog in one service, GC pauses, DB timeouts, retries).- Correlate spikes with deployment, traffic patterns, feature flags, cache misses, circuit-breaker activity.- Identify top contributors to p99 (e.g., external API latency, synchronous remote calls, blocking IO, lock contention).

Day 22–35: Short-term mitigations (quick wins)- Add request-level timeouts and retries with exponential backoff and jitter; make retries idempotent where needed.- Implement bulkheads and bulkhead threading/executor separation to isolate critical checkout paths from noisy neighbors.- Cache results aggressively for non-critical lookups; increase cache TTL and add circuit-breakers for slow external dependencies.- Deploy graceful degradation: skip non-essential steps (recommendations, analytics) during high latency.- Tune JVM GC or thread pools identified as hotspots.

Day 36–60: Longer-term architectural changes- Replace synchronous remote calls with async patterns or event-driven handoffs for non-blocking stages (e.g., confirm order async with eventual consistency).- Introduce hedged requests for high-variance external calls with rate limits to prevent overload.- Refactor hot paths to reduce tail sources (e.g., avoid global locks, partition state to reduce contention).- Move critical read paths to local caches or CQRS read models; introduce batching for DB writes where possible.- Add SLO-driven alerting and automated runbooks for p99 breaches.

How to measure success- Primary metrics: p50/p95/p99 latency for end-to-end checkout, error rate, checkout success rate.- Business metrics: conversion rate, abandoned carts during incidents.- Operational: number and duration of p99 spikes per week, mean time to mitigate (MTTM).- Targets by day 30: reduce frequency of p99 spikes by 50%; by day 60: achieve SLO (e.g., 99th < 300ms) or demonstrate clear path with architectural changes and reduced business impact.

Communication & governance- Daily 15-min incident triage first two weeks, then 3x/week.- Share dashboards and runbook updates with product and SRE; schedule postmortem on any spike > threshold.- Deploy changes behind feature flags for safe rollout and A/B measurement.

EasyTechnical

24 practiced

A team produces research results that stakeholders distrust due to lack of transparency. As an engineer partnering with researchers, propose practical reproducibility and transparency actions (tooling, pipelines, documentation) that can be implemented quickly to improve stakeholder trust without blocking research pace.

HardSystem Design

20 practiced

Describe a pragmatic monitoring and validation approach when performing a database schema migration that requires a multi-step deploy (expand-then-contract). Focus on application-layer changes, feature flags, query degeneration detection, and automated verification to ensure no user-facing errors during the migration window.

Sample Answer

Requirements:- Zero user-facing errors during migration window- Multi-step expand-then-contract deploy (backwards-compat)- Detect and prevent query degeneration (slow/incorrect queries)- Automated verification and safe rollback via feature flags- Low operational overhead, actionable alerts

High-level approach:1) Prepare DB and app changes for expand phase (add columns/tables/indexes) that are additive and backward-compatible.2) Deploy app code that can write both new and old shapes, guarded by progressive feature flags.3) Run verification (synthetic + shadow) to validate parity and performance.4) Switch reads to new schema, monitor for degeneration, then contract (remove old fields) once safe.

Core components and responsibilities:- Feature-flag system: per-endpoint, per-user gradual rollout (e.g., LaunchDarkly or homegrown). Flags control reads/writes to new schema paths and allow instant rollback.- Dual-write layer: application writes to both old and new schema during transition; read path optionally prefers new schema when flag enabled.- Shadow reads & canary traffic: send a copy of real requests to a canary service that uses new schema; compare responses (latency, correctness).- Verification service: automated checks (row-count parity, digest/hash of serialized responses, schema constraints, data migration consistency) run continuously.- Observability stack: metrics (RPS, p95/p99 latency per query type, DB query plans, CPU/IO), distributed traces, logs, and SLO-based alerts.- Query degeneration detector: runs anomaly detection on query latency and changes in execution plan (EXPLAIN/EXPLAIN ANALYZE diffs); flags queries with increased cost or plan changes.- Rollback orchestrator: ties feature flags + deployment rollback, can auto-disable flags on alert thresholds.

Data flow:- Client -> API (feature-flag check) -> if flag ON: API reads from new schema (and optionally old for verification) and writes to both. If OFF: use old schema.- Canary path receives duplicated requests using new code/schema; verification compares results asynchronously.

Automated verifications:- Functional parity: per-endpoint hashed response comparison with tolerances for non-determinism.- Referential/data integrity: row-counts, key statistics, uniqueness checks, nullability assertions.- Performance parity: compare p50/p95/p99 latencies and CPU/IO per query. If latency delta > threshold or plan cost increases > X%, mark as degeneration.- Plan regression: capture and diff execution plans; detect index misses or full-table scans.- Alerting rules: immediate rollback if: - >0.1% user errors (500/4xx) tied to migration flag - latency increase beyond SLO for canary cohort - query plan regressions for top-traffic queries

Operational practices:- Start with low-percentage canary rollout, expand by metric gates.- Run long-running background migration jobs with backpressure and resumability.- Maintain a migration runbook with rollback steps and data-fix tools.- Use blameless postmortems and metrics dashboards tailored to the migration (top N slow queries, failed parity checks).

Trade-offs:- Dual-write adds complexity and potential inconsistency; acceptable short-term for safety.- Shadow reads add load; restrict to sampled traffic to limit DB impact.- Automated rollback reduces human decision latency but needs conservative thresholds to avoid oscillation.

Why this works:- Feature flags decouple deploy from release, enabling safe, incremental exposure.- Automated parity + plan-diff detection catches silent data and performance regressions before users notice.- Canary + rollback orchestration minimizes blast radius while keeping operations manageable.

HardTechnical

29 practiced

Your team receives reports of increased error budgets consumption for an API during a marketing campaign surge. How would you triage whether the consumption is due to legitimate increased load, a bug, or malicious traffic? Describe the telemetry and controls you'd use to distinguish these causes and immediate mitigations for each.

Sample Answer

Situation: Error budget burn spikes for an API during a marketing surge.

Triage approach (fast, data-driven):1. Verify SLO context & timeline- Check SLO dashboards (error rate, availability, p99 latency) and correlate with campaign start time.

2. Telemetry to inspect (what I query immediately)- Traffic volume: requests/sec, new unique users, referrers, campaign UTM tags (Prometheus/Grafana).- Error breakdown: 4xx vs 5xx, per-endpoint error rate, top error messages (ELK/Cloud Logging).- Client metadata: IPs, ASN, geo, user-agent, device, API key / token, auth failures.- Traces: sample traces for failing requests (Jaeger/NewRelic) to see dependency failures and latencies.- Resource metrics: CPU, memory, connection queues, DB RPS, circuit-breaker counts.- Network/SYN/TLS patterns and WAF logs for anomalies.- Rate of session creation and captcha/JS-challenge failures.

How to distinguish causes:- Legitimate load: proportional increase in successful requests and in unique legitimate user agents/UTMs; errors primarily timeouts/503s with backend saturation signals (CPU, DB queues); per-user error rate stays low.- Bug/deployment: sudden rise in a specific endpoint or error signature across many legitimate clients, coinciding with recent deploys/config changes; traces show application exceptions or bad responses before downstream calls.- Malicious traffic: high proportion of requests from few IPs/ASNs or abnormal UA strings, missing campaign referrers, high request rate per client, suspicious endpoints hit, scraping patterns, high 401/403 or malformed requests, WAF alerts.

Immediate mitigations (fast, reversible):- If legitimate surge: - Autoscale frontends/backends, increase capacity for DB or add read replicas. - Apply graceful degradation: raise timeouts, enable cached responses, serve stale data, reduce nonessential processing. - Apply per-user quotas to protect availability.- If bug: - Roll back the offending deploy or flip feature flag. - Put the failing endpoint into maintenance/disable noncritical routes. - Patch & redeploy with canary; add better validation/tests.- If malicious: - Activate WAF rules, block offending IP ranges/ASNs, throttle suspicious UAs. - Rate-limit per IP/API key and require stronger bot challenges (CAPTCHA / JS challenge). - Blackhole abusive traffic at CDN/edge (CloudFront/Cloudflare) and engage security team/IDS.

Controls to implement post-incident:- Per-API quotas and adaptive rate-limiting, provenance tagging of campaign traffic, improved dashboards (errors by campaign/referrer, per-client error-rate), deploy-time smoke tests, circuit breakers and bulkheads for dependencies, automated canary rollback, and integration with WAF/IDS for automated mitigation.

This approach yields a fast diagnosis path and safe, role-appropriate mitigations for each root cause.

Unlock Full Question Bank

Get access to hundreds of Diagnosing Production Infrastructure and Reliability Challenges interview questions and detailed answers.

Join thousands of developers preparing for their dream job.