Diagnosing Production Infrastructure and Reliability Challenges Questions

Diagnosing production infrastructure and reliability problems and proposing a fix with trade-off analysis, rollout strategy, and monitoring or validation metrics. Covers spotting issues such as scaling limits, deployment complexity, observability gaps, technical debt, and single points of failure, whether working from internal telemetry and on-call context (dashboards, logs, traces, incident timelines, postmortems) or from external public signals about a company's systems (engineering blog posts, job postings, GitHub repos, status pages) for research or case-study style assessments. Includes proposing remediation, articulating trade-offs, defining SLIs and SLOs, planning a safe rollout, and specifying how you would validate the fix once it ships.

HardSystem Design

42 practiced

Design a phased rollout plan and validation strategy for moving from client-side polling to server-sent events (SSE) or websockets for near-real-time notifications. Address scalability, fallbacks for proxy limitations, monitoring, and how you'd measure user-perceived latency before/after.

Sample Answer

Requirements & constraints:- Near-real-time notifications (<500ms desirable), support millions of users, mobile & web clients, corporate proxies/NGINX that may block long-lived connections, graceful fallbacks, measurable user-perceived latency, minimal client changes.

Phased rollout plan:1. Pilot (1–5% of users): instrument new SSE/WebSocket service behind feature-flag. Route traffic via a dedicated gateway cluster (LB → connection manager → backend event fanout). Use opt-in users and internal power users.2. Canary (5–25%): expand by region and client type. Enable server-side connection throttling, per-tenant limits, and sticky routing. Collect detailed telemetry.3. Gradual ramp (25–100%): progressively shift segments, monitor resource usage, tune autoscaling rules (connections, CPU, memory).4. Cutover & cleanup: deprecate polling endpoints once SLA met. Keep fallback for downgrade window.

Architecture & scalability:- Use a connection manager (Kubernetes + horizontal autoscaling) that terminates SSE/WebSocket and forwards events via a fast pub/sub (Kafka or Redis Streams) to fanout workers.- Shard connections by tenant/region; use consistent hashing to preserve affinity.- Use per-connection heartbeats and windowed backpressure to prevent overload.- Scale: stateless connection managers + state in pub/sub; autoscale on active connections and event throughput.

Fallbacks for proxy limitations:- Detect failed long-lived connections via handshake/short timeout. Client attempts: 1. SSE/WebSocket 2. HTTP Chunked SSE via XHR 3. Long-polling (with exponential backoff)- Provide TLS ALPN and WebSocket-over-HTTPS to bypass proxy blocking where possible.- Server-side keepalive ping intervals configurable to satisfy corporate proxies; allow polling endpoint as a permanent fallback for strict environments.

Validation & monitoring:- Metrics: active connections, new connections/sec, events/sec, queue depth, error rates, reconnection rate, CPU/memory per pod.- SLOs: connection success rate >99%, event delivery within 500ms p95.- Distributed tracing: add trace IDs to events to measure end-to-end latency.- Synthetic testing: bots simulate clients across networks (mobile, corporate proxy) exercising all fallbacks.- Log anomalies: spike in reconnects or fallback usage triggers alerts.

Measuring user-perceived latency:- Instrument client to record timestamps: event-generation, server-send (if possible), client-receive, and UI-render. Send aggregated telemetry (histograms) to analytics.- Compute metrics: median/p95/p99 client-received latency, time-to-interaction (when notification visible), and percent of events delivered via real-time vs fallback.- A/B test: run side-by-side cohorts (polling vs SSE/Ws) measuring CPU/network usage, battery impact (mobile), and engagement metrics (click-through). Compare before/after baselines.- Success criteria: p95 latency improved by target (e.g., from 2s → <500ms), fallbacks <2% of users, no regressions in server cost per 10k users.

Trade-offs & notes:- WebSockets give bidirectional capability; SSE simpler for unidirectional streams and reconnection semantics.- Pub/sub choice affects durability and replay—Kafka for high durability, Redis Streams for low-latency fanout.- Operational complexity rises with long-lived connections; justify by UX improvements and engagement metrics in pilot.

This plan balances incremental risk, robust fallbacks for restrictive networks, measurable validation, and scalable architecture suitable for enterprise customers.

EasyTechnical

38 practiced

You're asked to perform a lightweight architecture assessment of a target company using only public information (blog posts, engineering job listings, GitHub, service status pages). Describe the checklist you would use to identify likely infrastructure bottlenecks such as scaling limits, observability gaps, or single points of failure. Give examples of concrete signals you'd look for in public artifacts.

Sample Answer

Start by clarifying scope: public-only inputs, non-invasive, timeboxed (e.g., 4–8 hours). Then run this lightweight checklist grouping signals into capacity, reliability, observability, and operational practices.

Checklist- Deployment & infra surface - Look for cloud provider mentions (AWS/GCP/Azure), Kubernetes, serverless in blogs/JD/GitHub. - Signals: “EKS/Fargate”, Terraform repos, IaC templates. If only single-region references or single AZ, suspect regional SPOF.- Scaling patterns - Check job listings for SRE/ops headcount, autoscaling experience, or manual scaling phrases. - Signals: blog posts about “manual vertical scaling”, GitHub repos with fixed-size instance types or no autoscaling configs.- Availability & resilience - Read status pages, incident reports, postmortems. - Signals: frequent incidents tied to a single service, “database failover took X hours”, single master DB references.- Observability & telemetry - Search for monitoring stacks (Prometheus, DataDog, NewRelic), logging (ELK), tracing (Jaeger, X-Ray). - Signals: lack of mention of tracing or SLIs/SLAs in engineering blogs, few observability tools in repos, sparse postmortems with vague metrics.- Operational maturity - Inspect job ads for CI/CD, chaos testing, runbooks, on-call rotation; check GitHub Actions/CircleCI configs. - Signals: “manual deploys”, absence of pipeline configs, or only basic unit-test steps.- Security & dependencies - Look for dependency update policies, vulnerability disclosures, or use of managed services. - Signals: large number of pinned outdated libs in repos, lack of security engineer roles.- Data & stateful services - Identify DBs, caches, queues from code or docs. - Signals: single queue consumer, hardcoded connection strings, no replication config visible.

Concrete examples- Job listing: “manage EC2 fleet, manual scaling” → likely scaling limits.- Status page: repeated “DB primary failover” incidents → single-master DB SPOF.- GitHub: terraform with provider.region hardcoded → limited region resilience.- Blog post: “we disabled tracing to reduce overhead” → observability gap.

Prioritize findings by business impact and recommend quick wins: add autoscaling, multi-AZ/region deployments, introduce tracing/SLIs, and implement IaC review. Provide evidence links and recommended next investigative steps (ask for architecture diagrams, metrics, runbooks).

MediumTechnical

26 practiced

A scientific research team complains about data quality issues (missing fields, inconsistent labels) in the telemetry you provide for a study. Propose a pipeline-level approach to improve data quality, including schema enforcement, validation hooks, backfill strategies, and processes to reduce future drift, while considering privacy constraints.

Sample Answer

Situation: The research team is receiving telemetry with missing fields and inconsistent labels, undermining study validity and slowing analysis.

Proposed pipeline-level solution (high level):1. Schema enforcement- Adopt a canonical, versioned schema (e.g., Avro/Protobuf/JSON Schema) stored in a schema registry.- Enforce schema at ingress: producers must serialize against the schema; any non-conforming messages are rejected or routed to a quarantine topic.- Use semantic types and enums for labels to avoid free-text drift.

2. Validation hooks- Implement lightweight validation service as part of the ingestion pipeline (Kafka Connect/Beam/Flink) that performs: - Structural validation (required fields, types) - Semantic checks (value ranges, controlled vocabularies) - Consistency checks (cross-field invariants)- Emit structured validation events (error type, record id, producer id) to a monitoring dashboard and metrics (Prometheus/Grafana).

3. Backfill & remediation- Identify bad ranges/partitions via validation events.- Create backfill jobs that: - Read raw/quarantined data, apply deterministic transformations + imputation rules, then write corrected records to canonical topic or data lake. - Maintain metadata: original record pointer, transformations applied, confidence score.- For unrecoverable records, mark as unusable; produce summary reports for stakeholders.

4. Reduce future drift- CI/CD for schema changes: require automated compatibility checks (backward/forward) and staged rollout with canary producers.- Producer SDKs and linting: provide client libraries that validate before send; include telemetry for producers (error rates).- Governance: change control board for schema updates, documented deprecation timelines, and SLAs for producers.

5. Privacy & compliance- Apply privacy-preserving transformations at ingress (Pseudonymization, hashing, tokenization) with key management separate from analytics environment.- Implement field-level access controls and encryption at rest/in-flight; keep PI in a separate secured store and only provide researchers with de-identified datasets.- Audit logs for all validation/backfill operations; include consent/PI metadata when required.

Trade-offs & implementation notes- Rejecting bad data improves downstream quality but requires producer remediation; quarantine + alerting balances immediacy and developer experience.- Real-time validation adds latency; batch validation can complement for heavy checks.- Start with high-impact required fields and expand validation iteratively.

Outcome: A versioned, observed, and governed ingestion pipeline that prevents most bad data at source, provides deterministic backfill for past issues, and enforces privacy while giving researchers transparent provenance and quality metrics.

MediumTechnical

25 practiced

Design a lightweight synthetic monitoring suite to detect regressions in critical user journeys (login, checkout, search) across global regions. Define test frequency, assertion types (latency, correctness), alert thresholds, and how to detect and avoid false positives due to transient external dependencies.

Sample Answer

Situation: Design a lightweight global synthetic monitoring suite for critical journeys (login, checkout, search).

Solution overview:- Architecture: small fleet of distributed synthetic runners (managed cloud VMs or serverless functions) in N global regions (e.g., us-east, eu-west, ap-south, ap-northeast). Central orchestration service schedules tests, collects metrics, and feeds an alerting/incident engine and dashboard. Tag each run with region, runner-id, dependency-version.

Test frequency (balanced for cost vs sensitivity):- Login and Search: every 1–3 minutes per region.- Checkout (higher business impact): every 30–60 seconds per region.- Full end-to-end checkout (including payment gateway): every 5–10 minutes (to limit external load).- Lightweight smoke (auth-only) every 30s for SLA-sensitive flows.

Assertion types:- Correctness: - HTTP status codes and end-state assertions (e.g., login -> user dashboard DOM element present, search returns >=1 result with expected schema, checkout results in order-confirmation ID). - JSON schema validation for API responses.- Latency: - Step-level timings (DNS, TCP, TLS, TTFB, render, API calls) and overall end-to-end time.- Side-effects: - For checkout, validate simulated order recorded in test environment or use no-op payment tokens; avoid creating real commerce artifacts.- Availability: - Synthetic-run success rate over sliding windows.

Alert thresholds and escalation:- Immediate alert if any single run fails correctness in >=2 distinct runners within same region within a 60s window.- Latency alerts: - Step-level: warn at 95th percentile > 2x baseline; critical at >3x baseline or > absolute thresholds (e.g., checkout E2E > 5s warn, >10s critical). - Use rolling baselines per region (7-day moving median + MAD) to adapt to diurnal patterns.- Global degradation: if >30% of regions show concurrent failure/latency spikes for >5 minutes, escalate to on-call and open incident.

False-positive mitigation (transient external dependencies):- Retry logic with exponential backoff: treat a single transient 1-of-3 failure as flaky; require 2-of-3 consecutive failures within short window to mark as real.- Cross-region correlation: if only one region fails, check region-specific infra (CDN, ISP outages). Only escalate cross-region correlated failures.- Dependency health tags: annotate runs with third-party dependency checks (e.g., payment gateway status API, CDN probe). Suppress alerts if dependency outage is confirmed and non-actionable.- Canary vs baseline separation: maintain canary runners against newly-deployed hosts; use control-run against stable baseline to distinguish deploy-induced regressions vs global external issues.- Noise suppression: integrate with anomaly detection (e.g., simple z-score or Holt-Winters) to avoid alerting on expected spikes; implement maintenance windows and blackout periods automatically from deployment pipeline.- Post-alert automated verification: when alert triggers, run deeper diagnostic suite (traces, HAR, logs) and only notify humans if diagnostics confirm service-side issue.

Operational notes:- Store detailed traces and HARs for each failing run; retain metadata for root-cause analysis.- Keep suite lightweight: focus on critical assertions + sampled deep diagnostics to control cost.- Provide SLAs for false-positive rate (e.g., <5% of alerts are noise) and iterate thresholds after initial tuning.

This design yields sensitive, region-aware detection while minimizing noise from transient external dependencies through retries, correlation, dependency tagging, and adaptive baselining.

HardTechnical

21 practiced

Your production incident shows duplicate events processed by multiple downstream services due to asynchronous retries. Design a comprehensive fix that covers idempotency, deduplication windows, message identifiers, and contract changes. Include a phased deployment plan that minimizes user impact.

Sample Answer

Requirements & constraints:- Stop duplicate processing across downstream services caused by async retries.- Preserve at-least-once delivery semantics while enabling safe reprocessing.- Minimal user impact, backwards compatibility during rollout, observable behavior.

High-level design:1. Canonical message ID: producers assign a globally unique, stable message_id (UUID v4 + namespace or composite of source+logical-id+timestamp). Message_id travels in envelope/header of every event.2. Contract change (versioned): introduce Envelope V2 containing message_id, created_at, idempotency_key (optional), and cause metadata. Keep Envelope V1 readable; consumers ignore unknown fields until migrated.3. Consumer-side idempotency store: each downstream service maintains a small write-optimized dedupe store (Redis with persistence or DynamoDB with TTL) keyed by message_id -> processed_metadata (status, result checksum, expiry).4. Deduplication window: configurable TTL per message (default 24h for business events, shorter for high-volume telemetry). TTL governs how long duplicates are suppressed or compared.5. Processing semantics: - On receive: atomically check-and-set message_id in dedupe store (SETNX semantics). If absent, insert and proceed. If present and marked completed, skip or return cached result. If present and in-progress, use optimistic locking / idempotent retry strategy (e.g., dedupe entry status: IN_PROGRESS -> prevent parallel processing).6. Idempotent handlers: make operations idempotent where possible (use upsert, idempotency keys for downstream writes, transactions with unique constraints).7. Observability & compensation: emit metrics (duplicate_count, dedupe_store_miss/hit), tracing with message_id, and compensating job to reconcile missed/skipped events.

Trade-offs:- Short TTL reduces storage but may allow late duplicates; long TTL increases storage and state.- Centralized dedupe service simplifies management but adds latency; embedded per-service store reduces coupling.

Phased deployment plan (minimize user impact):Phase 0 — Prep (low risk, 1–2 weeks)- Define contract V2 schema, client libraries to set message_id.- Provision dedupe stores per service; instrument tracing/metrics.- Add feature flags gating strict dedupe checking.

Phase 1 — Producer rollout (2–4 weeks)- Roll producers to emit message_id in envelope while still supporting V1. Validate message_id coverage and uniqueness via sampling.- Backward compatible: downstreams ignore new fields.

Phase 2 — Passive consumer adoption (2–4 weeks)- Update consumers to populate dedupe store in passive mode: record message_id but still process everything. Collect duplication stats and validate dedupe TTL sizing.

Phase 3 — Active dedupe behind feature flag (2–4 weeks)- Flip feature flag for a subset of traffic (canary): consumers enforce SETNX and skip already-completed message_ids. Monitor duplicate_metric, error rates, latency.- Use canary per-region or per-customer cohort.

Phase 4 — Gradual global enablement (1–2 weeks)- Ramp enforcement across all consumers; continue monitoring.- If issues, rollback by toggling feature flag (consumers still accept V2).

Phase 5 — Cleanup & hardening (2–4 weeks)- Remove V1-only code paths when all producers/consumers on V2.- Tune TTLs, retention, and optimize dedupe store cost.- Add automated reconciliation jobs for entries in IN_PROGRESS too long.

Operational runbook:- Quick rollback via feature flag to fall back to previous behavior.- Alert thresholds for duplicate surge, dedupe store errors, and processing latency.- Regular audits of message_id uniqueness and contract conformance.

This design ensures safe de-duplication using message identifiers and per-service idempotency, supports phased, low-risk rollout with feature flags and observability, and balances storage/latency trade-offs with configurable dedupe windows.

Unlock Full Question Bank

Get access to hundreds of Diagnosing Production Infrastructure and Reliability Challenges interview questions and detailed answers.

Join thousands of developers preparing for their dream job.