Covers the ability to diagnose, triage, and resolve complex technical problems end to end while demonstrating personal ownership. Candidates should show deep technical reasoning about system architecture, integration complexity, data migration considerations, and custom configuration trade offs. Expect discussion of root cause analysis, diagnostic techniques, reproducible debugging, and risk mitigation strategies. Candidates should be able to explain design trade offs, propose practical solutions, assess business impact, and describe collaboration with stakeholders and cross functional teams. Emphasis should be placed on concrete actions the candidate took, how they prioritized options, and the measurable results and lessons learned.
HardSystem Design
29 practiced
System design (hard): Design a resilient alerting delivery system that supports deduplication/grouping, rate-limited delivery to on-call engineers, mute windows, escalation policies, and multiple notification channels (SMS, email, Slack). Include how to handle alert storms, integration failures (e.g., SMS provider outage), and how to keep the system operational during org-wide incidents.
Sample Answer
Requirements:- Functional: ingest alerts from services, dedupe/group, apply policies (rate-limit, mute windows, escalation), send via SMS/email/Slack, support retries/fallbacks.- Non-functional: high availability, low latency for critical alerts, auditability, per-org isolation, survivability during provider outages and org-wide incidents.High-level architecture:- Ingest layer (API/gRPC + connectors) → Normalization & enrichment → Deduplication/Grouping Engine → Policy Engine (rate limits, mute, escalation) → Delivery Gateway → Channel Integrations (SMS, Email, Slack) → Observability & Admin UI.Core components:1. Ingest: idempotent endpoints, persistent write to ordered durable queue (Kafka partitioned by service/org).2. Normalization: canonical alert schema, attach metadata (severity, fingerprint).3. Dedup/Grouping: sliding-window, fingerprinting, cluster-state in distributed cache (Redis Cluster or Cassandra) to group similar alerts and increment counters.4. Policy Engine: evaluates on-call schedules, mute windows, per-destination rate-limits; persistent policy store (Postgres) and in-memory cache.5. Escalation Manager: orchestrates escalation chains, timers, retries, and dynamic on-call lookups (integrates with rota service).6. Delivery Gateway: per-channel workers, token buckets for global and per-recipient rate-limiting, circuit breakers per provider.7. Integrations: adapters with retry/backoff, fallback routing (e.g., if SMS provider down, fallback to Slack or voice), and dead-letter queues.8. Admin/Operations: UI to mute, override, inspect alerts; audit log in append-only store.Handling alert storms:- Backpressure via queueing and adaptive sampling: when backlog grows, move to grouping-first behavior (coalesce similar alerts into summary), escalate only top-N by severity, apply exponential backoff per source.- Auto-throttle non-critical channels; push only summary digests to paging channels.- Use bloom filters + counters to detect floods and auto-mute noisy sources with human review.Integration failures & provider outages:- Circuit breaker per provider; on trip, mark provider degraded and route to backup(s).- Guaranteed-at-least-once via persistent queues; use idempotency keys to avoid duplicates.- Graceful degradation: if all providers for a channel fail, surface to on-call via alternative channels and to Ops UI; send critical alerts to escalation contacts via voice calls (third fallback).- Health-checks and failover for critical components, run multiple providers for SMS/email.Keeping system operational during org-wide incidents:- Multi-region deployment with active-active Kafka and DB failover; keep read-only operations if necessary.- Provide emergency modes: “critical-only” path that bypasses nonessential processing, minimal dependencies (no external enrichment) to reduce surface.- Separate control plane vs data plane: control plane can be degraded without losing delivery for in-flight critical alerts.- Run dedicated on-call SRE rota for alerting system; expose circuit-breaker and backlog metrics to status page and alert on self-health.Data, observability and security:- Audit trail for each alert lifecycle; metrics (P99 latency, queue depth, delivery success, provider SLAs).- RBAC in Admin UI; encryption in transit & at rest; compliance for PIIs in messages.Trade-offs:- Strong consistency for deduplication (using distributed locks) vs availability — prefer eventual consistency with idempotency to maximize throughput.- Cost vs reliability: multiple provider redundancy increases cost but required for high-urgency paging.This design prioritizes resilience (multi-region, fallback providers, circuit breakers), operational control (mute/escalation windows, admin overrides), and safe behavior during storms (grouping, sampling, critical-only mode).
HardTechnical
26 practiced
Technical domain (hard): Describe safe practices for rolling out cross-service configuration changes in a microservices architecture (e.g., feature toggles, config flags, central config store). Cover validation, canarying config changes, schema/format evolution for configs, atomic rollback strategies, and how to avoid cascading failures due to invalid configs.
Sample Answer
Start with principles: validate early, deploy gradually, fail fast and safely, and make rollbacks atomic and observable.Validation- Validate config syntactically (JSON/YAML schema) and semantically (value ranges, required dependencies) before acceptance into central store (CI job + pre-commit hooks).- Unit/integration tests that load new config into a sandboxed instance and exercise critical code paths (feature flag on/off).Canarying config changes- Target a small subset of services/instances or traffic (by region/user-id hash) to pick up the new config first.- Use short-lived canary windows with health checks and automated metrics (error rate, latency, business KPIs). If thresholds breach, auto-revert.Schema/format evolution- Version config schemas. Support backward-compatible reads: new fields optional with sensible defaults; use feature negotiation where clients declare supported schema version.- Migrate in two phases: (1) deploy code that tolerates old+new schemas; (2) safely introduce stricter validation and remove legacy handling later.Atomic rollback strategies- Treat config changes as transactional: update central store with a version id and flag “active”. Rollout flips active pointer; rollback flips back.- Keep immutable config versions and implement fast switch-over (no need to redeploy code) plus automated rollback triggers when health checks fail.Avoid cascading failures- Enforce guardrails: kill-switch global override, per-service validation, max-value limits.- Fail closed for critical controls (reject dangerous config) and fail open only when safe.- Rate-limit config propagation and add exponential backoff on fetch errors.- Observe and alert: dashboards + automated playbooks for common failure modes.Example: feature-flag rollout — validate flag schema in CI, enable for 1% users, monitor errors and conversion, expand to 10/50/100% only after clear metrics; maintain immutable versions and a quick revert switch to previous version.
MediumTechnical
29 practiced
Problem-solving (medium): Design an alerting strategy for a microservice to minimize pager fatigue while ensuring high-severity incidents are caught. Consider SLOs, burn-rate alerts, multi-signal alerts, grouping, noise reduction (deduplication, suppressions), and seasonal traffic patterns. Describe concrete thresholds and how you'd validate them.
Sample Answer
Requirements:- Catch high-severity incidents quickly (user-impacting failures) while minimizing noisy/avoidable pages.- Preserve developer on-call sanity; route low-urgency issues to teams via tickets.- Support seasonal/traffic variation and deployments.Strategy (high-level):- Define clear SLOs (service-level objective) and drive alerts from error-budget / user-impact signals.- Use multi-signal alerts (correlated signals) and burn-rate alerts as primary paging triggers.- Reduce noise via deduplication, suppression during known noisy periods (deployments, ramps), grouping, and tiered notification channels.Concrete design and thresholds:- SLOs: Availability SLO = 99.9% (monthly) for successful user transactions; Latency SLO = p95 < 300ms.- Burn-rate alerts (primary pager): compute error budget burn rate over sliding windows. Use graduated thresholds similar to SRE guidance: - Critical page: burn-rate > 14 for 1 minute OR burn-rate > 6 for 5 minutes → page on-call. - High-priority (non-blocking page): burn-rate > 3 for 60 minutes → page or escalate. - Informational: burn-rate > 1 for 6 hours → create ticket.- Multi-signal paging (complementary rule): Page if BOTH error rate spike AND at least one supporting signal within 2 minutes: - Error-rate spike: errors/sec > baseline + 5σ or relative increase > 3x over last 5m - AND latency/p95 > SLO threshold (e.g., > 300ms) OR infra signal (CPU > 85% cluster-wide / instance restarts > 5/min).- Fast-degrade/blackout detection: if overall success rate falls below 90% for >1 minute, immediate page (system-wide outage).- Non-paging alerts: single-signal anomalies (isolated increased 5xxs w/o latency/infra signal), config errors for small customer subsets, or resource warnings → send to Slack/channel and auto-create ticket.Noise reduction and grouping:- Deduplication: fingerprint alerts by root-cause tags (route, exception type, stack trace) and collapse similar alerts into one incident.- Suppress during known maintenance/deploy windows; start a deployment marker that auto-suppresses noisy alerts for a short window (e.g., 5–10 minutes), but still capture metrics for postmortem.- Intelligent grouping: group by affected customer scope (global vs single-tenant) and by service component; only page for global-impact groups.- Rate-limit repeat pages: do not resend the same page for the same incident more than once every 15 minutes unless severity escalates.Seasonal traffic patterns:- Use dynamic baselines: compute anomaly thresholds relative to rolling historical windows (hour-of-day and day-of-week percentiles). For example compare current 5m error rate to the historical 95th percentile for that same hour/day.- Auto-scale aware thresholds: incorporate traffic-normalized metrics (errors per 1000 requests) rather than absolute errors.Validation and tuning:- Backtest on historical data: run alert rules against last 6–12 months of telemetry and measure false positives, time-to-detect (TTD), and number of pages. Target <=1 actionable page/week per on-call and TTD < 5 minutes for critical incidents.- Canary/chaos testing: inject failures in staging and run in production canary; verify alerts fire appropriately and not during canary-only false positives.- Post-incident review: after each page, log whether it was actionable. Maintain an alert-quality metric and perform monthly pruning (suppress/adjust rules with >80% false-positive rate).- Continuous feedback: on-call during rotation can mark pages as noise; automation creates tickets to tune thresholds.Runbook & ownership:- Every paging rule links to a concise runbook with immediate triage steps, key dashboards, rollback or mitigation actions, and expected escalation timeline.- Define owner for each alert and a review cadence (quarterly) to reassess thresholds with traffic changes.Why this minimizes pager fatigue:- Pages only on high-confidence multi-signal or burn-rate breaches, not on single noisy metrics.- Deduplication/grouping lowers duplicate pages.- Dynamic baselines and suppression handle seasonal and deployment-caused noise.- Continuous validation ensures thresholds remain tuned and actionable.
EasyTechnical
26 practiced
Explain Recovery Time Objective (RTO) and Recovery Point Objective (RPO). For a web payment service, propose RTO/RPO targets for three tiers of features: critical transactions, user profile updates, and analytics jobs. Describe how you would test and validate RTO/RPO periodically and what tooling/metrics you would rely on (e.g., backups, replication lag, failover tests).
Sample Answer
RTO (Recovery Time Objective) is the maximum acceptable time to restore a service after an outage. RPO (Recovery Point Objective) is the maximum acceptable amount of data loss measured backward in time (how far behind your recovered data can be).Suggested targets for a web payment service:- Critical transactions (payment processing, authorization): RTO ≤ 1 minute, RPO ≤ 0–5 seconds. Rationale: financial correctness and customer trust require near-zero downtime and minimal data loss.- User profile updates (address, payment method changes): RTO ≤ 15–60 minutes, RPO ≤ 1–15 minutes. Rationale: important but less time-sensitive; eventual consistency acceptable for short windows.- Analytics / batch jobs (reports, data warehousing): RTO ≤ 4–24 hours, RPO ≤ 24 hours. Rationale: tolerates longer recovery and reprocessing.How to test and validate periodically:- Run scheduled chaos/failover drills (monthly for critical stacks, quarterly for others). Simulate primary DB failover, app node termination, region outage.- Perform backup restore drills (full restore to a sandbox) to validate backup integrity and recovery time. Measure end-to-end restore duration.- Rehearse transaction replay: for critical transactions keep a durable transaction log or CDC stream; replay to a staging environment to validate RPO limits.- Measure replication lag, backup completion times, failover time, and application readiness checks during drills.Tooling / metrics to rely on:- Backups: timestamped backups, verification jobs, restore time metrics.- Replication: DB replica lag (ms/s), WAL/ binlog position, CDC consumer offsets.- Failover: automated failover times (seconds), health-check latency, orchestration logs (Kubernetes/HAProxy/Cloud failover).- Observability: SLOs, synthetic transactions (payment attempt every minute), SLIs (success rate, latency), alerting on lag/backup failures.- Automation: IaC scripts to spin up recovery environments and run smoke tests, and runbooks documenting manual steps.Periodic review: map business impact to RTO/RPO in an SLA, review after incidents, and adjust targets/tooling quarterly.
HardSystem Design
28 practiced
System design / operations (hard): You must perform a zero-downtime migration of a 2TB table for a service with high write throughput across multiple regions. Describe a complete strategy including dual-write or expand-contract pattern, chunked backfill approach, consistency verification, how to handle replicas and failovers, rollback plan, monitoring, and estimated validation time. Consider RPO/RTO constraints and bandwidth/cost trade-offs.
Sample Answer
Requirements & constraints:- Zero downtime for reads/writes, high write throughput across regions, table size 2TB, strict RPO (minutes) and RTO (<=1 hour).- Minimize cross-region bandwidth/cost.High-level approach:1. Clarify schema change type (add column / transform / split). Use expand-contract + safe backfill rather than destructive in-place.Plan (step-by-step):1. Prep: deploy versioned schema support (new column(s)/table). Add feature-flagged code that can read/write both old and new schema.2. Dual-write (expand-contract): - Expand: create new_table (same PKs, new schema) or add nullable columns. Start writing to both old and new (dual-write) from app servers behind a feature flag. Writes are idempotent and include write-version metadata. - Contract: after full backfill and verification, switch reads to new schema and stop writing old.3. Chunked backfill: - Backfill worker service reads old data in parallel, by token/range/PK hash, small batches (e.g., 10k rows) with concurrency tuned per region. - Use change-data-capture (CDC)/binlog to capture writes that occur during backfill; apply CDC stream to new table. - Order: snapshot backfill + ongoing CDC replay to ensure no lost updates. - Rate-limit per region to respect bandwidth/cost; prioritize hot partitions first.Consistency verification:- For each chunk, compute deterministic checksum (e.g., CRC64) of source and destination rows; compare per-chunk.- Maintain backfill ledger with chunk id, status, checksum, timestamp.- Run sampling/full diff for critical fields; for large data prefer per-row hash + Bloom filters to detect mismatches.- Use read-after-write tests and canary traffic to validate reads from new table.Replicas & failovers:- Perform backfill against primary-writeable region local replica when possible; replicate changes to other regions via existing replication.- For multi-region primary failover: ensure CDC consumer can resume from last LSN/offset; store offsets durably in coordinator (e.g., Kafka/DB).- Maintain compatibility: if failover switches primary, dual-write continues; CDC must capture from whichever primary becomes leader.Rollback plan:- If mismatch or perf regressions: flip feature flag to stop writing to new table (or revert reads to old), let CDC/ledger pause, and resume writes only to old schema. Keep new table for diagnostics.- For partial failures, abort backfill, repair data for affected chunks, re-run only those chunks.Monitoring & alerting:- Metrics: backfill progress (% rows/chunks), CDC lag (seconds), chunk error rates, write latency P50/P99, cross-region bandwidth, checksum mismatch count.- Alerts: CDC lag > threshold, chunk failure rate spike, read/write latency regressions.- Dashboards: per-region progress, top hot partitions, rolling validation results.Validation time estimate:- Example: 2TB ~ 2e12 bytes; assume avg row 1KB => ~2M rows.- With 100 parallel workers across regions doing 2000 rows/sec each => 200k rows/sec => ~10s to process 2M (but realistically overhead): expect 1–6 hours including CDC replay, verification, throttling.- Realistic estimate: 4–12 hours depending on bandwidth, validation intensity, and throttling.Trade-offs:- Dual-write increases write latency and complexity but minimizes RPO loss.- CDC+backfill reduces write impact but requires robust CDC and idempotency.- Bandwidth vs time: higher parallelism shortens window but increases cost and risk of DB resource contention.Key safeguards:- Automated canary reads, transactional idempotency, chunk-level retries with exponential backoff, and immutable audit logs for verification. Conduct a rehearsed dry-run on a staging copy before production.
Unlock Full Question Bank
Get access to hundreds of Technical Problem Solving and Ownership interview questions and detailed answers.
