Infrastructure and Service Level Concepts Questions

Understanding how infrastructure components and service level objectives influence reliability. Topics include the roles of containers, container orchestration platforms, load balancers, message queues, and regions or availability zones in fault isolation and recovery. Candidates should know the difference between a service level agreement and a service level objective, how error budgets are used, how a service level objective failure affects prioritization and incident response, and operational controls such as circuit breakers, exponential backoff, alerting, and runbooks. This topic covers the interplay between architecture choices and operational targets that ensure systems meet customer and business expectations.

HardTechnical

40 practiced

A planned cross-region failover succeeded but caused two hours of customer-visible consistency issues and breached the SLO. Draft the post-incident plan: timeline reconstruction, customer communications, root-cause analysis, remediation steps, error-budget accounting, and actions (runbook changes, tests) to prevent recurrence.

Sample Answer

Timeline reconstruction- T-48h: Change window scheduled and notified; failover runbook version v1.2 confirmed.- T-2h: Pre-failover checks passed (replication lag <100ms, health checks green).- T0: Initiated cross-region failover (promote secondary, update DNS & LB).- T+5m: Failover complete; services reported healthy, but write latencies rose.- T+10m–T+2h10m: Customers observed inconsistent reads (stale reads & lost writes) for ~2 hours. Ops rotated caches and retried writes; partial mitigation at T+1h50m.- T+2h10m: Final corrective action: forced re-sync and global traffic drain -> consistency restored.- T+4h: Post-failover verification and customer notices.

Customer communications- Immediate (within 30m): Incident banner + brief status on status page acknowledging degraded consistency and that failover completed; ask customers to avoid write-heavy operations while we investigate.- Hourly updates: progress, affected regions, ETA for resolution.- Post-incident (24h): Detailed postmortem, apology, actions, and SLA credit instructions.

Root-cause analysis- Primary cause: Race between DNS/LB cutover and async replication catch-up—promoted region accepted writes before all replica shards were fully synced; reads routed to replicas still behind.- Contributing factors: runbook assumed bounded replication lag; lack of synchronous handoff or write fencing; monitoring had no alert for inter-region replication drift; no traffic quiesce step.

Remediation (short/medium/long)- Short (<=72h) - Add immediate mitigation: enforce a “writes disabled until quorum” flag during promotion; document emergency rollback steps. - Issue one-time SLA credit; transparently share incident report.- Medium (2–4 weeks) - Update runbook to include pre-promotion checkpoints: replication lag thresholds, quorum checks, and explicit traffic-quiesce window; require checklist sign-off. - Implement an automated gating script that verifies shard sync and flips a “promote-allowed” token. - Add monitoring/alerting for cross-region replication lag and divergence with paging on threshold.- Long (1–3 months) - Re-architect: support leader election with distributed consensus or synchronous handoff for metadata/writes where feasible. - Introduce versioned read-after-write guarantees (client tokens or sticky sessions) to mask transient inconsistency.

Error-budget accounting- SLO breached window: 2 hours of consistency violation for affected customer set. Compute consumed error budget = (affected-user-percentage * 2 hours) relative to monthly SLO window. Example: if 10% users affected and monthly target 99.9% availability compute prorated credit and log budget debit; suspend non-critical launches until budget restored.

Actions (runbook changes, tests)- Runbook changes - Add explicit “quiesce writes” step, automated verification of replica catch-up, rollback decision tree, and stakeholder notification template. - Add pre-change checklist with owner sign-offs and automated gating.- Tests - Injected-failover chaos tests in staging that simulate replication lag and verify system refuses promotion until safe. - Nightly canary failovers with synthetic write/read verification across regions. - Add end-to-end client tests for read-after-write within 5 minutes post-failover.- Drills - Conduct tabletop and live failover drills quarterly, include incident commander and customer-comms role-play.

Accepting responsibility, we’ll implement the short-term mitigations within 72 hours, deliver updated runbook and monitoring in 2–4 weeks, and roll out architectural improvements over the next quarter. We’ll also provide SLA credits and a customer Q&A within 48 hours.

EasyTechnical

42 practiced

For a caching layer (CDN or in-memory cache) list the SLIs you would track (e.g., cache-hit ratio, cache-latency, freshness/staleness) and explain how each SLI relates to a user-facing SLO and how you would instrument and slice these SLIs (by endpoint, tenant, or region).

Sample Answer

Key SLIs to track for a caching layer and how they map to user-facing SLOs, plus instrumentation and slicing:

1. Cache Hit Rate (requests served from cache / total requests)- SLO relationship: High hit rate reduces origin load and typically improves user-perceived latency and availability (e.g., 95% of requests served within latency SLO).- Instrumentation: counter for total requests and cached responses (use x-cache or X-Cache-Status header, CDN logs). Expose as ratio; track moving windows.- Slicing: endpoint/path, tenant, region, content-type, device.

2. Cache Latency (p50/p95/p99 of time to serve from cache)- SLO relationship: Directly maps to user latency SLOs (page load, API response).- Instrumentation: histogram timers for cache lookup + serve time (client-facing and edge-to-origin for misses). Correlate with client RUM and backend traces.- Slicing: endpoint, tenant, region, POP (edge location).

3. Origin Offload / Origin Request Rate- SLO relationship: Protects origin availability and error budgets; SLO might limit percent of requests that hit origin.- Instrumentation: counter for origin requests, bytes to origin.- Slicing: endpoint, tenant, region, deployment/service.

4. Freshness / Staleness (fraction of responses within TTL or served stale)- SLO relationship: Ensures correctness and user trust (e.g., 99% of catalog reads ≤ TTL).- Instrumentation: evaluate response age using Date/Age headers, compare to expected TTL; track served-stale events.- Slicing: endpoint, tenant, content-type.

5. Cache Error Rate (5xx or malformed cached responses)- SLO relationship: Maps to availability SLOs; cached errors still harm users.- Instrumentation: counters for error responses served by cache and errors fetching from origin.- Slicing: endpoint, tenant, region, POP.

6. Eviction Rate & Capacity Utilization- SLO relationship: High evictions or full caches reduce hit rate and increase latency; affects capacity planning SLOs.- Instrumentation: counters/gauges from cache (evictions/sec, memory/slot usage).- Slicing: cache cluster, shard, tenant (if multi-tenant).

7. Bandwidth Savings / Cost Metrics- SLO relationship: Operational SLOs around cost and origin egress limits.- Instrumentation: bytes served from cache vs origin.- Slicing: endpoint, tenant, region.

Practical notes:- Use histograms for latency, counters for rates, and gauges for utilization. Tag metrics with endpoint/tenant/region/POP.- Combine with synthetic probes (testable paths) and RUM/traces to validate end-to-end experience.- Define SLOs in user-centric terms (e.g., 99% of GETs return <200ms and correct content). Derive acceptable thresholds for each SLI to back that SLO.- Alert on SLI regressions by slice (e.g., tenant-specific hit-rate drop) to enable fast, targeted remediation.

HardSystem Design

47 practiced

Design a scalable alerting pipeline for an organization with 1000 microservices and diverse SLOs. Cover telemetry ingestion, rule evaluation, deduplication/grouping, correlation with deployments, runbook linking, on-call scheduling, and how you ensure the alerting system itself meets reliability and latency requirements.

Sample Answer

Requirements:- Ingest high-cardinality telemetry from 1000 services (metrics, traces, logs) with SLO-based alerting.- Low-latency (<30s) for high-priority alerts, scalable to millions of metrics/sec.- Deduplicate, group, correlate with deployments, attach runbooks, integrate on-call schedules, and be highly reliable.

High-level architecture:Client agents → Ingestion layer (Kafka/NSQ) → Stream processing (Flink/Kafka Streams) + Metrics store (Cortex/Prometheus Remote Write) + Tracing store (Jaeger) → Alert Engine (stream & rule evaluator) → Alert Deduper/Correlator → Alert Router (Alertmanager / PagerDuty) → On-call & Runbook service.

Components & responsibilities:1. Telemetry ingestion- Agents (Prometheus exporters, OpenTelemetry) push to a resilient ingestion tier (kafka) for backpressure and buffering.- Use partitioning by service/tenant to scale and enforce quotas.2. Rule evaluation- Two paths: real-time streaming rules for SLO breaches (CEP in Flink) and long-window/aggregation rules in Prometheus/Cortex for lower-criticality checks.- Express rules with labels/templating and link to SLO metadata.3. Deduplication & grouping- Compute stable fingerprint: (service, SLO-id, error-type, cluster, fingerprint-hash).- Use sliding-window dedupe in stream layer; group related alerts by fingerprint and by similarity (anomaly signature hashing + vector similarity for noisy incidents).4. Correlation with deployments- Ingest deployment events from CI/CD (webhooks) into event store; enrich alert context with latest deploy metadata (commit, job id, canary stage).- Correlator checks temporal proximity and affected services to surface likely deployment-caused incidents.5. Runbook linking & enrichment- Each SLO/rule carries runbook_id. Alert payload includes runbook URL, playbook steps, diagnostic queries, and automated remediation scripts (opt-in).- Provide ChatOps snippets and a "Runbook quick-run" that runs pre-approved diagnostics.6. On-call scheduling & routing- Central schedule service (integrates with HR & PagerDuty) exposes team rotas and escalation policies.- Router routes by team and severity; supports escalation, deduped notification, suppression windows, and silence APIs.7. Reliability & latency of alerting system- SLOs: 99.95% availability for alert pipeline; p95 alert delivery latency target (30s) for critical alerts.- Design: partitioned, replicated Kafka + multi-zone Flink + HA alert router (k8s with anti-affinity). Persist state in replicated key-value store (etcd/consul or Flink state backend with RocksDB + S3 checkpoints).- Backpressure: throttle low-priority telemetry, adaptive sampling for high-cardinality labels.- Testing & resilience: chaos testing, replayable telemetry (retained in cold storage), canary deployments of rules.- Observability: end-to-end tracing, per-stage SLIs, alerting-on-alerting, automated failover scripts.

Trade-offs:- Streaming CEP gives low latency but is more operationally heavy than pure Prometheus. Hybrid model balances complexity vs performance.- Deduplication aggressiveness tuned to avoid masking distinct root causes; provide manual override.

Why this works:- Buffered ingestion and partitioning scale linearly.- Streaming evaluation + fingerprinting deliver low-latency, stable alerts.- Tight integration with deployments and runbooks reduces MTTT/MTTR.- HA design and SLIs ensure the alerting system itself remains reliable.

MediumSystem Design

52 practiced

Design a monitoring and alerting strategy for a REST API that must meet SLOs: 99.95% availability and p95 latency <200ms over a 30-day window. Traffic is ~10k RPS globally. Describe which SLIs to collect, alert thresholds for warnings vs paging, burn-rate detection windows, aggregation strategy, and runbook linking.

Sample Answer

Requirements & constraints:- SLOs: 99.95% availability and p95 latency < 200ms over rolling 30-day window- Traffic: ~10k RPS global- Alerts must distinguish warning vs paging, support burn-rate detection, and include runbook links

SLIs to collect (per region and global):- Success rate: fraction of requests with 2xx within acceptable error-class (exclude planned maintenance)- Latency distribution: p50, p95, p99 end-to-end request latency (client‑perceived)- Traffic: RPS, unique users (for dimensioning)- Saturation signals: CPU, memory, queue lengths, thread pool saturation, DB latency/errors, downstream error rates- Deployment & config changes: deploy events, config rollouts

Aggregation strategy:- Use 1s ingestion, aggregate to 10s/1m for short windows and 5m/1h for long-term. Compute global SLI as weighted average by successful requests (not simple mean) to avoid skew from low-traffic regions.- Keep per-service, per-region, and overall SLI timeseries to localize incidents.

Alert thresholds:- Warning alerts (non-paging): triggered when projected 30-day SLO breach at low burn rate or short-term degradations requiring action - Success rate drop causing burn-rate > 1x for 1 hour OR p95 > 200ms sustained over 5 minutes in a single region - Resource saturation > 70% for 5+ minutes- Paging alerts (pager duty): immediate human attention when burn-rate indicates rapid consumption - Burn-rate >= 5x (i.e., if current error rate continued would exhaust error budget within 6 days) observed for 5 consecutive minutes - Global p95 > 200ms for 2 minutes AND traffic > 20% of baseline (to avoid noisy short blips) - Success rate drops that imply <99.95% if sustained for 5 minutes globally (compute using rolling window)

Burn-rate detection windows:- Short: 5m and 1h for rapid detection (use to page when >=5x)- Medium: 6h for operational warnings (>=2x)- Long: 24h for planning and trend alerts (>=1x)

Implementation notes:- Use sliding-window error budget algorithm: burn-rate = (observed errors / allowed errors) * (window_length / SLO_window). Precompute allowed error budget for 30d.- Avoid alert fatigue: require sustained condition (e.g., consecutive buckets) and suppression during known maintenance. Deduplicate across regions.

Runbooks & automation:- Each alert links to runbook with: - Quick triage checklist: check region, recent deploys, upstream/downstream errors, resource metrics, circuit-breaker/queue size - Immediate mitigations: rollback, traffic-shift to healthy region, scale-up steps, circuit-breaker parameter adjustments - Postmortem template and communication checklist- Automate safe mitigations: traffic shifting, autoscale policies, and fast rollback pipelines with approvals.

Metrics to review post-incident:- Error budget remaining, root-cause service, latency percentile breakdown, traffic patterns, deploy correlation.

This design balances fast detection, low noise, regional granularity, and clear on-call actions so we protect the 30-day SLO while minimizing unnecessary pages.

HardSystem Design

41 practiced

You need to migrate a high-availability monolithic service to microservices without degrading SLOs. Provide a phased plan: how to decompose components, use the strangler pattern or API gateway, handle data migration and ownership, deploy canaries, and safely measure and rollback during migration.

Sample Answer

Phase 0 — Prep & Constraints- Capture SLOs/error budgets (latency p50/p95, availability) and current baselines.- Inventory modules, dependencies, data models, traffic patterns, peak windows.- Add extra observability: distributed tracing (Jaeger), metrics (Prometheus), logs (ELK), synthetic tests.

Phase 1 — Decompose & Design- Decompose by business capability (bounded contexts) not technical layers. Pick a low-risk, high-value first candidate (e.g., auth, billing) for pilot.- Define clear service contracts (API schemas), ownership, SLIs per service.- Design data ownership: each service should own its primary datastore; avoid shared writes.

Phase 2 — Strangler + API Gateway- Put an API Gateway in front (Kong/Envoy) to route requests; implement feature flags to toggle routes.- Use the Strangler Pattern: route a subset of traffic to new microservice while monolith still handles rest.- Start with read-only delegations: have new service call monolith for data reads until data ownership migrates.

Phase 3 — Data Migration & Consistency- Use Change Data Capture (Debezium/Kafka) to stream DB changes to new service’s datastore; backfill historical data.- Dual-write only with careful instrumentation and idempotency; prefer async CDC to avoid write latency regressions.- Implement versioned schema and anti-entropy reconciliation jobs; define strong/weak consistency boundaries and compensating transactions.

Phase 4 — Deploy Canary & Observe- Deploy microservice in k8s; use canary tooling (Flagger/Argo Rollouts) to shift traffic progressively (1%, 5%, 25%, 100%).- Run canary tests: synthetic transactions, SMIs, load tests, chaos experiments in isolated windows.- Monitor SLIs in real-time: error rate, latency percentiles, saturation metrics, business KPIs. Alert on SLO breaches and anomaly detection.

Phase 5 — Safety Controls & Rollback- Safety patterns: circuit breakers, bulkheads, retries with jitter, rate limiting at gateway.- Define automatic rollback criteria: e.g., error rate increase > X for Y minutes, latency p95 > baseline + Z%, SLO burn rate threshold. Automate rollback via canary controller.- Have manual kill-switch routes in gateway and database write-fallback to monolith.

Phase 6 — Cutover & Harden- Gradually move authoritative reads/writes after stable canaries; retire monolith endpoints incrementally.- Finalize ownership: remove dual-write paths, decommission old schemas.- Post-mortem, tune SLOs, capacity plan, runbook updates, and DR tests.

Concrete tools/patterns- Routing: Envoy/API Gateway; Canary: Flagger/Argo; CDC: Debezium → Kafka; Orchestration: Kubernetes; Observability: Prometheus + Jaeger + Grafana.

Why this minimizes SLO impact- Strangler + canaries localize risk and let you measure impact on real traffic.- CDC and async migration avoid blocking writes/latency spikes.- Feature flags, circuit breakers and automated rollback ensure fast recovery if SLOs degrade.

Unlock Full Question Bank

Get access to hundreds of Infrastructure and Service Level Concepts interview questions and detailed answers.

Join thousands of developers preparing for their dream job.