EasyTechnical

54 practiced

Explain the difference between latency and throughput. Provide concrete examples of systems where optimizing for latency reduces throughput and vice versa, and describe measurement and monitoring approaches to evaluate trade-offs for a backend that handles both batch and interactive workloads.

EasyTechnical

74 practiced

You open a service dashboard and observe: CPU usage across app servers steady at 30%, memory usage gradually increasing over several days, disk I/O spikes during traffic surges, and p95 latency rising from 100ms to 400ms. As a solutions architect, list the first five diagnostic steps you would take and the data points or quick checks you would collect to triage this as an on-call responder.

EasyTechnical

39 practiced

You are defining alert thresholds for a customer-facing API used by many clients. Describe an alerting strategy that balances early detection with noise reduction. Include severity tiers, multi-condition alerts, runbook integration, escalation pathways, and how to test alerts before enabling them in production.

Sample Answer

I’d design an alerting strategy focused on customer impact, signal quality, and clear operational playbooks.

1) Severity tiers (examples tied to SLAs)- Sev 1 (P1): API unavailable or error rate >5% for 5m or latency P95 >2s for 5m — immediate page to on-call and manager.- Sev 2 (P2): Degraded performance — error rate 1–5% or latency P95 1–2s for 10m — page on-call, notify dev team.- Sev 3 (P3): Trends/early warnings — sustained latency increase <1s or traffic anomaly — only Slack alerts to SRE/owner during business hours.- Sev 4: Informational metrics, reports.

2) Multi-condition alerts (reduce noise)- Combine conditions with sensible windows: alert only when error_rate > threshold AND traffic > baseline to avoid alerting on low-traffic noise.- Use anomaly detection for baselines (rolling 7d/28d) and require sustained breach (e.g., 3 consecutive windows).

3) Runbooks & context- Every alert links to a concise runbook: impact summary, telemetry links (traces, logs), common causes, immediate mitigations, stakeholder list, rollback steps, customer notification templates, RCA owner.- Include SLAs and customer-facing communication guidelines.

4) Escalation pathway- On-call handles initial triage within 5 min (P1), if not resolved in 15 min escalate to team lead; 30 min escalate to engineering manager and customer success for affected clients. Automate paging, SMS fallback, and include on-call rotations in the alert payload.

5) Testing before prod- Create a staging environment mirroring prod telemetry. Use synthetic traffic to simulate error/latency scenarios and fire alerts to a test routing (no pages). Run chaos tests and smoke tests; perform a “ring” rollout: enable informational → multi-condition → paging after 48h of stable behavior. Use alert suppression windows during deploys and maintain a runbook-driven post-testing review.

This approach balances early detection (trend/anomaly alerts) with noise reduction (multi-condition, baselines, sustained windows), ensures rapid, documented responses, and ties to SLAs and client communication.

HardTechnical

42 practiced

Case study: A video-transcoding backend handles 10k jobs/day with average CPU time 2 minutes and memory 1.5GB per job. Forecasts expect 10x growth in 6 months. Propose a capacity model including queueing, batch sizing, autoscaling or fleet sizing, cost estimation, and monitoring to validate the model during ramp.

Sample Answer

Requirements & assumptions:- Current: 10k jobs/day ≈ 7 jobs/minute. Average CPU time 2 min (vCPU-minutes) and 1.5GB RAM per job. Jobs are independent, variable arrival.- Forecast: 10x in 6 months → 100k jobs/day ≈ 70 jobs/min.- SLO: 95% jobs complete within 30 minutes; max concurrency latency budget 60 minutes.- Acceptable batching: small-scale batching permitted for efficiency (e.g., container re-use).

Capacity model (queueing + concurrency):- Model queue as M/M/c (Poisson arrivals) to size workers for target latency.- Arrival rate λ = 70 jobs/min. Service rate μ per worker = 1 job per 2 minutes = 0.5 jobs/min.- Required concurrent workers (c) ≈ λ/μ = 70 / 0.5 = 140 workers to keep utilization ~100%. For target utilization 70%: c = 140 / 0.7 ≈ 200 workers.- Use Erlang C to compute queueing delay; target 95% of jobs start within 5 minutes → iterate c to meet that.

Batch sizing / packing:- Run each job in a container; enable packing multiple lightweight transcoding tasks per host if CPU allows (e.g., host with 32 vCPUs can run up to 28 vCPU containers leaving overhead).- Preferred: use node types with high single-thread performance; where jobs are multi-threaded, choose based on CPU-profile.- Example: If using c5.4xlarge (16 vCPU, 32GB), map ≈14 concurrent jobs (1 vCPU per job) → one instance serves 14 jobs. For 200 workers => ~15 instances.

Autoscaling & fleet sizing:- Hybrid: baseline fleet = 70% of expected steady load (140 workers) always-on for latency predictability; autoscale up to peak 200–240 workers based on queue length and CPU utilization.- Scaling triggers: queue length > X (e.g., >30 pending) OR average CPU > 60% for 2 minutes → scale out; scale in when queue=0 and CPU <30% for 5 minutes.- Use warm pools / pre-warmed containers or spot instances with fallbacks for cost.

Cost estimation (example, cloud):- Assume instance price $0.50/hr for chosen size hosting 14 jobs. 200 workers ≈ 15 instances → 15 * 24 * 30 * $0.50 ≈ $5,400/month baseline. Add storage, networking, orchestration overhead -> estimate $7k–9k/month. For 10x growth scale costs ~10x if fully elastic; with optimization (spot/packing) expect 4-6x.

Monitoring & validation during ramp:- Metrics: arrival rate, queue length, start latency, job runtime, CPU/memory per container, host utilization, preemption events, cost per job.- Dashboards & alerts: queue length > threshold, P95 start latency > target, CPU saturation >80%, unexpected job failures rate >1%.- Validation plan: run canary increases (10% increments weekly). For each step: - Observe queueing model predictions vs real: compare measured wait times to Erlang C outputs. - Tune packing and scale thresholds; measure cost per job and latency. - Load-test synthetic traffic to validate autoscaling and warm-pool behavior.- Risk mitigation: pre-warm capacity before major ramps, use burstable reserved capacity, fallback to lower-quality transcode profiles if overloaded.

Trade-offs:- More baseline reduces cold-start latency but increases cost.- Aggressive packing lowers cost but risks noisy-neighbor; isolate critical jobs.- Spot instances lower cost but require capacity fallback.

This model gives a repeatable sizing method (λ, μ → required c), autoscaling rules, cost projection, and a monitoring/validation plan to iterate during the 10x ramp.

HardSystem Design

51 practiced

Design a monitoring strategy for a service expected to scale from 1,000 to 1,000,000 concurrent connections. Address metric collection sampling, cardinality control, aggregation and rollups, alerting at scale, and dashboard design approaches that keep operator workflow effective without overwhelming them with data.

Sample Answer

Requirements & constraints:- Track latency, errors, throughput, resource usage for service growing 1k → 1M concurrent connections.- Low agent overhead, bounded storage/cost, actionable alerts, fast drilldowns for operators.

High-level approach:- Use OpenTelemetry for instrumentation, a scalable pipeline (ingest via Kafka/OTLP → stream processor → long-term TSDB like Cortex/Mimir or ClickHouse for metrics, and ElasticSearch for logs/traces index).- Separate telemetry by purpose: high-cardinality logs/traces; low-cardinality aggregated metrics.

Metric collection & sampling:- Client/edge: capture high-frequency raw events only for a small sampled percentage (e.g., 0.1–1%) of connections for full traces.- Use adaptive sampling: increase sample rate on anomalies or error spikes.- For metrics, emit aggregated counters/histograms per host / service, not per-connection. Use client SDK to bucket histograms at source.

Cardinality control:- Enforce a label policy: allow only a small fixed set of high-value labels (service, region, instance_type, deployment). Reject or hash free-form IDs (user_id, session_id) at ingest or use hashed/rollup keys for limited buckets.- Use dynamic label cutoffs with sampling for new label values and a leaderboard to surface top-K values; older low-volume values roll up into “other”.

Aggregation & rollups:- Store high-resolution (10s) metrics for 7 days, mid-resolution (1m) for 30 days, and rollups (5–15m) for 1 year.- Pre-aggregate at ingestion: sum/counter deltas, quantile approximation (HDR/TDigest) at source for histograms to reduce cardinality.- Keep raw traces only for sampled flows; index trace metadata (span counts, root latency, error flag) for correlation.

Alerting at scale:- Shift from per-entity alerts to SLO-based and aggregated alerts: - Define SLOs and burn-rate alerts (error rate, latency p99) per service and per region. - Use anomaly detection on aggregated series (e.g., change-point detection or streaming z-score) and burst detection on error counts.- Alert deduplication & routing: group alerts by cause (root metric, host group), suppress flapping with adaptive cooldowns, and throttle noisy alerts.- Tactical alerts: page on high-severity SLO breaches; send tickets/Slack for degraded service that isn’t critical.

Dashboard design & operator workflow:- Multi-layer dashboards: 1. Executive health: single-pane with SLOs, throughput, active incidents. 2. Service overview: key KPIs (p50/p95/p99 latency, error rate, connections), regional breakdown, trend lines. 3. Drilldown panels: instance/resource maps, top-K error sources, traces sample list.- Use cards with summary + one-click drilldown to correlated logs/traces; provide pre-built runbooks and suggested next steps on dashboards.- Reduce noise: default dashboards show aggregated views; enable filters for deep diagnostics (top 10 tags). Use anomaly highlights and changepoints to guide attention.- UX: fast links from alert → dashboard → filtered trace/log view.

Operational considerations & trade-offs:- Trade storage vs fidelity: heavy rollups and sampled traces save cost but delay root-cause for rare bugs—mitigate with adaptive sampling on anomalies.- Start with conservative label policy and evolve via telemetry reviews; provide developer tooling to flag invalid labels early.- Invest in correlation metadata (trace IDs in logs, metrics with trace hints) to reduce time-to-resolution.

Example numbers:- Sampling: 0.5% traces normal → auto-scale to 5% if error rate doubles for 10m.- Retention: 10s resolution 7 days, 1m for 90 days, 15m for 365 days.

This strategy balances signal fidelity, cost, and operator effectiveness as the service scales from thousands to millions of concurrent connections.

Performance Monitoring & Observability Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank