Driving Results and Customer Impact Questions
Stories where you improved infrastructure reliability, performance, or user experience. Show quantified results when possible: 'Improved backup recovery time from 4 hours to 30 minutes', 'Reduced manual operations by 70% through automation', 'Eliminated single point of failure impacting 500 users'. Show how you identified the problem, proposed solution, and delivered impact.
HardSystem Design
81 practiced
Design an observability pipeline that ingests 1 billion metrics/day and 10 TB logs/day, supports 30-day metric retention and 90-day log retention, and provides alerting with sub-minute latency while minimizing cost. Discuss choices for ingestion, aggregation, storage formats (TSDB vs columnar), indexing strategies, sampling, and query performance trade-offs.
Sample Answer
Requirements & constraints (clarify): ingest 1B metrics/day (~11.6k metrics/sec) and 10 TB logs/day, metric retention 30d, logs 90d, sub‑minute alerting, minimize cost.High-level architecture:- Ingest: agents (Prometheus exporters/OTel) -> Kafka (durable buffer, partition by service/tenant) -> stream processors (Flink/KSQ/Go consumers) -> hot path + cold path.- Hot path (real‑time alerts/sub‑minute): stream processors compute rollups, run alert rules, write high‑ingest TSDB (Cortex/Thanos or M3) with short write latency.- Cold path (cost): raw logs and metrics batch to object storage (S3) as compressed columnar (Parquet/ORC) and time-series chunks for long‑term queries. Use schema + partitions: date/tenant/service.Ingestion & aggregation:- Metrics: ingest raw, immediately run rollups (1m/5m) and pre-aggregate high‑cardinality series. Use pushdown aggregation in stream processors to reduce cardinality. Strict label cardinality limits and tag normalization at ingestion.- Logs: structured ingestion (JSON) -> extract key fields; push into Kafka then to two stores: Elasticsearch/Opensearch or a lightweight indexer (Loki/ClickHouse) for recent 7–14 days, and object storage for cold.Storage formats:- TSDB for metrics: use a purpose-built TSDB (Cortex/M3/Prometheus remote) optimized for fast append and queries; store raw high‑resolution for 7 days, downsampled (1m) for remaining 23 days stored as separate series.- Columnar for analytics/cost: Parquet on S3 for both metrics (aggregates) and logs (batchified). Columnar gives high compression and cheap storage for long retention.Indexing strategies:- Logs: keep lightweight inverted index for recent hot window (fields: tenant, service, level, trace_id). For cold logs rely on partition pruning + min/max column statistics in Parquet; avoid full-text indexing for cold tier.- Metrics: index by metric name + hashed label set; maintain series metadata store with TTL eviction. Use bloom filters and time-partitioned blocks to speed reads.Sampling & cardinality control:- Apply client-side and server-side sampling for high-volume low-value metrics and noisy logs (probabilistic sampling for debug-level). Use dynamic sampling: threshold-based or reservoir sampling per series.- Enforce label cardinality limits, normalize labels, and use histograms/summaries instead of many unique labels.Query & alerting trade-offs:- Sub-minute alerts run against hot TSDB and stream processors (no cold reads). Keep alert evaluation stateful in stream processors to avoid TSDB load.- Ad‑hoc queries hitting cold storage are higher latency (seconds–minutes); optimize via cached materialized views for common queries and use query federation (hot index + cold Parquet).- Trade-off: more aggressive downsampling and sampling reduces cost but loses fidelity for troubleshooting. Keep raw logs/metrics in cold storage for forensic, but accept slower retrieval.Cost & scaling:- Tiered storage: hot NVMe/EBS for 7–14 days, warm object storage for remainder. Use auto-scaling ingestion consumers. Compress data (Snappy/Zstd), use Parquet columnar for cold to save egress/IO.- Use multi-tenant quotas & budgets, alert on cost anomalies.Operational concerns:- SLOs: alert latency <60s, ingestion durability SLA, query SLAs for hot/cold tiers.- Backfill: rebuild indexes from cold Parquet as needed.- Observability of pipeline: instrument ingestion, lag metrics, consumer lag, error rates.Summary trade-offs:- Prioritize stream processing + TSDB for low-latency alerts; use columnar object storage for cost‑efficient retention and analytics. Control cardinality and sample aggressively to reduce cost while preserving forensic cold copies.
EasyTechnical
81 practiced
Write a Python script that reads a large server log file (format: timestamp level component message) and streams it to compute the top 10 components by ERROR count without loading the whole file into memory. Include explanation about time and memory complexity and how you'd run or containerize this in production to process logs daily.
Sample Answer
Approach: stream the file line-by-line, parse fields, maintain a fixed-size counter for ERROR counts per component (collections.Counter). At the end (or periodically), output top 10. This uses O(k) memory where k = number of distinct components (usually small) and never loads the whole file.Key points:- Time: O(n) where n = number of lines (single pass).- Space: O(k) where k = distinct components (usually << n).- Edge cases: malformed lines (skipped), different timestamp formats, huge cardinality of components (consider eviction/LFU or approximate counting like Count-Min Sketch).Production / containerization:- Wrap script into a small image (python:3.11-slim), COPY script, set ENTRYPOINT.- Run daily via cronjob or Kubernetes CronJob mounting log storage (e.g., hostPath, EFS) or ingest logs from a central system (Fluentd/Fluent Bit -> Kafka/S3) and process per-shard.- For scale, stream from log storage (S3/Kafka) and parallelize by time ranges; export results to monitoring (Prometheus pushgateway) or a dashboard (Elasticsearch/Kibana, Grafana). Add retries, structured logging, metrics, and alerting if ERROR spikes.
python
#!/usr/bin/env python3
import sys
from collections import Counter
def stream_top_errors(path, top_n=10):
counts = Counter()
with open(path, 'r', encoding='utf-8') as f:
for line in f:
# expected format: "timestamp level component message"
parts = line.strip().split(None, 3)
if len(parts) < 3:
continue
_, level, component = parts[:3]
if level.upper() == 'ERROR':
counts[component] += 1
# most_common returns ordered list
return counts.most_common(top_n)
if __name__ == '__main__':
if len(sys.argv) != 2:
print("Usage: top_errors.py /path/to/logfile")
sys.exit(2)
results = stream_top_errors(sys.argv[1])
for comp, cnt in results:
print(f"{comp}\t{cnt}")HardTechnical
71 practiced
A JVM service is experiencing GC-induced latency spikes that manifest at P99 under moderate load. Provide a detailed diagnostic and mitigation plan: which JVM flags and GC collectors you would consider tuning, how you'd collect heap and GC metrics, trade-offs of resizing heap vs changing collector, and alternative approaches like moving cache off-heap or improving object allocation patterns.
Sample Answer
Start with goals and scope: reduce P99 GC pauses under moderate load without regressing throughput or memory safety. Split into Diagnosis and Mitigation.Diagnosis (data first)- Collect GC logs (Use -Xlog:gc* for Java 11+; Java 8: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/path/gc.log). Include metadata: PID, build, container limits.- Gather JVM metrics (jcmd GC.class_histogram, jmap -heap, JFR recordings) and process-level metrics: CPU, RSS, container memory/cgroup limits, swap, page faults.- Monitor application metrics aligned with traces: request latency histograms, throughput, heap usage, allocation rate, tenuring distribution, promotion failures, and safepoint time.- Correlate pauses to traces (OpenTelemetry/Jaeger) and OS events (dstat, perf) to ensure GC is root cause.Key GC knobs & collectors to try- Start by inspecting current collector. For low-latency P99 consider: - G1GC (default on modern JDKs): tune -XX:MaxGCPauseMillis, -XX:InitiatingHeapOccupancyPercent, -XX:ConcGCThreads, -XX:ParallelGCThreads, and set -XX:MaxGCPauseMillis=200→100ms iteratively. - ZGC or Shenandoah (if using JDK 11+ builds that support them): near-zero pause collectors for large heaps; enable when allocation rate and footprint justify. - If using Parallel/ CMS, consider migration: CMS deprecated; switch to G1 or ZGC.- Heap sizing flags: -Xmx, -Xms (set equal for predictable GC behavior), -XX:NewRatio or -XX:NewSize/-XX:MaxNewSize for young gen tuning.- Survivor/tenuring: -XX:MaxTenuringThreshold, -XX:SurvivorRatio to reduce premature promotion.- Ergonomics: disable biased locking if skewed.Metrics to collect continuously- GC pause durations and counts, allocation rate (bytes/sec), promotion rate, full GC frequency.- Heap occupancy by region/generation, Eden/S0/S1/Old sizes.- Safepoint time and GC concurrency fractions.- JFR/async-profiler snapshots during spikes for allocation hotspots and lock contention.Mitigations (short-term to long-term)- Short term: - Increase heap slightly to reduce frequency of full GCs (trade-off: larger heap → longer full-GC pause in non-concurrent collectors; with concurrent collectors larger heap often helps). - Set Xms = Xmx to avoid resizing-induced pauses. - Tune G1 to have more concurrent threads: -XX:ConcGCThreads, -XX:ParallelGCThreads and adjust region size (-XX:G1HeapRegionSize). - Reduce MaxGCPauseMillis target gradually; measure effect.- Medium term: - Move large caches off-heap (e.g., Caffeine with off-heap or native caches, Redis/Memcached). Off-heap reduces live set and allocation pressure; costs: serialization, complexity, potential native memory leaks. - Use object pooling cautiously — only for high-allocation churn objects where pooling reduces GC cost without increasing complexity. - Reduce allocation rate in hot paths: reuse buffers (e.g., Netty ByteBuf pooled allocator), avoid temporary objects in tight loops, use primitive arrays.- Long term: - Migrate to ZGC/Shenandoah if application needs huge heap with low pause SLAs; trade-offs include availability of JDK version, slightly higher CPU and immature tuning. - Redesign memory-heavy components (caches, large graphs) into external stores or sharded services.Trade-offs: resizing heap vs changing collector- Increasing heap: reduces GC frequency but increases pause duration for stop-the-world full GCs on non-concurrent collectors; with concurrent collectors, it increases GC work but often lowers pause frequency. Larger heap also increases memory pressure on host/container.- Changing collector: concurrent collectors (G1/ZGC/Shenandoah) reduce pause time but may increase CPU and overall throughput variance; they require testing under realistic load and often newer JDKs.Operational checklist- Add alerting on GC pause > threshold, allocation rate spikes, and full GC frequency.- Automate nightly/CI perf tests that run allocation-heavy scenarios and collect GC/JFR.- Capture heapdumps on OOM or prolonged high-old-gen usage.- Run targeted allocation profiling during spike windows (async-profiler, JFR).Example tactical sequence1) Enable verbose GC logging + JFR, correlate with traces.2) Set Xms=Xmx to current comfortable Xmx; increase by 10-20% if old gen regularly near max.3) If using Parallel/CMS, move to G1 with conservative MaxGCPauseMillis and tune InitiatingHeapOccupancyPercent.4) If P99 still unacceptable and heap large, evaluate ZGC/Shenandoah in canary.5) Parallel effort to move caches off-heap or externalize, and reduce allocation hotspots.This plan balances immediate fixes, measurement-driven tuning, and architectural changes to sustainably reduce P99 GC latency.
HardTechnical
67 practiced
Your company must reduce infrastructure spend by 30% without degrading availability or customer experience. Propose a phased plan that includes rightsizing, using spot instances, storage tiering, cache optimization, and SLO-driven cost allocation. Explain safeguards and metrics you will monitor to ensure minimal customer impact during cost reductions.
Sample Answer
Approach: treat this as a risk-managed program: measure baseline, pilot changes in low-risk workloads, iterate and scale while using SLOs/error budgets to guard customer impact.Phased plan1) Discover & baseline (2–4 weeks)- Inventory compute/storage/cache per service, traffic patterns, and SLIs (p99 latency, error rate, availability, cache hit ratio).- Tag resources for cost attribution.- Establish SLOs and error budgets (e.g., 99.95% uptime, p99 latency thresholds).2) Rightsizing pilot (2–6 weeks)- Use telemetry (CPU, mem, heap, GC, queue lengths) and percentile utilization to produce rightsizing recommendations.- Pilot on non-critical services: reduce instance sizes/replica counts where 95th-percentile utilization << capacity.- Safeguards: automated canary rollback, automatic scale-out policies, retain 10–20% reserve headroom; run load tests; monitor error budget burn.3) Introduce Spot Instances (4–8 weeks)- Move stateless, fault-tolerant workers & batch jobs to spot pools with diversified AZs and instance types; keep control-plane and critical frontends on on-demand.- Safeguards: graceful shutdown handlers, checkpointing, fallback to on-demand autoscaling when spot interruption rate or job retry failures spike; ring-fenced capacity group of on-demand to maintain headroom equal to worst-case spot eviction rate.4) Storage tiering & lifecycle policies (3–6 weeks)- Classify data by access patterns; move cold objects to cheaper tiers (infrequent/archival), compress/compact logs, set lifecycle rules for snapshots.- Safeguards: testing for restore times, SLA for RPO/RTO per data class, cache warm-up for recently downgraded tiers.5) Cache optimization (2–6 weeks concurrently)- Improve TTLs, size LRU caches based on working set, offload heavy DB reads to read-replicas, tune eviction policies, introduce multi-layer caches (edge + in-memory).- Safeguards: rate-limits and circuit-breakers between cache miss storms and backing stores; staged TTL increases with monitoring of origin load.6) SLO-driven cost allocation & automation (ongoing)- Chargeback showback: allocate cost to services weighted by their error-budget impact and traffic; prioritize optimization where cost per error-budget saved is lowest.- Automate rightsizing/spot decisions via policy engine that respects SLOs and explicit guardrails.Safeguards & rollout controls- Canary & progressive rollout (5% → 25% → 100%) with automated rollback triggers tied to SLO violations.- Maintain error-budget thresholds: block further cost-saving changes for a service if error budget burn > threshold.- Keep a minimum on-demand capacity pool equal to expected peak demand for 24–48 hours.- Multi-AZ / multi-instance-type diversification for spot use; checkpointing and graceful drain for stateful processes.Metrics & dashboards to monitor (real-time + alerting)- Customer-facing SLIs: p50/p95/p99 latency, error rate, availability (SLO compliance).- Error budget burn-rate per service.- Infrastructure metrics: CPU/memory/IO utilization percentiles, autoscale events, queue depths.- Cache: hit ratio, miss latency, eviction rate.- Storage: IOPS, egress/restore latency, cost per GB by tier.- Spot: interruption rate, time-to-replace, fallback activation count.- Business metrics: cost per active user, cost per request, spend vs target by team.Expected outcome & cadence- Immediate 5–10% savings from rightsizing and storage tiering in pilots; 10–25% additional from spot adoption and cache optimization over 3–6 months. Continuous review quarterly, with runbooked rollback and an SLO-first gating process to ensure no degradation of customer experience.
EasyTechnical
87 practiced
Your team backlog contains 200 tickets mixing reliability work, feature requests, and technical debt. Explain a framework you would use to prioritize reliability work that has direct customer impact. Describe metrics and decision criteria (for example, SLO burn rate, user-facing incidents, ROI), how you would present recommendations to product managers, and how you would measure success after prioritization.
Sample Answer
Situation: Our backlog has ~200 mixed tickets; I must surface and prioritize reliability work with direct customer impact while aligning with product priorities.Framework I’d use:- Triage & categorize: label tickets as Reliability (user-facing), Reliability (internal), Feature, Tech debt.- Scoring matrix for reliability tickets with weighted criteria: - SLO burn rate (30%): sustained burn > 2x baseline in 24h scores high. - User-facing incidents (25%): number of unique customers affected / outage duration. - Severity & frequency (15%): P0/P1, recurrence rate. - ROI / cost of failure (15%): estimated lost revenue, support cost, churn risk. - Effort & risk (15%): estimated engineer-days and deployment risk.- Compute a priority score = weighted sum; flag tickets above threshold for immediate action.Metrics & decision criteria:- Primary: SLO burn rate, number of user-facing incidents, user error rate, customer-reported tickets.- Secondary: MTTR, deployment risk, estimated engineering effort, business impact ($/hour).- Thresholds: e.g., if SLO burn rate >2x or >1 customer-impacting P1 in 7 days → high priority.How I’d present recommendations to PMs:- One-page summary per high-priority item: problem statement, affected customers, metric evidence (graphs of SLO burn / incidents), proposed solution, effort estimate, expected impact (SLO improvement, reduced incidents, cost avoided), and trade-offs.- Prioritized list (top 10) with an “error budget” impact view: show remaining error budget and how each fix consumes or restores it.- Use a dashboard demo (SLOs, burn charts, incident map) to make the case visually and actionably.- Propose a short runway: which items to schedule this sprint vs. next quarter, and offer alternatives (quick mitigation vs. long-term fix).Measuring success after prioritization:- Leading indicators: reduction in SLO burn rate, fewer customer-facing incidents in target window (30/60/90 days), improved MTTR.- Business outcomes: decreased support tickets, reduced revenue impact estimates, improved NPS or customer complaints for affected features.- Process metrics: % of prioritized reliability items completed on time, reduction in repeat incidents for same root cause.- Follow-up: run blameless post-implementation checks, compare before/after dashboards, and iterate scoring weights based on observed impact.This approach balances quantitative evidence, business impact, and engineering cost to align SRE work with customer and product priorities.
Unlock Full Question Bank
Get access to hundreds of Driving Results and Customer Impact interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.