Performance Engineering and Cost Optimization Questions
Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.
MediumTechnical
74 practiced
A mobile client tolerates some staleness. Propose a design to deliver a news feed that balances freshness and cost. Include caching, background refresh, prefetching, and how you would measure the marginal UX improvement vs incremental cost.
Sample Answer
Requirements & constraints (clarify):- Mobile clients can accept bounded staleness (e.g., up to 2–5 minutes for top stories, longer for older items).- Goal: maximize perceived freshness and engagement while minimizing backend cost (API, DB, bandwidth, push).High-level design:- Client-side L1 cache: in-memory + on-disk with TTL per feed-section (e.g., TopStories TTL=60s, ForYou TTL=300s).- Edge cache (CDN + API gateway L2): cache feed payloads keyed by user-segment + window; short TTLs & stale-while-revalidate.- Backend: real-time event pipeline updates article timestamps and ranking; materialized feed snapshots per segment updated asynchronously.Background refresh:- On app foreground: show cached feed immediately, then trigger a conditional refresh (If-Modified-Since / ETag) to fetch deltas.- Periodic background refresh (OS-allowed): low-frequency for active users (e.g., every 2–5 min) with exponential backoff if inactive; use silent push (APNs/FCM) to notify of high-priority content.Prefetching:- Predictive prefetch for next-screen content (article body, images) when on Wi-Fi or charging and user historically taps > threshold; prefetch only high-probability items (top N).- Network-aware rules: disable prefetch on cellular or low battery; throttle concurrency.Cost/freshness optimization mechanisms:- Delta APIs (only changed IDs + metadata) to reduce payload size.- Bloom-filter/last-seen cursors to avoid resending same items.- Ranker that degrades refresh frequency for low-value segments (long-tail topics).Measuring marginal UX improvement vs incremental cost:- Metrics: - UX: time-to-freshness (median seconds to show < TTL content), feed engagement (CTR, dwell time), retention, NPS. - Cost: API requests per user, bandwidth per user, CDN hit-rate, backend CPU & DB ops, push volume.- Experimentation: - Run A/B/n tests varying TTLs, background frequency, and prefetch aggressiveness. For each variant measure delta_UX and delta_cost. - Compute marginal benefit: marginal UX improvement per unit cost = Δ(engagement metric) / Δ(cost) (e.g., additional clicks per $1000). - Use uplift curves: plot UX metric vs cost to find elbow (diminishing returns). Pick operating point where incremental cost yields acceptable ROI (set by business).- Example decision rule: if increasing refresh rate from 5min→1min increases CTR by 2% but doubles API cost, compute revenue/benefit per click; accept only if revenue_per_click * delta_clicks > incremental_cost.Trade-offs & resilience:- Conservative defaults to protect battery and network; allow power-user opt-in for aggressive freshness.- Graceful degradation: if backend strained, increase TTLs and serve “stale-while-revalidate” content with visual freshness indicators.This design balances immediate responsiveness with cost via caching layers, conditional and background refresh, predictive prefetch, and systematic A/B measurement to choose the freshness-to-cost operating point.
HardTechnical
48 practiced
Design storage layout for a time-series metrics system ingesting 5k series at 1k samples/sec each, retention 2 years, and requirement that queries for last 24 hours return under 1 second. Discuss compression, downsampling, partitioning, and expected storage cost trade-offs.
Sample Answer
Requirements recap: ingest 5,000 series × 1,000 samples/sec = 5M samples/sec; retention = 2 years; interactive queries for “last 24h” must return < 1s. Key constraints: extremely high ingest, large retention window, low-latency recent queries.High-level design- Use a distributed time-series storage (append-optimized TSDB) with hot (recent) and cold (long-term) tiers.- Shard by time + series hash: create time-based buckets (e.g., 1h shards) and hash series id across nodes within each time shard to balance writes.- Keep recent 24–72 hours in hot tier (SSD/NVMe + in-memory index). Older data moved to cold tier (object storage / HDD) with compacted files (columnar / block format).Compression- Use time-series encodings (Gorilla-style XOR for values, delta/delta for timestamps) achieving 5–10× on floats and timestamps.- Store blocks (e.g., 64KB - 1MB) with per-block min/max and bloom/bitmap indexes to enable skip scans.- Compress blocks with fast codecs (LZ4/Zstd-1) for fast decompression during queries.Downsampling & retention policy- Keep raw data for short period (e.g., 7–30 days) in hot/cold raw form for high-fidelity analysis.- Create downsampled rollups (1m, 5m, 1h aggregates) stored for entire 2-year retention. Aggregates include count, sum, min, max, maybe stddev.- Use deterministic hierarchical rollups produced in streaming (e.g., in ingest pipeline) to avoid reprocessing.Partitioning & indexing for low-latency 24h queries- Partition by time shard (hour) to limit files scanned to last 24 shards (or fewer if larger shards).- Within shards, use series-level partitioning (series hash) so a single series query goes to one node/shard.- Maintain an in-memory/SSD index for the hot 24h window: per-series pointers to blocks in time order plus bloom filters to avoid IO.- Support query fan-in: parallel scan of the 24h shards across nodes; use block-level pruning via time-range and value indexes to keep latency <1s.Storage & cost calculation (example)- Assume raw sample ~16 bytes (8B timestamp, 8B float). Ingest = 5M *16B = 80MB/s ≈ 6.9TB/day raw.- With 8× compression (Gorilla + blocking) → ~0.86TB/day.- Raw retention 30 days → 25.8TB. Remaining 23 months stored as 1m aggregates: 5k series × (60*24*365*2 / 1) ≈ 5k * 1,051,200 ≈ 5.26B points; if aggregate stored as 24B → ~126GB per year? (note: recalc carefully for client). Practical estimate: downsampling reduces multi-year cost by 10–60×.- Cold tier on S3/Blob + lifecycle policies yields lowest $/GB; hot SSD cluster sized to hold 24–72h compressed raw + indexes (e.g., few TBs).Trade-offs- More raw retention => higher fidelity but much higher cost and larger hot storage; affects query latency if not well indexed.- Higher compression ratio lowers storage cost but increases CPU and decompression latency; choose fast codecs and smaller block sizes for low-latency reads.- Aggressive downsampling reduces storage dramatically and simplifies long-term queries, but loses granularity and prevents some analyses.- Finer time-shards improve parallelism and compaction but increase metadata and management overhead.Operational notes- Autoscale ingestion nodes and shard placement; use backpressure and batching.- Continuous background compaction to merge small blocks and build rollups.- Monitor query P95 and IO; keep the last-24h index memory budget to meet <1s SLAs.This design balances cost and latency by keeping recent data hot and indexed for sub-second queries, using time-series-specific compression, and moving to downsampled cold storage for long retention.
HardSystem Design
58 practiced
Design a workload-shaping system to gracefully handle a 100x traffic surge from an API while preserving high-value traffic. Include admission control, priority queues, token-bucket limits, backpressure mechanisms, and how you would surface degraded behavior to customers.
Sample Answer
Requirements & constraints:- Handle sudden 100x API traffic surge while preserving high-value traffic (paid customers, SLA calls).- Low latency for priority traffic, graceful degradation for best-effort.- Observable, reversible, and enforceable controls; minimal code changes to existing services.High-level architecture:Client -> Ingress Load Balancer -> Admission Controller (edge) -> Priority Broker -> Service Pool (workers) -> Downstream servicesAux: Token service, Rate DB, Telemetry & Customer-facing status APICore components & responsibilities:1. Edge Admission Controller- Classifies requests (auth token, API key, header flags) into priority classes: P0 (critical), P1 (paid), P2 (free).- Enforces global concurrency cap and rejects/queues extra requests with HTTP 429 + Retry-After.2. Priority Broker + Queues- Per-priority in-memory queues (bounded). Strict priority scheduling with configurable shares (weighted fair queuing).- Dequeue workers pull from queues respecting weights so P0 always outruns lower classes.3. Token-bucket limits- Global + per-customer token buckets stored in fast datastore (Redis). Tokens replenish per policy (r/s) and burst capacity.- Admission checks both priority class and token availability; if tokens exhausted, respond with 429 or demote to lower priority queue (soft degradation).4. Backpressure & degradation- If service pool saturates, broker slows intake, returns 503/429 with informative error indicating degraded mode and estimated retry.- Graceful feature-dropping: for P2 requests, offer degraded responses (cached, stale data, simplified payload) using flags returned in headers.5. Surface degraded behavior- Customer-visible headers: X-Service-Mode, X-Rate-Limit-Remaining, X-Estimated-Retry.- Real-time status page + per-customer telemetry in dashboard showing throttling reason and usage.- Webhooks/alerts for high-value customers when their requests are being throttled.Data flow & scaling:- Admission and token checks are O(1) via Redis caches replicated per edge region. Queues scale horizontally; workers auto-scale based on queue depth and SLOs.- Use circuit breakers to drop downstream calls when latency/ER increases.Trade-offs:- Strong guarantees require stateful coordination (Redis) vs fully stateless edges — chosen for precise per-customer limits.- Soft-degradation increases complexity (fall-back responses, caching) but preserves experience.Observability & ops:- Instrument queue depths, token-bucket drains, rejects by reason.- Run chaos tests simulating 100x surge; tune weights and burst sizes.- Provide runbook and customer communication templates for degraded operation.
MediumTechnical
54 practiced
For a read-heavy relational database workload, outline optimization strategies including indexing, read replicas, materialized views, partitioning, and denormalization. Provide criteria for choosing among these options.
Sample Answer
Start by profiling: measure query patterns (most frequent SELECTs), read latency, cardinality, hot rows, write rate, and acceptable staleness. Then consider these optimizations with trade-offs and selection criteria.Indexing- Use B-tree for range/PK lookups, hash or inverted indexes for exact matches, and bitmap for low-cardinality analytics.- Create covering indexes (include columns in index) to eliminate lookups.- Pros: low latency, low operational cost. Cons: write overhead, storage.- Choose when queries filter/sort on stable columns and write rate is moderate.Read replicas- Offload reads to async replicas or use read-only replicas per region.- Pros: linear read scale, geo-local latency reduction. Cons: eventual consistency, replication lag, extra cost.- Choose when read volume >> write volume and some staleness is acceptable.Materialized views- Precompute expensive joins/aggregations; refresh on schedule or on commit.- Pros: fast complex query responses. Cons: storage, refresh cost, complexity.- Choose when queries are CPU/IO-bound and results can tolerate refresh latency.Partitioning- Range or hash partition to enable pruning, faster full-table scans, and easier maintenance (archival).- Pros: improves scan/query and maintenance for large tables. Cons: requires careful key design; some queries may still hit many partitions.- Choose when table size is very large and queries target specific key ranges.Denormalization- Duplicate fields or pre-join data to avoid joins for hot read paths.- Pros: minimal read latency. Cons: update anomalies, higher storage, complex write flows.- Choose when read latency is critical and update patterns are manageable.Decision criteria (summary)- Latency sensitivity vs acceptable staleness- Read/write ratio and write throughput- Query complexity (heavy joins/aggregates) vs simple lookups- Data size and hot partitions- Operational complexity and cost constraintsCombine techniques: e.g., partitioned tables + covering indexes + read replicas, or materialized views for aggregates with replicas for scaled reads. Monitor metrics and iterate.
HardSystem Design
87 practiced
Design an observability and A/B performance experimentation platform that allows safe testing of cost-saving optimizations. Describe how you would capture metrics, create cohorts, run statistical tests on latency and error metrics, and attribute cost changes to experiments.
Sample Answer
Requirements & constraints:- Safely run experiments that may reduce infra/app cost while preserving SLOs (latency, error)- Real-time observability, per-cohort metrics, automated statistical testing, cost attribution- Support rollouts, feature flags, rollback, multi-tenant scale, low overhead on productionHigh-level architecture:Client services → SDK/sidecar (metrics + tracing + decisioning) → Ingest + Feature-Flag/Experiment API → Streaming pipeline (Kafka) → Metrics store (Prometheus/ClickHouse), Trace store (Jaeger), Aggregation/feature-store → Experimentation engine + Stats service → UI & Alerts → Cost attribution service → Data warehouse for backfill/finance.Components & responsibilities:1. SDK/sidecar: assign cohort (randomized, stratified), tag traces/metrics with experiment id, sample detailed traces for tail latency.2. Ingest pipeline: high-throughput, enrich events with metadata (tenant, resource tag), dedupe, push to real-time aggregator.3. Aggregation layer: compute per-cohort time-windowed metrics (p95, p99, error rates, QPS, resources consumed — CPU, mem, cloud billing tags).4. Experimentation engine: supports A/A, A/B, multi-armed bandits; enforces guardrails (SLO thresholds), calculates statistical tests, and handles ramping & rollbacks.5. Stats service: implements sequential testing (e.g., sequential probability ratio test, Bayesian A/B with credible intervals) for latency distributions (use nonparametric tests like Mann-Whitney or bootstrap for p90/p95) and proportion tests for errors (Bayesian Beta or frequentist chi-sq with adjustment).6. Cost attribution: map resource usage to experiments via tags; compute delta cost = baseline cohort cost - experiment cohort cost normalized by traffic; use regression to control for confounders (traffic, time-of-day).7. UI & Alerts: experiment dashboard, cohort comparison, significance, cost impact estimate, auto-rollback triggers when SLO degradation exceeds threshold.Data flow & metrics capture:- SDK emits event stream: {trace_id, experiment_id, cohort, latency_ms, status, resource_tags, timestamp}- Aggregator computes per-minute aggregates and distributions, stores raw samples for tail analysis.- Billing collector ingests cloud billing (by tag) and maps to experiment_id via resource_tags.Statistical testing & safety:- Use pre-experiment power analysis to size cohorts for detecting minimal detectable effect on p95 and error rate.- For latency: test distributions with bootstrapped CI on percentiles; use sequential/bayesian methods to allow early stopping while controlling Type I error.- For error rates: use Bayesian Beta-binomial or G-test with correction.- Multi-metric decisioning: define loss function combining SLO violations and cost delta (e.g., cost savings weighted against p95 increase). Only approve experiments with >X% cost savings and <Y% SLO regression with high posterior probability.- Rollout policy: gradual ramp (1%→5%→20%→100%) with automated checks at each step.Attribution & reporting:- Attribution = normalized cost delta per 1k requests, plus projected annualized savings. Use regression to remove noise (traffic/cycle effects). Provide per-tenant breakdown for chargeback.- Store experiment audit log and reproducible report for compliance.Scalability & trade-offs:- Streaming + columnar store for high-cardinality metrics; sample traces for storage cost control.- Trade-offs: stricter sampling reduces observability; Bayesian sequential testing reduces sample needs but requires priors.- Security/compliance: PII stripping in SDK, RBAC for experiments.This design ensures safe, measurable cost-optimization experiments with robust statistical guarantees, end-to-end attribution, and automated safety controls for production rollouts.
Unlock Full Question Bank
Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.