Performance Engineering and Cost Optimization Questions

Engineering practices and trade offs for meeting performance objectives while controlling operational cost. Topics include setting latency and throughput targets and latency budgets; benchmarking profiling and tuning across application database and infrastructure layers; memory compute serialization and batching optimizations; asynchronous processing and workload shaping; capacity estimation and right sizing for compute and storage to reduce cost; understanding cost drivers in cloud environments including network egress and storage tiering; trade offs between real time and batch processing; and monitoring to detect and prevent performance regressions. Candidates should describe measurement driven approaches to optimization and be able to justify trade offs between cost complexity and user experience.

MediumTechnical

46 practiced

Design an incremental backfill strategy for a partitioned table that minimizes reprocessing and compute costs. Include steps for identifying affected partitions, writing idempotent jobs, checkpointing, and verifying correctness after the backfill completes. Explain trade-offs between parallelism and cluster cost.

Sample Answer

Requirements & constraints:- Incremental backfill a partitioned table (date partitions) minimizing reprocess and compute.- Safe to resume, idempotent, verifiable, cost-aware.

Design overview (steps):1. Identify affected partitions- Use source/system of truth change-log or run a diff: compare last-successful-run partition list + source metadata (max event timestamp per partition) to find partitions with new/changed data.- If no change-log, use checksums/row-counts per partition to detect differences.

2. Create idempotent jobs- Implement processing so re-running a partition produces same result: write to temporary staging (e.g., table_staging/partition=...), validate, then atomically swap/merge into target using transactional operations (INSERT OVERWRITE for Hive/Delta MERGE).- Include deterministic sorting/aggregations; avoid offsets that depend on previous runs.

3. Checkpointing & orchestration- Track per-partition state in a durable metadata table: partition, status (pending/running/success/failure), attempt_count, started_at, finished_at, checksum.- On job start set status=running; on success set success + store output checksum and row_count.- Use retries with exponential backoff; avoid concurrent runs for same partition by optimistic locking (update where status=pending).

4. Execution plan & parallelism vs cost- Batch partitions into groups sized to fit cluster resources. Parallelism increases throughput but raises transient cluster cost.- Use autoscaling and spot/preemptible workers to reduce cost; cap max concurrent partitions.- If partitions are uneven, prioritize biggest ones first or use size-based bins.- Trade-offs: high parallelism reduces wall time but increases peak compute + failure blast radius; conservative parallelism reduces cost and simplifies failures but takes longer.

5. Verification after backfill- Per-partition checks: row counts, checksums, key foreign-key referential checks.- End-to-end sample queries comparing pre-backfill snapshots or business metrics.- Run data quality tests (expectations) and alert + rollback or requeue failing partitions.

Example (Spark pseudo):

python

# load source for partition p, write to staging path, compute checksum, then atomic swap/merge

Key trade-offs & best practices:- Prefer idempotent writes + per-partition checkpoints to avoid full reprocess.- Keep partition granularity aligned with data change patterns.- Use monitoring dashboards for progress and cost; start conservative parallelism, then tune.

MediumTechnical

49 practiced

Describe how you would instrument a distributed data pipeline to provide actionable telemetry: what traces, metrics, and logs to capture; how to correlate events across stages; and how to design trace IDs and sampling to keep observability costs low while still catching tail latencies and intermittent failures.

Sample Answer

Start by defining observability goals: detect failures, surface data quality issues, measure latency and throughput, and enable root-cause across stages (ingest → transform → store).

Traces:- Capture a distributed trace per data batch or message (batch_id / message_id as trace root). Instrument key stages (ingest, validate, transform, enrich, write) and external calls (S3, DB, APIs).- Record span tags: dataset, partition, offset, schema version, record counts, bytes, retry attempts, and error codes.- Emit timing for queue wait, processing, backpressure, and I/O.

Metrics:- High-cardinality metrics avoided; use dimensional metrics with bounded labels.- Pipeline-level: throughput (records/sec), success/failure counts, processing latency (p50/p95/p99) per stage, backlog size, watermarks, and resource utilization (CPU/mem/io).- Data-quality metrics: null-rate, schema-mismatch rate, dedup rate, late-arrival count.

Logs:- Structured logs (JSON) with trace_id, span_id, batch_id, dataset, stage, and contextual fields. Include exceptions, validation failures, and schema diffs. Keep verbose logs local/archived; surface aggregated errors centrally.

Correlation:- Propagate a single trace_id (and parent span ids) through messages and metadata headers. Attach batch_id and offsets to any retry or forked work. Use consistent field names across components and inject into storage metadata for retrospective joins.

Trace ID and sampling:- Use deterministic trace_id: e.g., sha256(dataset + batch_id + timestamp) to correlate reruns. Apply adaptive sampling: low-rate full-trace sampling for steady traffic (1-5%), but always sample: - failed traces - traces exceeding latency SLO (tail-sampling) - traces touching critical datasetsImplement tail-based sampling in ingest (collect lightweight bookmarks/metadata for all traces, then request full spans for interesting ones). This keeps storage/cost low while catching intermittent and tail issues.

Retention and cost control:- Aggregate and rollup metrics; downsample traces older than X days; store full traces for recent window. Use alerts on p99 latency, failure spikes, backlog growth, and data-quality thresholds.

This design balances actionable granularity with cost by propagating consistent IDs, prioritizing failures and tails for full traces, and using bounded-dimension metrics and structured logs for fast querying.

HardSystem Design

55 practiced

A global analytics query requires joining datasets stored in different cloud regions; egress costs and query latency are high. Propose architectural alternatives (federated queries, periodic cross-region replication, caching aggregated summaries, moving compute to data) and quantify trade-offs for cost, freshness, and development complexity.

Sample Answer

Requirements & constraints (clarify): need to join datasets across regions for global analytics. Key goals: minimize egress cost, meet query latency SLA (interactive vs batch), keep freshness (real-time vs daily), and limit dev/ops complexity.

Alternatives (with quantified trade-offs):

1) Federated queries (querying remote region on demand)- Cost: high egress per query (example AWS ~ $0.02–$0.09/GB outgoing); cost scales with query frequency & volume.- Latency: high/variable (cross-region RTT + remote scan time) — tens to hundreds of ms per call, often seconds for large scans.- Freshness: realtime (no replication delay).- Dev complexity: low–medium (use built-in engines: BigQuery federated, Athena/Glue, Presto).- When to pick: low-query-volume, need freshest data, prototypes.

2) Periodic cross-region replication (ETL/CDC -> replicate to a central region)- Cost: medium (one-time egress for replication + storage in destination). Example: daily 1 TB replicated -> egress cost ≈ $20–$90/day depending on price — cheaper than repeated federated reads if many queries.- Latency/freshness: configurable — minutes to hours depending on frequency.- Latency for queries: low (local reads).- Dev complexity: medium — build CDC pipelines (Debezium, Dataflow, Glue), handle idempotency, conflict resolution.- When: many queries, acceptable near-real-time staleness.

3) Caching aggregated summaries (precompute joins/aggregates in each region or centrally)- Cost: low (small storage for summaries, compute cost for scheduled jobs). Egress minimal if summaries stored locally.- Freshness: low-latency if frequent refresh (minutes), otherwise coarser (hourly/daily).- Latency: excellent for dashboarding/OLAP.- Dev complexity: medium (design aggregation model, materialized views, invalidation).- When: analytics focus on aggregates, SLAs require low latency, tolerate reduced granularity.

4) Move compute to data (run compute in each region then combine small results)- Cost: low-medium (compute in-region cheap; only small result egress). Example: reduce 1 TB -> 10 MB aggregated results per region, egress negligible.- Latency: low (parallel regional compute, combine step adds small RTT).- Freshness: realtime/near-real-time if streaming enabled.- Dev complexity: high (distributed job orchestration, consistent execution, schema/versioning).- When: large raw datasets, joins are associative/aggregatable, need freshness and low egress.

Trade-off summary (example numeric heuristic for frequent queries):- If queries/day > 1000 and dataset >100s GB: replication or move-compute is cheapest.- If queries are ad-hoc and data must be real-time: federated queries.- If use-cases are aggregate-heavy (dashboards): caching summaries.

Hybrid recommendation: implement move-compute for heavy analytical joins (run regional Spark jobs to produce compact join keys/aggregates), maintain near-real-time CDC replication for critical low-latency lookups, and expose federated queries for ad-hoc deep dives. Start with cost modeling using representative volumes (GB scanned per query * queries/day * egress $/GB) and pilot one approach to validate latency and operational effort.

HardTechnical

43 practiced

Propose a scalable and cost-efficient strategy to deduplicate 10B daily records. Use probabilistic data structures (Bloom filters, HyperLogLog) for filtering candidates, partitioning to localize work, and incremental runs to avoid full reprocessing. Explain false-positive/negative implications and how you'd validate correctness.

Sample Answer

Requirements & constraints:- Deduplicate ~10 billion records/day with bounded latency and low cloud cost.- Prefer approximate pre-filtering, exact final decisions, incremental processing, and horizontal scale.

High-level strategy:1. Partitioning: shard by stable key (e.g., user_id % N or date+hash) to localize state, enable parallelism, and keep per-partition working sets small enough to fit in memory/cache on workers.2. Two-stage filtering: - Probabilistic fast-pass: maintain a distributed Bloom filter per partition (or per-day per-partition) to quickly reject obvious new records. Configure size and hash functions to target a low false-positive rate (e.g., 0.1–0.5%) while keeping memory small. - Cardinality tracking: use HyperLogLog per partition/day to estimate uniqueness and guide resource allocation (e.g., increase workers where HLL shows high unique counts).3. Exact verification: candidates that pass Bloom check undergo exact dedupe using a local key-value store (e.g., RocksDB or Redis on worker, or partitioned DB) storing canonical record IDs or fingerprints. Use strong stable fingerprint (SHA-256 truncated) to avoid collisions.4. Incremental runs & retention: - Maintain per-partition state with TTLs. Run incremental jobs that only process new batches and query/update per-partition state. Periodically compact state (merge fingerprints) and rotate Bloom filters daily to bound memory. - For late-arriving data, keep a write-ahead log and run reprocessing windows.

False positives / negatives:- Bloom filters: only false positives (treat new record as seen) — risk is missed unique records. Mitigation: tune Bloom parameters; ensure exact verification step only rejects once both Bloom+exact store claim seen. If Bloom false positive would block exact check, design flow so Bloom only filters out obvious non-candidates; still allow sampled bypass for validation.- HyperLogLog: small relative error (~1–2%) acceptable for provisioning, not for final correctness.- Fingerprint collisions: extremely low but cause false positives. Use sufficiently large hash and, if critical, compare full keys.

Validation & monitoring:- Backfill sample: route a sampled percentage (e.g., 0.1–1%) of Bloom-rejected records to exact dedupe to estimate Bloom FPR in production.- Ground truth runs: periodically run a full exact dedupe on a smaller time window (e.g., 1M records) and compare against probabilistic pipeline to measure both false positives and false negatives.- Metrics: per-partition unique counts (HLL), Bloom FPR estimate, fingerprint collision rate, downstream duplicates observed, processing latency, cost per million records.- Alerts: sudden jump in HLL vs stored uniques, increase in downstream duplicate rates.

Trade-offs:- Memory vs FPR: larger Bloom reduces FPR but costs more memory; partitioning granularity impacts state size and parallelism overhead.- Cost vs strict correctness: probabilistic pre-filters reduce compute but require sampling/validation to control missed uniques.

This design balances throughput, cost, and accuracy: probabilistic structures reduce work by >90% in typical skewed traffic, partitioning enables horizontal scale, and exact verification plus sampling provide correctness guarantees and measurable error bounds.

EasyBehavioral

59 practiced

Behavioral: Tell me about a time you reduced cost for a production data pipeline. Use the STAR format to describe the Situation, Task, Action, and Result. Focus on what measurements you collected to justify changes and how you validated there was no negative impact on correctness or user experience.

Sample Answer

Situation: At my previous company I owned a nightly Spark-based ETL that populated our analytics warehouse. Cloud compute costs had risen 40% year-over-year; this pipeline alone accounted for ~22% of our daily EMR/Dataproc bill because it ran large, underutilized clusters for 3 hours each night.

Task: My goal was to reduce pipeline cost by at least 30% within one quarter without degrading data correctness, freshness, or dashboard SLAs used by product and finance teams.

Action:- Measured baseline: I collected job-level metrics for two weeks — CPU/Executor utilization, shuffle/read/write sizes, task skew, per-stage runtime, cluster idle time, and downstream dashboard latency and error rates.- Identified inefficiencies: small files causing IO overhead, wide transformations causing expensive shuffles, and fixed-size cluster provisioning.- Implemented changes incrementally: 1. Rewrote two transformations to use map-side aggregations and reduceByKey to cut shuffle volume (~45% less shuffle bytes). 2. Introduced file compaction job to merge small files prior to ETL. 3. Switched to autoscaling clusters with spot instances and tuned executor memory/cores based on observed utilization.- Validation plan: for each change I ran A/B runs for 7 nights — kept a control run (current pipeline) and a candidate run (with change). I compared row counts, hash checksums on key tables, metric reconciliation against previous day, and end-to-end dashboard values. I also monitored job failure rates and 95th-percentile dashboard refresh latency.

Result:- Cost: reduced nightly pipeline compute cost by 52%, and overall monthly pipeline spend by ~35%.- Correctness: row counts and table-level checksums matched the control across all A/B runs; automated data quality asserts passed.- UX: dashboard freshness SLA improved (median latency down 18%, 95th percentile down 30%).- Stakeholder impact: Presented results with metrics and signed off by analytics and finance teams; changes rolled into production with monitoring alerts added.

This taught me that measurement-driven, small iterative changes plus A/B validation are key to cutting costs without sacrificing data reliability.

Unlock Full Question Bank

Get access to hundreds of Performance Engineering and Cost Optimization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.