Optimization and Technical Trade Offs Questions

Focuses on evaluating and improving solutions with attention to trade offs between performance, resource usage, simplicity, and reliability. Topics include analyzing time complexity and space complexity, choosing algorithms and data structures with appropriate trade offs, profiling and measuring real bottlenecks, deciding when micro optimizations are worthwhile versus algorithmic changes, and explaining why a less optimal brute force approach may be acceptable in certain contexts. Also cover maintainability versus performance, concurrency and latency trade offs, and cost implications of optimization decisions. Candidates should justify choices with empirical evidence and consider incremental and safe optimization strategies.

EasyTechnical

54 practiced

You inherit a monolithic backend where P95 latency and error rate have gradually increased. List and explain the first five steps you would take to identify, quantify, and prioritize bottlenecks, including both code-level and infrastructure-level checks and whom to involve in each step.

MediumSystem Design

51 practiced

Design an optimization plan for a search backend that handles 10,000 QPS across 500M documents and has a 95th percentile query latency target of <100ms. Describe indexing strategy, sharding, caching, query routing, hardware choices (memory vs SSD), and how you would measure, iterate, and validate improvements. Include the tradeoffs between memory usage, freshness, and cost.

Sample Answer

Requirements & constraints:- 10k QPS, 500M docs, p95 <100ms, acceptable freshness (define SLA: e.g., <5s), budget-conscious.

High-level approach:- Use distributed inverted-index search (like Lucene/Elasticsearch/Solr) with tuned sharding, replicas, aggressive caching, and a coordinator layer for smart routing.

Indexing strategy:- Inverted index with field-level analyzers and doc-values for aggregations.- Nearline writes: write-ahead log → apply to in-memory buffer → flush to SSD segments. Support near-real-time refresh configurable per index (e.g., 1–5s).- Use separate indices for hot vs cold data (time-based or access-based), and store large payloads in a lightweight document store.

Sharding & replication:- Start with N primary shards so per-shard doc count <= ~5–10M (=> 50–100 shards). Reason: smaller shard = lower tail latency.- 1–2 replicas for read throughput and HA; route read QPS across replicas to reduce per-node QPS.- Use shard awareness: colocate related shards to reduce cross-node fanout when possible.

Caching:- Query-result cache for frequent identical queries.- Filter cache / bitmap cache for common filters.- Field-value/doc cache for hot documents.- Tiered cache: L1 in-process (LRU) for microseconds hits, L2 distributed cache (Redis or CDN for static responses).- Cache invalidation tied to refresh window; prefer short TTLs for freshness on hot queries.

Query routing & execution:- Coordinator nodes accept queries, consult query router: - Use query profiling to detect high-fanout queries; rewrite or precompute heavy aggregations. - Fan-out minimization: short-circuit queries to only shards containing relevant data (use routing keys or partitioning). - Reduce network overhead by returning top-K per shard and merging centrally.- Support query rehearsal / plan caching for frequent patterns.

Hardware choices:- Memory vs SSD: - Keep hot inverted indexes / term dictionaries and caches in RAM (memory-first improves p95). - Store postings and large segments on NVMe SSDs with good IOPS; use OS page cache plus warmed Lucene caches.- Node sizing example: 48–128GB RAM, 16–32 vCPU, NVMe 1–4TB; network 25Gbps. More RAM reduces SSD reads but increases cost.

Measuring, iterating, validating:- Instrument: per-shard latency histogram, QPS, CPU, GC, SSD IOPS, cache hit rates, network.- SLOs: p95 latency, error rate, freshness.- Load test: replay production traffic (or synthetic) with realistic distributions; measure tail latencies.- A/B experiments: change one variable (shard size, replica count, cache TTL) and compare p95 and cost.- Profiling: slow-query logs, flame graphs to find hotspots; iterate by index tuning, query rewriting, shard rebalancing.

Trade-offs:- Memory vs cost: more RAM reduces p95 by serving more from memory, but increases cost. Use targeted RAM for hot shards and tiered storage for cold data.- Freshness vs cache effectiveness: aggressive caching increases hit rate/cost savings but delays visibility of new docs. Mitigate with short TTLs for hot endpoints and write-through for critical items.- Shard count vs fanout: more shards reduce per-shard load but increase coordination overhead and merge cost; choose shard size that balances per-shard latency and merge overhead.

Final checklist for rollout:- Define freshness & p95 SLOs- Start with 50–100 primaries, 1–2 replicas; measure per-node QPS- Warm caches and run load tests- Monitor and iterate: tune refresh intervals, cache TTLs, and shard allocation to meet p95 <100ms within budget.

EasyTechnical

83 practiced

List the key metrics, logs, and traces you would collect to justify and measure an optimization decision for a backend service. For each metric, explain why it matters and typical alert thresholds you would use when validating an improvement (e.g., CPU utilization, latency percentiles, GC pause, error rate).

Sample Answer

Situation: When proposing or validating a backend optimization (e.g., CPU/binary upgrade, caching, query rewrite, horizontal scaling), you need objective observability to prove improvement and ensure no regressions.

Key metrics, why they matter, and sample alert/validation thresholds:

- Latency percentiles (P50/P95/P99): primary user-facing metric. Improvements should reduce tail (P95/P99). Validation: P95 decrease ≥ 10% or absolute P99 < target (e.g., <200ms). Alerts: P95 > 2x baseline or P99 > SLA (e.g., 500ms).

- Throughput (requests/sec, ops/sec): ensures capacity. Why: shows if optimization affects capacity. Validation: throughput sustained at same or higher level under load. Alerts: sudden drop >20%.

- Error rate (4xx/5xx, application errors): critical safety check. Why: optimizations must not increase errors. Validation: error rate unchanged or reduced; alert if error rate > 0.5% or increases by >100% relative to baseline.

- CPU utilization and saturation (per core): why: identify CPU-bound regressions or headroom. Validation: same work at lower CPU or more headroom; alert if sustained CPU > 80–90% or load > cores*1.5.

- Memory usage / GC metrics (heap usage, GC pause time, frequency): why: detect memory leaks or higher GC overhead. Validation: lower average heap or fewer GC pauses; alerts when GC pause >100ms or GC time >10% of CPU time.

- Service concurrency/resources (thread pool queues, connection pool utilization): why: reveals queuing/backpressure. Validation: queue lengths decrease; alerts if queue depth > threshold (e.g., > 100) or pool saturation > 90%.

- I/O metrics (disk latency, network RTT, DB query latency): why: optimizations can shift bottlenecks to I/O. Validation: no significant regressions; alerts if DB p95 latency > baseline*2 or disk latency >10ms.

- External dependency success and latency (downstream APIs, DB calls): why: ensures we didn't hide problems by aggressive retries/caching. Alert if dependency error rate > baseline or latency spikes.

- Resource-specific metrics for caches (hit ratio, eviction rate): why: measures effectiveness of caching. Validation: hit ratio increase (e.g., +15%), eviction rate drops; alert hit ratio < expected (e.g., <70%).

- Business/SLI metrics (orders completed, conversions): why: ultimate justification is business impact. Validation: no drop; prefer positive lift. Alerts if key business metric drops >5%.

Logs and traces to collect and why:- Structured request/response logs with request IDs, status codes, latency, user id/context: to correlate regressions and scope incidents.- Error/stack traces with sampling: to debug root causes.- Distributed traces (span duration, service breakdown) for P95/P99 flows: to pinpoint where tail latency lives and verify optimization reduced time in targeted spans.- Resource/GC logs for memory issues.

Tracing/Correlation strategy:- Ensure distributed tracing with sampling skewed toward high latency and errors (always sample errors, 100% for P99-slow requests) so you can inspect problematic flows.

Typical validation process:1. Establish baseline (steady load, representative traffic).2. Apply optimization in a canary with identical load.3. Compare the above metrics over same interval; require no increase in error rate, no new bottleneck exceeding thresholds, and measurable improvement in target metric (e.g., P95 latency -10%).4. Rollback criteria: error rate doubled, P99 exceeds SLA, or critical business metric degrades >5%.

This set gives quantitative signals to justify the change and guardrails to detect regressions.

HardSystem Design

62 practiced

Design a sharding and rebalancing strategy for a global user database that minimizes downtime and cross-shard transactions. Describe shard key selection, resharding techniques (range vs hash vs lookup table), online rebalancing, traffic routing, and how to ensure transactional integrity during moves.

Sample Answer

Requirements & constraints:- Global scale: low-latency reads/writes near users, multi-region failover- Minimize downtime and cross-shard transactions- Strong transactional integrity for user-related operations- Smooth capacity growth and operational simplicity

High-level approach:- Use a hybrid sharding strategy: stable hash-based primary sharding by a high-cardinality, immutable user attribute (user_id UUID or account_id) to minimize cross-shard hotspots, plus a lightweight lookup/placement service to enable flexible mapping and resharding.

Shard key selection:- Primary key: user_id (UUID) hashed (consistent hashing) -> evenly distributes load and avoids range hotspots.- Secondary locality: map user’s preferred region to a regional read-replica for low-latency reads; writes go to home shard determined by consistent hash.- Avoid mutable keys (email, phone) to prevent rehash triggers.

Resharding techniques & when to use:- Consistent hashing with virtual nodes: handles incremental capacity changes with minimal remaps.- Range sharding is useful only for analytics or range-scan-heavy workloads; avoid for global user DB because it causes hotspots and costly splits.- Lookup-table (metadata service) supports controlled migration: mapping logical shard IDs to physical nodes (useful for manual rebalancing, blue/green moves, tenant isolation).

Online rebalancing / migration flow:1. Provision target shard nodes and warm caches/replicas.2. Update placement service to add target shard but do NOT redirect traffic yet.3. Start asynchronous data copy: use change-data-capture (CDC)/binlog to stream live updates from source shard to target; maintain a per-user change sequence number.4. Once initial copy reaches sync threshold, enter “dual-write” or “shadow-read” window: - Dual-write: application (or proxy) writes to both source and target for migrated user IDs. - Shadow-read: read traffic still from source until cutover validation passes.5. Cutover: atomically update placement mapping for affected user_id ranges via compare-and-swap in placement service; update routing caches (short TTL) and health-checks.6. Drain remaining writes (use CDC to apply last deltas), verify checksums, then decommission source or reassign.

Traffic routing:- Use a lightweight, highly-available routing layer (envoy/proxy or client-side lib) that consults placement service/cache. Cache mappings with short TTL and watch/subscribe for updates to minimize lookup latency.- For region affinity, route reads to nearest read-replica; ensure read-after-write by either routing recent writes to leader or using session stickiness until replication lag is below threshold.

Ensuring transactional integrity:- Keep per-user transactions single-shard by sharding on user_id; for multi-user transactions, prefer application-level orchestration (sagas) rather than distributed 2PC.- During migration, enforce a writable single source of truth using CAS on placement service; dual-write requires conflict resolution by monotonic versioning (last-write based on per-user logical timestamp or vector clock) or strong leader election per user.- Use CDC-based log positions to guarantee no missing updates; during cutover pause non-essential writes briefly if needed (milliseconds) to guarantee atomic mapping switch.- Validate with checksums and quorum reads following move; run consistency checker that compares hashes before finalizing.

Operational considerations & trade-offs:- Consistent hashing + virtual nodes minimizes remapping but can cause imbalance; monitor and add capacity via virtual node reassignment.- Dual-write increases complexity; use only for short windows with strict validation.- Avoid global distributed transactions; prefer idempotent operations, sagas for cross-shard workflows.- Automate rebalancing with capacity metrics (CPU, IOPS, replication lag) and safety gates (replication health, checksum pass).

Security, observability, rollback:- Log all placement changes, support immediate rollback by reverting placement mapping + replaying CDC.- Expose metrics: migration progress, replication lag, mismatch rates, routing cache hit ratio.- Encrypt in transit, authenticate placement service updates, and run automated canary rebalances.

This design prioritizes minimal downtime and few cross-shard transactions by hashing on immutable user_id, using consistent-hash placement with a lookup service for controlled, online resharding, CDC-based live migration, atomic placement updates for cutover, and application-level patterns (sagas) for cross-shard consistency.

MediumTechnical

80 practiced

Compare token bucket and leaky bucket algorithms for API rate limiting. Discuss centralized vs distributed enforcement, accuracy vs latency tradeoffs, memory and storage requirements, and how you'd enforce per-user limits at a scale of 100k RPS across multiple regions.

Sample Answer

High-level difference:- Token bucket: tokens accumulate at a fixed rate; allows bursts up to bucket capacity. Flexible, good for bursty traffic and fair-share smoothing.- Leaky bucket: constant outflow (fixed rate); enforces a steady rate and drops/excess requests. Simple, predictable but less burst-friendly.

Centralized vs distributed enforcement:- Centralized (single service/DB): simple correctness (single source of truth) but introduces latency, a single point of failure, and limited throughput.- Distributed (local nodes, caches, or CRDTs): lower latency, higher availability and throughput; harder to keep counts exact and requires synchronization or eventual consistency.

Accuracy vs latency tradeoffs:- Strong consistency (central store, e.g., Redis with INCR+TTL or Lua scripts): high accuracy, higher cross-region latency, limited QPS per instance.- Approximate/low-latency (local token buckets with periodic sync, client-side leaky bucket, or probabilistic counters): low latency and high throughput, but can briefly exceed limits (bounded error).

Memory and storage:- Per-user state: at 100k RPS and say 1M active users, lightweight state (~16–64 bytes/user) matters. In-memory stores (Redis cluster) are typical. Persisted logs for audit only; use TTLs to evict inactive users.- Sharding keys by user ID ensures distribution of memory and CPU.

Enforcing per-user limits at 100k RPS across regions (practical design):1. Local edge enforcement: implement per-user token buckets at edge (CDN or regional API gateway) to reject obvious bursts quickly—keeps tail latency low.2. Centralized quota policy and reconciliation: global control plane holds canonical limits and periodically syncs parameters to edges via pub/sub (low RPO). Use consistent hashing to route a small subset of writes to a global counter for long-term accounting.3. Strong enforcement for critical limits: route high-value users or suspicious traffic to a central Redis cluster using Lua scripts for atomic token consumption.4. Mitigations: rate-limit headers, backoff hints, and sticky hashing to avoid hot keys. Autoscale Redis shards and gateways; use circuit-breakers and burst tokens so transient cluster lag doesn’t cause undue blocking.5. Observability and safety: monitoring of reject rates, error budgets, and cross-region reconciliation jobs that fix drift.

Trade-offs summary:- Use token bucket at edges for UX-friendly bursts and low latency; rely on centralized checks for billing/abuse when absolute accuracy is required. Balance cost and accuracy by tiering enforcement: local fast checks + selective global authority.

Unlock Full Question Bank

Get access to hundreds of Optimization and Technical Trade Offs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.