HardTechnical

33 practiced

Tail latency (99.99th percentile) is causing occasional user-facing timeouts. Propose architectural changes at the service layer to reduce tail latency, addressing queueing, concurrency limits, and cascading dependencies in a microservices environment.

Sample Answer

Clarify goals & constraints:- Target: reduce 99.99th-percentile service-layer latency causing user timeouts without breaking throughput or correctness.- Assumptions: microservices, some sync RPCs, background async pipelines for heavy work.

Proposed architectural changes (with reasoning and trade-offs)

1) Enforce per-endpoint concurrency limits + bounded queues- Use a fixed-size worker pool (thread/process) per service endpoint with a bounded request queue instead of unbounded queues that cause head-of-line blocking.- If queue is full, apply immediate failure or graceful degradation (HTTP 503 or lightweight cached response) to avoid queueing delay that inflates tail latency.Trade-off: small increase in rejected requests; prevents long tail.

2) Backpressure & adaptive rate-limiting- Propagate backpressure upstream (429 + Retry-After) and implement token-bucket rate limits per client.- Use adaptive admission control (shed load when 95th/99th latency crosses thresholds) to preserve p99/p999.

3) Bulkheads and resource isolation- Partition resources (CPU, thread pools, DB connections) per critical path, ensuring a slow non-critical pipeline can't exhaust global resources.- Use separate connection pools and priority queues for high-priority user-facing flows vs batch jobs.

4) Circuit breakers & timeouts per dependency- Strict, short deadlines for downstream calls (client-side timeout << client total timeout) and circuit breakers to fail fast and avoid cascading waits.- Use exponential backoff with jitter for retries and limit retry attempts; implement hedged requests for very latency-sensitive RPCs (send duplicate to a second instance after short delay).

5) Async handoff for long jobs- Move long-running or variable-time operations to asynchronous pipelines (enqueue to Kafka/ PubSub / SQS), immediately return a 202 and provide polling/webhook for completion.- For synchronous needs, return partial results quickly and fill in later.

6) Monitoring, SLAs & observability- Track queue lengths, queue wait times, worker utilization, service-level latencies (p50/p90/p99/p99.99), per-dependency latencies, and rejection rates.- Alert on growing queue wait or rising p99.

7) Autoscaling + CPU-aware scheduling- Scale worker pool based on service CPU, queue length, and tail latency, but ensure scale-up cooldowns are tuned to avoid oscillation.- Prefer vertical isolation (cgroups) or container resource limits to cap noisy neighbors.

Concrete example (thread pool + bounded queue semantics):- Configure executor with N workers, maxQueue=M. If queue full -> return 503 or push to async queue.- Set downstream RPC timeout = clientDeadline - safetyMargin (e.g., clientTimeout 2s → rpcTimeout 1s).

Why this works:- Bounded queues + fail-fast prevent unbounded queuing that creates extreme tails.- Bulkheads and resource isolation stop a single slow dependency from cascading.- Circuit breakers and timeouts ensure failures surface quickly and are handled, protecting overall latency budget.- Async handoff moves variable-duration work off the critical path.

Edge cases & next steps:- Determine N and M via load testing and queueing-theory models (M/M/c/K) to balance throughput vs reject rate.- Run chaos experiments to validate resilience under partial outages.

EasyTechnical

33 practiced

Define SLO, SLA, and SLI in the context of latency engineering for data pipelines. Give an example SLI and an SLO for a streaming ingestion pipeline that must deliver events to the data lake within 5 seconds 99% of the time, and explain how you'd measure and report this SLI.

Sample Answer

Definitions:- SLI (Service Level Indicator): a measurable metric that quantifies a specific aspect of service performance (e.g., latency, success rate).- SLO (Service Level Objective): a target or threshold for an SLI that you aim to meet over a defined period (operational goal).- SLA (Service Level Agreement): a formal contract with customers that often includes SLOs plus legal/financial consequences for breaches.

Example for the streaming ingestion pipeline:- SLI (concrete): Fraction of events whose end-to-end ingestion latency ≤ 5 seconds, where latency = timestamp_written_to_data_lake − event_produced_timestamp. Measured as: successful_events_with_latency_<=5s / total_events_ingested.- SLO (concrete): SLI >= 99% over a 30-day rolling window (and optionally a safety short-term SLO: >= 99% over a 1-hour rolling window).

How to measure and report:1. Instrumentation: - Attach producer timestamp (event_produced_ts) at source. - Record lake_write_ts when the event is persisted to the data lake. - Emit a metric for each event: latency_seconds = lake_write_ts − event_produced_ts and an outcome tag: latency_ok = latency_seconds <= 5s. - Use synchronized clocks (NTP/chrony) or include producer and ingestion offsets to correct skew.

2. Metric collection: - Expose as Prometheus counters: events_total and events_latency_ok, or use a histogram/summary of latency with bucket at 5s. - For high volume, use sampled histograms or aggregate within streaming processors (e.g., Flink sidecar).

3. Aggregation and evaluation: - Compute the ratio over chosen windows: ratio = sum(events_latency_ok) / sum(events_total) for rolling 1h and 30d windows. - Calculate error budget = (1 − SLO) × window_events; track burn rate.

4. Reporting and alerts: - Dashboard showing real-time ratio, latency percentiles (p50,p95,p99), event count, and error budget remaining. - Alerting rules: page on rolling 1-hour SLI < 99% or error budget burn rate > threshold; create PagerDuty + Slack notifications. - Postmortems and root-cause tags (backpressure, downstream writes, GC pauses, network) for breaches.

Edge considerations:- Define whether very-late events (e.g., >24h) are included or filtered.- Handle retries/deduplication so metrics count final persisted events.- Partition SLI by source, region, and event type for targeted debugging.

SLA note:- Convert the SLO into contractual SLA terms only after agreeing on measurement window, exclusions (maintenance), and compensation model.

MediumSystem Design

49 practiced

Design a caching strategy for a frequently-read user lookup table used by ETL jobs and real-time APIs where updates occur hourly. Requirements: low read latency, eventual consistency within 2 minutes after update, and low memory footprint. Explain cache invalidation, TTL, write-through vs write-back, and monitoring signals to detect stale reads.

Sample Answer

Requirements recap: low read latency, eventual consistency within 2 minutes after hourly updates, low memory footprint.

Design (summary)- Use Redis as an in-memory read cache with cache-aside + invalidation via pub/sub. Primary store = authoritative DB.- Cache keys store (value, version_ts). Values kept compact (only fields needed by reads); consider protobuf/JSON compression.- Eviction: configure Redis maxmemory with allkeys-lfu (or volatile-lru on TTL keys) to minimize memory.

Cache lifecycle & invalidation- Update flow (source-of-truth update occurs hourly): 1. DB update finishes and writes a monotonic version_ts for affected users. 2. DB publish event (Kafka/Redis pubsub) with user ids + version_ts. 3. Cache subscribers either: - Invalidate keys immediately (delete key) OR - Update cached value with new payload (push).- For safety, also set TTL of 2 minutes on cached entries. This guarantees eventual consistency within 2 minutes if invalidation is missed.

TTL and consistency tradeoffs- Short TTL (~2 minutes) meets the eventual-consistency SLA and bounds memory by letting infrequently used entries expire.- For very hot keys, rely on pubsub immediate invalidation to prevent staleness while avoiding frequent DB reads.

Write-through vs write-back- Recommend write-through (writes go to DB first, then update/invalidate cache on confirmation). Reason: simple correctness, predictable durability, and low risk for ETL/analytics correctness.- Write-back (write-behind) risks data loss and makes staleness window unpredictable — avoid for this use case.

ETL & real-time considerations- ETL jobs: bulk-warm cache before heavy reads (parallel prefetch) or read directly from DB for large scans to avoid evicting cache.- Real-time APIs: read from Redis; on cache miss, fetch from DB, populate cache with TTL.

Memory optimizations- Store only necessary columns, compress payloads, use hashes in Redis to reduce overhead, and use LFU eviction for long-tail usage.- Consider separate Redis namespaces for “hot” vs “cold” data with different memory/eviction policies.

Monitoring & detecting stale reads- Metrics to track: - Cache hit ratio and latency (per key cohort) - Pubsub invalidation lag (time between DB version_ts and cache invalidation) - Read-after-write mismatch rate: sample reads after known updates and compare cache vs DB (periodic consistency checks) - Staleness histogram: distribution of age(version_ts_now - cached_version_ts) - Error rates and increased DB read rate (indicates cache misses/leaks)- Alerts: - Invalidation lag > 5s (depending on acceptable) or > 120s → critical - Staleness > 2 minutes for >X% of sampled keys - Cache hit ratio drops below threshold or sudden spike in DB read QPS- Logging: include version_ts in returned payloads (or response headers) so services and tests can detect stale data.

Why this meets requirements- Reads are low-latency via Redis.- Combination of immediate invalidation + 2-minute TTL guarantees eventual consistency within 2 minutes.- LFU/compact storage + TTL bounds memory usage.- Observability lets you detect and alert on stale reads and invalidation failures.

EasyTechnical

32 practiced

Explain the difference between partitioning and indexing in databases. Provide concrete examples showing how each affects query latency in a time-series ingestion workload and for high-cardinality joins in a data warehouse.

EasyTechnical

24 practiced

How do you distinguish CPU-bound work from IO-bound work in a data processing job? List measurable indicators (CPU utilization, iowait, disk latency, network RTT), commands/tools to confirm (top, iostat, iotop, perf, strace), and a short checklist you would follow when triaging a slow job.

Sample Answer

Distinguishing CPU-bound vs IO-bound work relies on measurable signals and targeted tools.

Measurable indicators- CPU-bound: - High CPU utilization (near 80–100% CPU across cores) - Low iowait (<5%) - Short sys/user time ratio consistent with compute-heavy functions - Low disk/network latency and small queue depth- IO-bound: - High iowait (>10–20%) - High disk latency (ms — e.g., >10–50ms depending on storage) - High disk queue depth, high throughput saturation (MB/s) - High network RTT/throughput limits, many blocked syscalls

Commands / tools to confirm- top / htop — CPU%, load average, per-thread CPU- vmstat — iowait, context switches- iostat -x 1 — device util (%util), avgqu-sz, await, svctm- iotop — per-process I/O bandwidth and I/O% (Linux)- perf / flamegraphs — CPU hotspots, cycles, call stacks- strace -c or -tt -T — syscalls timing to see blocking reads/writes- sar / dstat — historical CPU/I/O metrics- tcpdump / ping / traceroute — network RTT and packet issues- cloud metrics (CloudWatch, Stackdriver) for disk/network IO limits

Quick triage checklist for a slow job1. Reproduce and note symptoms (slow end-to-end, long stage, specific task).2. Check cluster-wide metrics: overall CPU, memory, disk, network, load avg.3. Run top/htop on the worker node to see CPU vs sleeping/iowait threads.4. Use iostat/iotop to spot saturated disks or heavy per-process IO.5. Profile with perf or application-level profiler to find CPU hotspots.6. strace a slow process to detect blocking syscalls or long reads/writes.7. Check network: ping/iperf for latency/bandwidth limits, check shuffles (Spark).8. Verify configuration/limits: disk IOPS, network egress, JVM GC pauses, file system (NFS) latency.9. Remediate: if CPU-bound → optimize algorithm, parallelism, increase cores; if IO-bound → add caching, increase IOPS, partition data, tune batch sizes, use faster storage or push more parallelism to hide latency.10. Validate improvements and document root cause.

Latency Analysis & Optimization Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank