Algorithm Design & Real-Time System Optimization Questions
Algorithm design techniques and real-time optimization strategies applicable to distributed systems and latency-sensitive architectures. Covers scheduling, resource management, concurrency, distributed algorithms, load balancing, and performance optimization under strict latency requirements.
MediumSystem Design
53 practiced
Design a deadline-aware scheduler for an inference cluster where each request contains a soft or hard deadline and a priority. Describe algorithms and data structures you would use (e.g., earliest-deadline-first, multiple priority queues), how you would support preemption or cancellation, and approaches to avoid starvation and ensure fairness.
Sample Answer
Requirements:- Functional: accept inference requests with soft/hard deadlines and integer priority; return result before deadline when possible; allow cancellation.- Non-functional: low tail latency, high throughput, GPU/CPU resource constraints, SLOs per request type.High-level design:- Front-end API → Dispatcher → Scheduler → Worker pool (CPU/GPU containers) → Monitoring + Admission controller.Core scheduling algorithm and data structures:- Use a two-level scheme: admission + hybrid scheduling. - Admission: keep an estimated service-time distribution per model (from profiling) and reject or mark as “will-miss” if infeasible given current load. - Scheduler: Maintain multiple priority queues per priority class; within each queue use Earliest-Deadline-First (EDF) ordering. Represent queues with min-heaps keyed by deadline. Also maintain a global min-heap of (effective_deadline, priority_weight) for quick selection.- For batching: per-model batchers combine compatible requests while preserving earliest deadline constraints (batch deadline = min(deadlines in batch)).Preemption and cancellation:- Preemption: implement cooperative preemption. Workers check preemption points (between micro-batches or layers); if a higher-priority request with earlier deadline arrives and is admitted, worker can serialize current job state (if supported by model frameworks) or finish current micro-batch and switch. For GPUs, prefer fine-grained batching limits to reduce preemption latency.- Cancellation: client can cancel; scheduler removes request from queue/ batch if not started. If running, signal worker to stop and free resources ASAP.Avoiding starvation and ensuring fairness:- Weighted fair-share across priorities: assign weight to priority classes; use aging: increase effective priority of waiting requests over time (decrease their effective_deadline or boost priority_weight) so soft low-priority jobs eventually run.- Separate hard-deadline lane: strict EDF for hard deadlines with admission control to guarantee feasibility; soft-deadline lane uses best-effort with aging.- Backpressure and SLO-aware admission: when overloaded, preferentially accept hard deadlines and high-weight classes; expose SLAs to clients.Monitoring and feedback loop:- Continuously measure service times, miss rates, queue lengths. Tune admission thresholds, batch sizes, and aging rates. Use circuit-breaker to shed load and autoscale worker pool (scale GPUs for sustained high priority load).Trade-offs:- Strict preemption (checkpointing/stateful) is costly; cooperative preemption reduces overhead but increases latency for preempted tasks.- Aggressive batching improves throughput but can increase tail latency and misses; solution: deadline-aware batch windows that close early if earliest deadline is near.This design balances deadline guarantees, throughput (via batching), and fairness (via aging and weighted admission), and is practical for ML inference constraints (GPU batching, profiling-driven admission).
EasySystem Design
67 practiced
Design a single-node, low-latency serving process to execute an ensemble of 5 small models per request. Target: P95 end-to-end latency <150ms at 2000 RPS. Describe process architecture (threads/processes), batching strategy, CPU/GPU allocation decisions, I/O optimizations, memory layout, and how you'd instrument per-model latencies.
Sample Answer
Requirements clarification:- Single node, target P95 <150 ms at 2000 RPS (i.e., 2k requests/s)- Each request must run an ensemble of 5 small models- Low-latency priority over maximum throughputArchitecture overview:- One process (container) with subcomponents: network acceptor, request pre/post-processing threads, a batching layer, and one or more inference worker threads bound to GPU(s) or CPU.- Use a single GPU if models are small and fit GPU memory; otherwise fall back to CPU. Prefer GPU for throughput & latency if GPU inference time per sample ≪ CPU.Processes/threads:- Main process: manages lifecycle.- Network acceptor thread(s): NIO HTTP/gRPC server with a small thread pool to parse requests and push lightweight descriptors into a lock-free queue.- Preprocessing worker pool: M threads (M ~= #CPU cores - 2) performing tokenization/feature extraction; write tensors into a preallocated shared memory pool (zero-copy).- Dynamic micro-batcher: single thread per model-grouping that aggregates tensors into micro-batches with a short max wait (e.g., 2–8 ms) and a max batch size tuned to keep GPU utilization high but latency bounded.- Inference worker(s): for GPU, 1-2 threads driving CUDA streams; for CPU, a thread-pool with pinned NUMA affinity.Batching strategy:- Per-model micro-batching: because this is an ensemble of 5 models, maintain per-model batchers so each model’s optimal batch size can differ.- Request-level coordination: each incoming request gets an envelope referencing 5 model inputs. Preprocess produces 5 tensors (or shared tensor if same input). Each model’s batcher aggregates its tensors independently, issues inference when batch is full or max-wait timeout reached.- Use dynamic batching with adaptive timeouts: monitor latency and adjust max-wait/window (e.g., 2ms up when under latency budget, shrink when P95 creeps up).- For small models, batch sizes of 8–64 often hit sweet spot on GPU.CPU/GPU allocation decisions:- GPU for model inference if models are neural nets and inference latency per sample on GPU (with small batches) is < per-model CPU time and overall ensemble latency fits budget.- Assign preprocessing exclusively to CPU; inference to GPU. Pin preprocessing threads to CPU cores and isolate GPU driver interrupts.- If multiple models are tiny (e.g., <5ms each), run the 5 models sequentially on GPU in the same stream per batch to avoid context switch overhead; for independent models that can run concurrently, use multiple CUDA streams to overlap kernels.I/O optimizations & memory layout:- Zero-copy: receive request payloads into preallocated memory, directly convert to input tensors without copying using memory-mapped buffers or shared memory.- Use pinned (page-locked) host memory for CPU->GPU transfers to reduce latency.- Preallocate tensor buffers per batch size to avoid allocator overhead; reuse pools.- Align memory for SIMD and use contiguous layout expected by frameworks (NCHW/NHWC as model does).- Use fused preprocessing kernels where possible (e.g., vectorized tokenization) to reduce CPU time.Concurrency & CUDA:- Use multiple CUDA streams: assign one stream per model or per inference worker to overlap kernel execution and host transfers.- Use CUDA events to measure and synchronize per-model kernels without blocking.Instrumentation (per-model latencies):- Emit high-resolution timestamps at these points: request-received, preprocess-start/end, enqueued-to-batcher, batch-dispatched-to-infer, model-kernel-start/end, postprocess-start/end, response-sent.- Measure per-model latency inside the inference pipeline: time between model dispatch and model-kernel-end (including queue delay and transfer).- Export histograms and percentiles (Prometheus histograms or OpenTelemetry metrics) per model and overall end-to-end.- Correlate traces with request id using distributed tracing (OpenTelemetry) so you can see per-request per-model spans. Record batch-size and wait-time tags.- Add alerts when per-model P95 or queue wait-time grows.Sizing/back-of-envelope:- 2000 RPS -> 0.5 ms per req if fully parallel, but ensemble imposes multiple ops. Aim to keep per-request mean inference time across 5 models + pre/post <= 40–80 ms so P95<150ms.- Tune batcher timeouts to trade throughput vs latency; keep micro-batches small (8–32) and timeouts 2–8 ms to avoid queueing spikes.Other practical notes:- Warm-up models and keep them resident (no lazy loads).- Use FP16/bfloat16 kernels if accuracy allows to reduce kernel latency.- Use priority scheduling for latency-critical requests if mixed workloads exist.- Test under load (loadgen) to measure P95 and auto-tune batch sizes/timeouts.This design minimizes copies, isolates CPU/GPU responsibilities, uses micro-batching per-model to balance utilization and latency, and provides per-model instrumentation so you can monitor and adjust to keep P95 <150 ms.
MediumTechnical
95 practiced
Explain hedged requests (sending duplicates to multiple backends) and design how to implement hedging at an inference gateway to reduce tail latency. Include criteria for issuing hedges, deduplication and cancellation techniques, and the cost/throughput trade-offs involved.
Sample Answer
Hedged requests send duplicate copies of a request to multiple backends and use the fastest response to reduce tail latency. For an inference gateway, design goals are: minimize 95–99.9th percentile latency while controlling extra cost and backend load.Design:- When to hedge (criteria) - Static delay: issue a hedge if primary backend hasn't responded within T_base (e.g., median p50 + slack). - Dynamic/adaptive: use per-backend latency histograms and request-specific SLAs to set T_base per model/version; increase hedging probability when observed tail latency or error rate rises. - Probabilistic hedging: only hedge X% of requests (higher for high-priority or latency-sensitive requests).- How to send: send primary to preferred backend; after T_base, issue duplicate(s) to alternate backends or replicas with possibly lower model fidelity.Deduplication & cancellation:- Use request IDs and versioning so backends tag responses with the same ID; gateway treats first successful response as canonical.- Cancellation: send a cancel/abort signal (gRPC cancel, HTTP/2 RST_STREAM) to slower replicas once winner arrives. Note many inference runtimes still charge CPU/GPU for work already started; cancellation reduces queuing but not always compute already consumed.- Idempotency: ensure model servers can handle duplicate requests safely; make side effects (logging, metrics) idempotent or offloaded.Engineering considerations:- Resource-aware routing: prefer hedging to less loaded replicas to avoid amplifying congestion.- Priority/sla queues: avoid hedging low-priority traffic.- Telemetry: measure latency distributions, extra compute cost, and successful-hedge rate to tune thresholds.Cost / throughput trade-offs:- Hedging reduces tail latency but increases effective QPS and resource consumption roughly by (1 + hedge_rate * num_duplicates). This can worsen queuing if system is near capacity.- Benefits highest when system has spare capacity or when latency variability dominates (stragglers). If CPU/GPU constrained, better options: autoscaling, prioritization, model quantization, or heterogeneous serving (fast but slightly less accurate model).- Use adaptive hedging and conservatively hedge only critical requests; monitor cost-per-latency-improvement (e.g., dollars per ms saved) to decide policy.Example policy:- p50/p90 stable: no hedges.- p95 > SLA or single request type marked critical: T_base = max(1.5*p50, fixed_ms); probabilistic hedging 20% → increase if tail persists.This approach balances latency gains with cost, uses adaptive signals to avoid amplifying congestion, and relies on request IDs + cancellation to minimize wasted work.
HardTechnical
91 practiced
For a recommender system with embedding tables larger than any single node's memory, design algorithms for sharding embedding tables across machines to minimize tail read latency under heavy read load. Discuss placement heuristics, hot-shard caching, replication vs partitioning trade-offs, routing decisions, and strategies to handle skewed access patterns.
Sample Answer
Clarify goals & constraints: minimize tail (99th/99.9th) read latency under heavy read-dominated traffic, embedding tables exceed single-node RAM, network cost and CPU for deserialization matter, updates are relatively infrequent.Design summary:- Sharding model: hybrid of range/hash partitioning + selective replication. Partition large table by embedding id using consistent hashing for balanced primary ownership; track partitions in a lightweight metadata service.- Placement heuristics: - Start with hash-based uniform partitioning; monitor per-partition QPS and latency. - Rebalance by moving "cold" ranges to consolidate and free nodes for hot replicas. - Use affinity-aware placement: co-locate partitions often queried together (same user/session) to reduce multi-hop reads.- Hot-shard caching: - Per-node LRU/ARC in-memory cache for recently/frequently accessed embeddings (cache items keyed by (table,id)). - Global hot index: stream telemetry (QPS per id) to a controller that maintains a hot set (top-K by weight). Promote top hot ids for memory replication. - Use negative caching for misses and small Bloom filters to avoid wasted RPCs for non-existent ids.- Replication vs partitioning trade-offs: - Partitioning gives minimal memory duplication and deterministic ownership; cheaper writes. But hot keys create tail spikes. - Replication reduces read tail by serving hot keys locally; increases memory and update fanout. Use selective replication of only hot ids (tiered replication: 1..R replicas based on heat). - For consistency, use eventual consistency with version vectors for embeddings (reads tolerate slightly stale vectors) or lightweight version stamps for strictness when needed.- Routing decisions: - Client-side routing via a small routing library using the metadata (consistent-hash + hot-index). Prefer local cache hit; on miss, route to nearest replica (lowest RPC latency) using latency-aware load balancing. - Fallback: multi-get RPC that fetches multiple ids from multiple shards in parallel; use hedged reads for requests whose latency exceeds threshold—issue concurrent read to another replica and take first reply. - Batch coalescing at frontend: aggregate ids across concurrent requests to reduce RPCs.- Handling skew: - Detect skew via online counters and decay windows. For heavy hitters: - Create on-demand replicas placed across many nodes (read fan-out) and update hot-index. - Materialize compressed quantized versions on more nodes for read-only approximate fetches when strict accuracy not required. - Use adaptive caching TTL: increase TTL for hot keys to reduce eviction churn. - Rate-limit or degrade gracefully: for extremely hot keys, serve a cached fallback (e.g., last-known embedding, or average) to avoid saturating store.- Metrics & ops: - Track 95/99/99.9 latency, RPC queue length, per-id QPS, cache hit ratio, replication lag. - Automate hot promotion/demotion via thresholds and cooldowns; simulate changes via shadow traffic before commit.Trade-offs recap: selective replication + client routing + hedged reads minimize tail latency while keeping memory overhead bounded. Conservative consistency and monitoring keep correctness and operational safety.
MediumTechnical
48 practiced
Implement (in Java or Python) a concurrent prioritized queue with aging for inference requests. The queue should normally return the highest priority item but gradually increase the effective priority of older items to prevent starvation. Provide interface definitions, key methods, and explain your approach to ensure thread-safety and reasonable performance.
Sample Answer
Approach (brief):Use a thread-safe PriorityBlockingQueue to store requests wrapped with base priority and enqueue timestamp. Because Java's comparator is static and doesn't re-order items as time passes, on polling we greedily scan up to N top candidates (peek/drain few), compute their effective priority = basePriority + agingRate * ageSeconds, pick the currently best, and reinsert the rest. This prevents starvation while remaining performant (scanLimit bounded). Synchronize scans to ensure thread-safety.Interface and implementation (Java):Key points / reasoning:- We keep basePriority immutable and compute "effective" priority on poll using current time; agingRatePerSec controls how fast older requests climb.- Using PriorityBlockingQueue keeps normal ordering and high throughput for inserts.- To handle time-dependent ordering, we scan a bounded number of top elements (scanLimit). This avoids full-queue reheap costs and prevents starvation in practice because older items will surface into the scanned top-N as they age.- scanLock is available if you choose stricter synchronization for complex operations; current implementation uses atomic queue methods and is safe because drainTo + put operations are thread-safe; if strict mutual exclusion is required across multiple queue operations, wrap poll logic with scanLock.lock().Concurrency and performance considerations:- put/offer are O(log n); poll is O(k log n) where k = scanLimit (drainTo and reinsert cost). Keep scanLimit small (e.g., 16 or 32).- This trades exact global optimality for bounded latency and throughput—practical for ML inference where starvation matters but absolute ordering across time is less critical.- Tuning: agingRatePerSec and scanLimit depend on traffic and priority distribution; can add metrics to adapt them.Edge cases:- Very large queues: increase scanLimit or run background reheap thread to periodically rebuild order.- Tight latency: choose small scanLimit to reduce overhead.- Equal effective priorities: fallback uses basePriority comparator.
java
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.locks.*;
import java.time.Instant;
public class AgingPriorityQueue<T> {
public static class Request<T> {
final T payload;
final int basePriority; // higher means more important
final long enqueuedAtMillis;
public Request(T payload, int basePriority) {
this.payload = payload;
this.basePriority = basePriority;
this.enqueuedAtMillis = System.currentTimeMillis();
}
}
private final PriorityBlockingQueue<Request<T>> queue;
private final double agingRatePerSec; // how much priority increases per second
private final int scanLimit; // how many items to scan on poll to choose best
private final ReentrantLock scanLock = new ReentrantLock();
public AgingPriorityQueue(double agingRatePerSec, int scanLimit) {
// comparator is a fallback by basePriority (highest first)
this.queue = new PriorityBlockingQueue<>(11, Comparator.comparingInt((Request<T> r) -> -r.basePriority));
this.agingRatePerSec = agingRatePerSec;
this.scanLimit = Math.max(1, scanLimit);
}
public void put(Request<T> req) {
queue.put(req);
}
public boolean offer(Request<T> req, long timeout, TimeUnit unit) throws InterruptedException {
return queue.offer(req, timeout, unit);
}
// Poll that applies aging by scanning top scanLimit candidates
public Request<T> poll(long timeout, TimeUnit unit) throws InterruptedException {
long deadline = System.nanoTime() + unit.toNanos(timeout);
// Try fast-path: peek and return if queue small
Request<T> head = queue.poll();
if (head == null) {
long remain = deadline - System.nanoTime();
if (remain <= 0) return null;
return queue.poll(remain, TimeUnit.NANOSECONDS);
}
// We have at least one item; collect up to scanLimit-1 more
List<Request<T>> candidates = new ArrayList<>();
candidates.add(head);
queue.drainTo(candidates, scanLimit - 1); // already have one in list
// If no more candidates and others exist, we might be fine to return head.
// Compute effective priority and pick best
Request<T> best = null;
double bestScore = Double.NEGATIVE_INFINITY;
long now = System.currentTimeMillis();
for (Request<T> r : candidates) {
double ageSec = Math.max(0, (now - r.enqueuedAtMillis) / 1000.0);
double score = r.basePriority + agingRatePerSec * ageSec;
if (score > bestScore) {
bestScore = score;
best = r;
}
}
// Reinsert others
for (Request<T> r : candidates) {
if (r != best) queue.put(r);
}
return best;
}
public int size() { return queue.size(); }
}Unlock Full Question Bank
Get access to hundreds of Algorithm Design & Real-Time System Optimization interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.