Model Deployment and Serving Questions

Covers techniques and practices for deploying machine learning models and serving predictions to downstream systems or users. Key areas include selection among batch inference, real time inference, and streaming inference based on trade offs such as latency, throughput, cost, and prediction staleness; common serving architectures and where they are appropriate including dedicated inference services, serverless functions, and edge deployment; deployment strategies for safe releases such as canary, shadow, blue green, and rolling updates; packaging and operationalization practices including containerization, orchestration, model artifacts, model versioning, and model registry practices; scaling and performance considerations such as batching and micro batching, autoscaling, hardware acceleration and model optimization techniques; interface and integration concerns including request and response formats for application programming interfaces, timeouts and retry policies, and online versus offline feature pipelines and feature serving; validation and experimentation such as A and B experiments for live validation, metrics for rollout decisions, and monitoring for model performance degradation and data drift; and integration with continuous integration and continuous deployment pipelines including automated model tests, validation gates, rollout automation and rollback strategies. For junior candidates, expect discussion of trade offs between approaches, recognition of appropriate choices given constraints, and an understanding of a basic deployment architecture.

HardSystem Design

68 practiced

Design an architecture for low-latency personalized recommendations that maintain per-user session state and support 100k concurrent users. Discuss state partitioning, co-location of compute and storage, consistent hashing, and strategies to maintain <50ms tail latency.

Sample Answer

Requirements & constraints:- 100k concurrent users, per-user session state, personalized low-latency (tail <50ms), strong read path for recommendations, model scoring (ONNX/TorchScript) possibly on GPU/CPU.

High-level architecture:- Ingress API gateway → routing layer → session-aware frontend nodes (stateless) → co-located state shards (Redis/KeyDB/rocksdb+serving) → model scoring service (GPU pool + CPU fallback) → personalization aggregator → client.

State partitioning & consistent hashing:- Partition user sessions by user_id using consistent hashing into N state shards. Each shard runs a Redis instance (or clustered) pinned to a subset of frontend/compute pods (sticky placement). Consistent hashing with virtual nodes ensures smooth re-sharding and minimal movement. Keep session keys small (binary protobuf) and TTLs to limit memory.

Co-location of compute & storage:- Co-locate Redis shard on same host (or same AZ & rack) as model scoring workers that primarily serve that shard. This reduces network hops and jitter. Use Kubernetes node affinity + PodAntiAffinity to ensure distribution and colocated pairings. For GPU workers, colocate CPU-side Redis to avoid cross-host network for hot sessions.

Maintaining <50ms tail latency:- Serve majority via cached pre-computed candidate lists stored per-session in Redis; keep top-50 candidates updated asynchronously (background updater). Scoring pipeline: fetch session + candidates in one pipelined Redis GET/MGET (batch), run lightweight model (quantized or distilled on CPU); if GPU needed, batch requests with very short batching window (1–5ms) to amortize cost.- Use connection pooling, pipelining, and Redis LOCALITY to keep p99 low. Use circuit breaker and fallback: if scoring slow, return cached ranking or heuristic (popularity + recency).- Monitor and autoscale: shard-level metrics (QPS, latency, memory). Use reactive autoscaling of frontend pods and GPU workers per shard.- Consistency and correctness: session state is primary in Redis; persist cold snapshots to durable store (S3 + async checkpoint). For re-sharding, use a two-phase migrate: mark keys migrating, replicate to new shard, then cutover.

Trade-offs:- Co-location increases resource coupling and operational complexity but reduces tail latency. Consistent hashing simplifies re-shard but increases complexity for cross-user features—store global embeddings in distributed read-replica store. Prioritize eventual consistency for non-critical signals; keep session reads strongly consistent.

This design balances state locality, cached candidates, fast lightweight models, and careful batching/fallbacks to hit sub-50ms tail latency at 100k concurrent users.

HardTechnical

57 practiced

You're serving a large LLM on serverless GPU backends and seeing inconsistent warm states causing tail-latency spikes. Propose techniques (warm pools, precompiled kernels, model snapshots, light-weight proxies) to reduce tail latency while controlling cost, and evaluate trade-offs.

Sample Answer

Approach: treat this as a trade-off optimization between latency SLOs (tail latency), cost, and complexity. I’d combine several techniques in a tiered strategy so we hit tail-latency targets while minimizing always-on GPU spend.

Proposed design:- Warm pool (cold/warm tiers): Maintain a small pool of warm GPU instances sized to absorb typical bursty traffic (p99 SLO target). Use autoscaling to grow the pool on predictable signals (time-of-day, recent request rate) and scale down conservatively with cooldowns. Keep a very small spare capacity for short spikes.- Model snapshots / checkpoint restore: Instead of full model reloading, persist memory snapshots (in-GPU or fast SSD) of the fully initialized model state. On warmup, restore the snapshot into GPU memory which is faster than re-run initialization and avoids expensive parameter downloads.- Precompiled kernels & memory planning: Ahead-of-time compile fused kernels (attention, GELU, fused matmuls) for target GPU architectures and store them with image/artifact. Use static memory planners so allocation is fast and deterministic, avoiding JIT stalls.- Lightweight proxy front-end: Deploy a small CPU-based proxy that handles quick routing, batching, and graceful degradation. Proxy can return estimated wait times, route to cached shorter responses, or serve smaller distilled models while the full instance warms.- Multi-tier serving: Serve a distilled/quantized version as cold-start fallback (lower cost, lower quality) while full model warms; route small / non-critical queries to distill model.

Implementation considerations & trade-offs:- Cost vs latency: Larger warm pool -> lower tail latency but higher idle GPU cost. Use predictive scaling + cooldowns to limit cost. Snapshots and precompiled kernels reduce warm time dramatically, allowing a smaller pool.- Complexity: Snapshots and kernel compilation add engineering complexity and storage/versioning burdens. Must ensure snapshot compatibility across driver/OS/GPU firmware; include migration plan.- Freshness / determinism: Snapshotting can complicate model updates—need safe invalidation and rolling snapshot builds. Precompiled kernels can break across GPU upgrades; maintain CI to rebuild artifacts.- Memory fragmentation & OOM risk: Static memory planning reduces OOM but is fragile across model variants—support multiple profiles per model size.- Reliability: Lightweight proxy adds a single point to scale; make it redundant and autoscaled.

Expected impact:- Restoring from snapshots + precompiled kernels can reduce warm times from minutes to seconds, dropping p99 by an order of magnitude and enabling a much smaller warm pool.- Lightweight proxy + distilled fallback smooths user experience for rare cold starts and reduces perceived latency without paying for large always-on capacity.

Monitoring & ops:- Track p50/p95/p99, cold-start percentage, cost per request. Alert on increases in cold-start rate; automate snapshot rebuilds when model or runtime changes.- Periodic load tests and chaos experiments to validate warm-pool sizing and snapshot restore times.

Summary: Use snapshots + precompiled kernels to minimize per-instance warm time, maintain a small predictive warm pool to meet p99, and add a CPU proxy + distilled fallback to smooth rare cold starts. This balances tail-latency targets and controlled GPU spend while acknowledging added engineering and operational complexity.

HardTechnical

54 practiced

Implement pseudo-code for a model canary controller in Python that routes a configurable percentage of traffic to a candidate model, computes sliding-window metrics (latency, error-rate, a primary business KPI), and automatically promotes or rolls back based on thresholds. Explain statistical safeguards you would use to avoid noisy decisions.

Sample Answer

Approach: run a controller loop that adjusts traffic split, collects per-request metrics into sliding windows, runs statistical tests (e.g., sequential A/B tests using a Bayesian or truncated Sequential Probability Ratio Test (SPRT)) to decide promotion/rollback, and applies safety rules (minimum sample size, minimum observation time, effect-size threshold, multiple-metric gating, hysteresis).

python

# python
import time, math
from collections import deque
from scipy import stats  # assume available for z-test; in real infra use robust libs

class CanaryController:
    def __init__(self, baseline_name, candidate_name, config):
        self.base, self.cand = baseline_name, candidate_name
        self.split = 0.0  # fraction to candidate
        self.config = config
        # sliding windows store tuples: (timestamp, latency_ms, is_error, kpi_value)
        self.base_win = deque()
        self.cand_win = deque()

    def route_decision(self):
        # Return True to route to candidate probabilistically
        return random.random() < self.split

    def ingest(self, record, to_candidate):
        win = self.cand_win if to_candidate else self.base_win
        win.append(record)
        cutoff = time.time() - self.config['window_seconds']
        while win and win[0][0] < cutoff:
            win.popleft()

    def summarize(self, win):
        n = len(win)
        if n==0: return None
        lat = [r[1] for r in win]
        err = [1 if r[2] else 0 for r in win]
        kpi = [r[3] for r in win]
        return {'n':n, 'lat_mean':sum(lat)/n, 'err_rate':sum(err)/n, 'kpi_mean':sum(kpi)/n}

    def z_test_proportion(self, p1, n1, p2, n2):
        # two-proportion z-test (approx); returns z, p
        p_pool = (p1*n1 + p2*n2)/(n1+n2)
        se = math.sqrt(p_pool*(1-p_pool)*(1/n1 + 1/n2))
        if se==0: return 0,1
        z = (p1 - p2)/se
        p = 2*(1 - stats.norm.cdf(abs(z)))
        return z, p

    def t_test_mean(self, x1_mean, n1, x1_var, x2_mean, n2, x2_var):
        # Welch's t-test approx using stats.ttest_ind if real samples available.
        se = math.sqrt(x1_var/n1 + x2_var/n2)
        if se==0: return 0,1
        t = (x1_mean - x2_mean)/se
        # degrees of freedom approximation omitted for brevity
        p = 2*(1 - stats.norm.cdf(abs(t)))
        return t, p

    def evaluate(self):
        s_base = self.summarize(self.base_win)
        s_cand = self.summarize(self.cand_win)
        if not s_base or not s_cand: 
            return 'insufficient'
        # Minimum samples safeguard
        if s_base['n'] < self.config['min_n'] or s_cand['n'] < self.config['min_n']:
            return 'insufficient'
        # Compute statistics (for latency and error-rate and KPI)
        # For simplicity use mean and approximate variances from windows
        lat_base = [r[1] for r in self.base_win]; lat_cand = [r[1] for r in self.cand_win]
        kpi_base = [r[3] for r in self.base_win]; kpi_cand = [r[3] for r in self.cand_win]
        import numpy as np
        lbm, lbn = np.mean(lat_base), len(lat_base); lbv = np.var(lat_base, ddof=1)
        lcm, lcn = np.mean(lat_cand), len(lat_cand); lcv = np.var(lat_cand, ddof=1)
        kbm, kbn = np.mean(kpi_base), len(kpi_base); kbv = np.var(kpi_base, ddof=1)
        kcm, kcn = np.mean(kpi_cand), len(kpi_cand); kcv = np.var(kpi_cand, ddof=1)
        # Error rates
        err_base = s_base['err_rate']; err_cand = s_cand['err_rate']

        # Statistical tests with multiple-metric gating
        # 1) KPI should be non-inferior (one-sided) within delta
        delta = self.config['kpi_delta']
        # we test (kcm - kbm) >= -delta -> if true candidate non-inferior
        diff = kcm - kbm
        # approximate z-test on means
        se_kpi = math.sqrt(kcv/kcn + kbv/kbn)
        if se_kpi==0: return 'insufficient'
        z_kpi = (diff + delta)/se_kpi  # positive z supports non-inferiority
        p_kpi = 1 - stats.norm.cdf(z_kpi)
        # 2) Error rate should not be significantly higher
        z_err, p_err = self.z_test_proportion(err_cand, s_cand['n'], err_base, s_base['n'])
        # 3) Latency should not regress beyond threshold
        se_lat = math.sqrt(lcv/lcn + lbv/lbn)
        z_lat = (lbm - lcm - self.config['lat_delta'])/se_lat if se_lat>0 else 0
        p_lat = 1 - stats.norm.cdf(z_lat)

        # Combine decisions: require KPI non-inferior (p_kpi < alpha), err not worse, lat not worse
        alpha = self.config['alpha']
        if p_kpi < alpha and p_err > alpha and p_lat > alpha:
            return 'promote'
        if p_err < alpha and err_cand > err_base + self.config['err_abs_threshold']:
            return 'rollback'
        if p_lat < alpha and lcm > lbm + self.config['lat_abs_threshold']:
            return 'rollback'
        return 'hold'

    def act(self, decision):
        if decision == 'promote':
            self.split = 1.0
        elif decision == 'rollback':
            self.split = 0.0
        elif decision == 'hold':
            # gradually increase if allowed
            self.split = min(1.0, self.split + self.config.get('ramp_step', 0.1))
        # apply hysteresis delay, cooldowns are handled outside

Statistical safeguards explained:- Minimum sample size and minimum observation window prevent early noisy decisions.- Use non-inferiority tests for primary KPI (candidate can be slightly worse within business delta).- Use two-proportion test for errors and t/Welch tests for means; consider Bayesian sequential tests or SPRT in production to control type I/II errors under continuous monitoring.- Apply multiple-metric gating (all critical metrics must pass) and adjust alpha with Bonferroni or prefer hierarchical testing to avoid overcorrection.- Require effect-size thresholds, cooldown/hysteresis, and manual override for high-risk releases.

MediumTechnical

68 practiced

Design a test strategy to catch data leakage and ensure preprocessing parity between training and serving. Include unit tests for transformations, integration tests that compare sample outputs, and a staging shadow run that validates model outputs against expected baselines.

Sample Answer

Start by defining goals: prevent data leakage (no future/label info in features), guarantee preprocessing parity (identical transforms in train and serve), and detect drift in staging.

1) Unit tests for transformations- Test deterministic behavior, stateless vs stateful separation, and inverse/round-trip where applicable.- Example (pytest) for a scaler + tokenizer pipeline:

python

def test_normalizer_tokenizer_deterministic():
    text = "Hello WORLD!"
    assert normalize(text) == normalize(text.upper())
    assert tokenize(normalize(text)) == ["hello","world"]

- Test stateful fit/transform consistency:

python

def test_scaler_fit_transform_consistent():
    arr = np.array([[1.],[2.],[3.]])
    s = MyScaler().fit(arr)
    assert np.allclose(s.transform(arr), s._manual_transform(arr))

- Test that pipeline prevents label leakage: ensure transforms don't access y:

python

with pytest.raises(AssertionError): feature_transformer.fit(X, y)

2) Integration tests: compare sample outputs- Serialize a canonical test dataset; run train-time preprocessing and serving preprocessing implementations and compare outputs element-wise and distributionally (e.g., KS test).

python

train_proc = train_transform(batch)
serve_proc = serve_transform(batch)
assert np.array_equal(train_proc, serve_proc)
assert ks_2samp(train_proc.flatten(), serve_proc.flatten()).pvalue > 0.05

- Test ordering, missing-value behavior, and categorical encoding maps identical (hashes of encoder state should match).

3) Staging shadow run- Deploy model behind feature-flag in staging; run live traffic in shadow mode producing: - Predictions_trainpipeline (what training pipeline would produce) - Predictions_servepipeline (actual serving pipeline)- Monitor mismatch rate, prediction distribution (KL divergence), and latency. Set thresholds: - Acceptable feature parity mismatch <0.1% - Prediction drift KL < 0.01 or change in top-1 class <1%- Log inputs, transformed features, model inputs, and predictions for audit.

4) Automation & CI- Run unit + integration tests on every PR. Fail CI on parity mismatches.- Add signed hashes of serialized preprocessing artifacts to ensure deployed artifact equals tested one.

5) Observability & Alerts- Runtime checks that validate schema, null-counts, categorical unknown rates; trigger rollback if thresholds exceeded.

Rationale: unit tests catch logic bugs; integration tests ensure implementation parity and statistical equivalence; shadow runs catch environment and data differences before production. Together they prevent leakage and maintain consistent behavior.

MediumSystem Design

60 practiced

Design an inference service for a moderately sized LLM (~7B parameters) that must handle 100 qps with average request length 200 tokens. Specify recommended compute (GPU type and count), batching strategy, autoscaling approach, model artifact storage, and discuss cost vs latency trade-offs.

Sample Answer

Requirements recap: 7B-parameter model, average request 200 tokens, 100 QPS → ~20,000 tokens/sec sustained. Target SLOs: p95 latency < 300–500 ms (adjustable).

Compute recommendation- Preferred GPU: NVIDIA A100 80GB (or H100 if available). A single A100 80GB running a well-optimized 7B model (FP16 + kernel fusions, cuBLAS/cuDNN/FlashAttention) can serve ~4–8k tokens/sec depending on implementation and prompt lengths. - Suggested cluster baseline: 4 × A100 80GB GPUs. With quantization (4-bit / NF4) and optimized kernels, each GPU can often reach ~5k tokens/sec → ~20k tokens/sec total, meeting 100 QPS at average 200 tokens.- Min/Max sizing: min 2 GPUs for low baseline traffic and resiliency; scale up to 8 GPUs for traffic spikes or higher SLO headroom.

Batching strategy- Use dynamic token-aware batching (not just request-count): accumulate requests into batches until either max tokens per batch (e.g., 1024–2048 tokens) or max latency wait (e.g., 10–30 ms) is reached. This keeps GPU utilization high while bounding extra queue latency.- Implement per-request padding trimming and grouped-by-sequence-length batching to reduce wasted compute.- Use batched beam/greedy decode and FlashAttention where possible; consider token streaming for very low-latency tails.

Autoscaling approach- Scale on a combination of metrics: - Primary: request queue length (tokens pending) and estimated tokens/sec demand. - Secondary: p95/p99 inference latency and GPU utilization.- Policy: if queue length implies >20% above current capacity for 2 consecutive intervals → add 1 GPU; if utilization <30% and queue low for N intervals → remove 1 GPU. Use cooldowns (2–5 min) and warm pools (pre-warmed GPU instances) to avoid cold-start latency.- Fast-path elastic workers: keep a small pool of hot warm-standby instances (1–2 GPUs) to absorb spikes quickly.

Model artifact storage & deployment- Store model checkpoints in object storage (S3/GCS) with immutability and versioning; register versions in a model registry (MLflow/MLMD or internal).- Keep quantized and FP16 artifacts: a full FP16 checkpoint (for high-accuracy runs) and a quantized 4-bit/NF4 checkpoint (for cost-optimized runs). Keep conversion scripts in CI.- Use containerized inference images and an initialization process that loads model into GPU memory at pod start; use checkpoint sharding if memory constrained.- Cache commonly used model in local SSD on nodes for faster cold-starts.

Cost vs latency trade-offs- Larger single GPUs (A100 80GB/H100) reduce inter-node communication and often reduce latency but increase per-node hourly cost. Fewer big GPUs vs many smaller GPUs: choose fewer big GPUs for simpler batching and lower latency tail.- Quantization and optimized kernels greatly reduce cost and improve throughput at tiny accuracy cost. Use quantized model for most traffic; keep FP16 for high-fidelity requests.- Aggressive batching increases throughput and reduces cost per token but raises per-request tail latency. Tune max batch tokens and wait timeout to balance SLO.- Autoscaling granularity: fast scaling reduces latency during spikes but increases cost due to more warm instances; slower scaling saves cost but can violate latency SLO.- Operational recommendation: start with 4 A100 80GB + quantized model, dynamic batching, warm pool of 1–2 GPUs, autoscale 2–8 GPUs. Monitor p95/p99 latency, GPU utilization, and cost per 1k tokens; iterate thresholds to reach desired cost/latency balance.

Unlock Full Question Bank

Get access to hundreds of Model Deployment and Serving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.