Model Deployment and Inference Optimization Questions

Clarify SLOs & workload
I’d confirm traffic shape (QPS (queries per second), burstiness), input sizes, and correctness/latency trade-offs (accuracy vs quantization). Aim: p50 20ms, p99 200ms for a heavy model.

High-level architecture

• Ingress → lightweight front proxy (Envoy) → request router (per-model) → inference tier (GPU/CPU/TPU pools) → response.
• Observability: latency/queue metrics, per-stage tracing.

Serving frameworks

• Consider Triton for multi-backend, batching, model versioning; TorchServe for PyTorch pipelines; BentoML for flexible pipelines. Triton’s dynamic batching and model repository help meet SLOs.

Batching / dynamic batching

• Use small fixed batch sizes for p50; enable dynamic batching with max latency budget (e.g., 5–10ms wait) to increase throughput without exceeding p99.
• Adaptive batching: adjust batch timeout based on current tail latency and queue length.

Autoscaling & warm pools

• Combine reactive autoscaling (CPU/GPU util, queue length) with predictive scaling from traffic forecasts.
• Maintain a warm pool of pre-loaded model instances (or containers) to avoid cold starts. Keep a small number of GPU-backed instances idle.

Cold-start mitigation

• Lazy-weighted routing to warm instances; lightweight fallback to smaller/quantized model for quick responses when heavy model not ready.

Model partitioning & compilation

• Use model compilation (ONNX Runtime with TensorRT/XLA) and quantization (FP16/INT8) to reduce latency.
• Consider model sharding for extremely large models (sequence parallelism or pipeline parallelism) but weigh added network latency.

Hardware

• Use GPUs for heavy models; choose A100/H100 or inference-optimized instances with Tensor Cores. For low-cost/less heavy models, CPU with AVX512 and ONNX may suffice.

Caching

• Cache recent/identical inputs and partial outputs (embedding cache) at front proxy. Use TTL and cache hit-miss metrics.

Throughput vs tail-latency trade-offs

• Larger batches maximize throughput but increase tail latency. Use hybrid strategy: benchmark batch-size vs p99, enforce max wait time, and prefer smaller batches under bursty/latency-sensitive traffic.

Monitoring & iterative tuning

• Continuously monitor p50/p99, per-batch latency, queue times. Run canary experiments for quantized/compiled models and measure accuracy-latency trade-offs.

Worked capacity example (from a folded near-duplicate): with single-request GPU compute time of 25ms at batch size 1 and a CPU fallback of 80ms/request, one GPU replica saturates at 1/0.025s = 40 requests/sec, and one CPU replica at 1/0.080s = 12.5 requests/sec. Targeting 1000 requests/sec (queries per second) while keeping utilization at roughly 60% (so bursts do not blow through the p99 target) needs about 1000/(400.6) ≈ 42 GPU replicas, versus roughly 1000/(12.50.6) ≈ 133 CPU replicas for the same throughput - a concrete illustration of why the GPU path is chosen for this SLO (service-level objective) even though a single GPU instance costs more than a single CPU instance: replica count, not raw compute, is what needs to be provisioned and paid for at this scale.

Requirements & constraints:

• Re-score 100M user-item pairs nightly (throughput target ~100M predictions / ~8–12 hours)
• Low cost, reliable, auditable, and safe-to-publish results with validation gates

High-level architecture:

1. Orchestration: Airflow / Cloud Composer schedules pipeline nightly.
2. Feature read: Feature store (Feast) or Parquet snapshots on S3/GCS, joined via Spark.
3. Batch compute: Spark on EMR (or Dataproc) for large-scale distributed inference; Dataflow/Beam is an alternative if you favour streaming semantics or serverless.
4. Model artifacts: Stored in model registry (MLflow, SageMaker Model Registry) with immutable versioned artifacts (SavedModel / TorchScript / ONNX).
5. Output: Write scored results to S3/GCS (partitioned), then load to serving DB / feature-store table for downstream use.
6. Validation & publish: Holdout checks, sampling, canary publish, metrics & drift checks; manual/automated promotion.

Compute choices & rationale:

• Spark on EMR: good for heavy joins and wide shuffles (feature joins across users/items). Use EMR with EC2 instance fleets (spot + on-demand mix) or EMR-managed spot to reduce cost while keeping a few on-demand masters.
• If preference is serverless and simpler ops, Dataflow with autoscaling is viable but can be costlier for heavy shuffle workloads.
• Use containerized UDFs (e.g., spark.mapPartitions calling TorchScript/TF SavedModel) to run optimized native inference.

Handling model artifacts:

• Export model as performant, portable artifact: TF SavedModel with tf.function or TorchScript/ONNX for CPU inference.
• Store in model registry with metadata (hash, validation metrics, tag “approved”).
• At job start, worker bootstrap downloads a specific model version from registry to local disk or cache (S3 Fuse or container image). Use checksums to verify integrity.
• Prefer local caching on each node (download once per node) and load into memory to avoid repeated network I/O; if model is small, broadcast it via Spark’s broadcast variable (for tiny models).

Minimize cost:

• Spot/preemptible instances for workers; reserve a minimal on-demand master (and a small pool of on-demand workers to finish critical partitions).
• Use instance fleets / mixed instance types to improve spot availability.
• Right-size instance families: CPU-optimized for small models, memory-optimized if feature join heavy.
• Batch inference optimizations:
  • Vectorize inference: batch many pairs per model call to use BLAS.
  • Quantize / use int8 where acceptable to reduce CPU/GPU usage.
  • Use model compilation (TorchScript, XLA) where helpful.
  • Cache feature joins and commonly accessed side tables in memory (persist RDDs/DataFrames) across stages.
  • Filter/prune pairs early (business rules) to reduce the 100M baseline if possible.
• Use spot-aware retry and checkpointing to avoid reprocessing large amounts on preemption.

Data flow and scaling:

• Partition user-item pairs by user or shard key; tune Spark partitions to roughly 2–4x number of executor cores.
• Each executor loads cached model artifact, reads local partition features (or from broadcast), batches inputs, runs vectorized inference, emits scores.
• Use shuffle-optimized joins and data locality.

Validation before publishing:

• Automated unit tests on small known datasets every run.
• Dry-run stage: write scored results to a staging bucket and compute:
  • Statistical checks: distribution of scores vs. baseline (Kolmogorov-Smirnov, mean/variance), rate of NaNs.
  • Business logic invariants: top-k consistency, monotonicity for known signals.
  • Regression tests: compare against previous “golden” model on a holdout sample (e.g., 1% or fixed seed 10k pairs); require metrics (AUC, MAPE) to meet thresholds.
  • Data drift checks on input feature distributions.
• Canary publish: promote a small partition (e.g., 1% of users) to production endpoints and monitor live metrics for 24 hrs.
• Human-in-the-loop gating: alert on fails; allow manual approval in Airflow UI.
• Post-publish monitoring: telemetry of online impact, latency, CTR/engagement metrics, automatic rollback if severe regression.

Trade-offs:

• EMR + spot saves cost but increases orchestration complexity; Dataflow is simpler but potentially more expensive.
• Broadcasting model works only when artifact fits memory; large models require local disk caching.
• Heavier validation reduces risk but increases time-to-publish; balance via sampling and staged rollouts.

This design balances scalability (Spark partitions, autoscaling), cost (spot instances, quantization, caching), and safety (model registry, checksum, staged validation/canary) for nightly re-scoring of 100M pairs.