Model Deployment and Inference Optimization Questions

Requirements:

• Support single and batched text classification
• Include per-request contextual metadata (request_id, user_id, optional session/context)
• Deterministic, explicit error codes and messages
• Lightweight JSON over HTTPS, POST
• Versioned contract (/v1/classify)

High-level contract:

• Endpoint: POST /v1/classify
• Headers: Authorization: Bearer <token>, Content-Type: application/json, X-Request-Timeout (optional client hint)
• Body: either "inputs" (array) or single "input" (string/object). Each item can carry metadata override.

Request JSON examples:

{
  "request_id": "req_12345",
  "user_id": "user_678",
  "model": "text-classifier-v2",
  "inputs": [
    {"id":"i1","text":"I love this product!","metadata":{"language":"en"}},
    {"id":"i2","text":"This is terrible.","metadata":{"language":"en"}}
  ],
  "options": {"top_k":3}
}

Single input shorthand:

{
  "request_id":"req_54321",
  "user_id":"user_999",
  "input":{"id":"i1","text":"Neutral statement."}
}

Response JSON example (200 OK):

{
  "request_id":"req_12345",
  "model":"text-classifier-v2",
  "results":[
    {"id":"i1","predictions":[{"label":"positive","score":0.97},{"label":"neutral","score":0.03}]},
    {"id":"i2","predictions":[{"label":"negative","score":0.99}]}
  ],
  "latency_ms":42
}

Errors (use HTTP status + machine-readable body):

• 400 Bad Request: INVALID_INPUT - malformed JSON or missing text
• 401 Unauthorized: AUTH_FAILED
• 413 Payload Too Large: BATCH_TOO_LARGE - include max_batch_size
• 422 Unprocessable Entity: FORMAT_ERROR - per-item problem (return items with errors)
• 429 Too Many Requests: RATE_LIMIT_EXCEEDED - retry-after header
• 500 Internal: MODEL_ERROR - non-retryable without backoff
  Error body example:

{"request_id":"req_12345","error":{"code":"BATCH_TOO_LARGE","message":"max batch size is 64","details":{"max_batch_size":64}}}

Timeouts & retries for client SDKs:

• Per-request timeout recommendation: 10s default for single, 30s for batches (client-adjustable)
• Retries: idempotent by request_id. Retry only on transient errors: 429, 503, network timeouts. Use exponential backoff with jitter (initial 200ms, multiplier 2, max 5 attempts). Do NOT retry on 4xx except 429.
• Honor Retry-After if provided.
• Clients should stream large batches or chunk to max_batch_size (e.g., 64).

Contract testing:

• Validate schema for single vs batch, round-trip tests for request_id propagation, error scenarios (per-item failures), and rate-limit/retry flows.

High-level summary: ONNX Runtime (ORT), TensorRT, and OpenVINO are production-focused inference runtimes with different strengths: ORT for portability and broad backend support, TensorRT for maximum NVIDIA GPU throughput/low-latency, and OpenVINO for Intel CPUs, iGPUs, and VPUs with strong CPU/edge optimization.

ONNX Runtime

• Performance advantages: Good out-of-the-box performance across CPUs and GPUs; vendor-optimized execution providers (CUDA, ROCm, DirectML, OpenVINO) and graph optimizations (fusions, constant folding).
• Supported hardware: CPUs (x86/ARM), NVIDIA/AMD GPUs via providers, Windows ML/DirectML, specialized accelerators via plugins.
• Conversion pain points: ONNX export fidelity varies by framework/operator version; opset mismatches and custom ops may require rewriting or custom kernels.
• When to choose: Multi-platform deployments where portability, fast iteration, and easy integration with ML infra matter (cloud + edge hybrid).

TensorRT

• Performance advantages: Best throughput/latency on NVIDIA GPUs via aggressive layer fusion, kernel autotuning, FP16/INT8 quantization, and optimized memory planning.
• Supported hardware: NVIDIA GPUs / Jetson.
• Conversion pain points: Requires ONNX or framework export compatible with TRT; unsupported ops need plugins; dynamic-shape handling and calibration for INT8 add engineering overhead.
• When to choose: High-throughput GPU inference on NVIDIA hardware (server GPUs or Jetson) when maximizing performance and minimizing latency is critical.

OpenVINO

• Performance advantages: Optimized for Intel CPUs, integrated CPU vectorization, low-latency on iGPUs and Movidius VPUs; good CPU inference performance without a GPU.
• Supported hardware: Intel x86 CPUs, integrated GPUs, Intel VPUs (Neural Compute Stick), Windows/Linux.
• Conversion pain points: Requires conversion to IR or OpenVINO format; some ops and newer frameworks need workarounds; quantization/tooling quirks.
• When to choose: Edge or on-prem CPU/iGPU/Intel VPU deployments where GPU is unavailable and power/latency constraints exist.

Trade-offs / practical guidance:

• Start with ORT for cross-platform proof-of-concept; profile.
• If deploying to NVIDIA and need peak perf, convert to TensorRT and invest in plugin/INT8 calibration.
• For CPU/Intel-edge targets, convert to OpenVINO for best CPU/generic-edge latency.
• Keep an automated CI conversion & validation pipeline and unit tests to catch numerical/regression issues after conversion.

Situation & goals
I would treat the p99 jump from 100ms → 300ms as high-priority SLO (service-level objective) regression. Immediate goals: protect customers, gather signals, and identify root cause across model, runtime, infra, and network.

Safety & mitigation (first 10–30 mins)

• Roll forward traffic to stable canary/previous model version or throttle new version via traffic split (e.g., 0% new). If platform supports, promote rollback automatically for high-error/latency.
• Enable rate-limits, circuit-breakers, and priority-queueing for high-value traffic paths.
• Increase autoscaling headroom temporarily to avoid queueing.

Signal collection

• Turn on detailed sampling: p99, p50, 95/99 CPU, memory, GPU utilization, GC, context-switches, socket queue lengths.
• Increase tracing sampling to 100% for affected routes for short window.
• Launch synthetic tests (micro-benchmarks) that call model with representative inputs from cache and from cold-start to isolate I/O vs compute.

Profiling & local repro

• Run CPU/GPU hot-profiler (e.g., py-spy, Nsight) against the new model on an identical instance type to find kernel hotspots, memory copies, or operator regressions.
• Collect model-level metrics: FLOPs (floating-point operations), operator breakdown, attention lengths, tokenizer overhead.
• Check framework and dependency versions (e.g., PyTorch, CUDA, cuDNN, Triton) for known perf regressions.

Distributed tracing & request flow

• Use traces to break down time into: ingress load‑balancer, network, request routing, preprocess, model infer, postprocess, response.
• Look for tail latency amplification patterns: queueing at endpoints, retries, retries cascading to upstream services.
• Correlate trace spans with instance IDs to see if particular nodes are slow (noisy neighbor or mis-sized).

Orchestration & instance sizing

• Inspect pod/container lifecycle: OOMs, restarts, throttled CPU cgroups. Confirm resource requests/limits correct for new model.
• Run SLO-aware pod profiling: check startup cold time, GPU memory fragmentation, NUMA (non-uniform memory access) placement.
• If ML runtime changed (TorchServe/TF-Serving/Triton), check config like batch size, dynamic batching latency vs throughput trade-offs.

Networking

• Measure RTTs, retransmits, and socket queue drops. Validate MTU, LB health checks, and connection multiplexing (HTTP2 keepalive).
• Rule out network config changes coincident with deploy (new VPC, sidecars, service mesh proxy version).

Synthetic canary experiments

• Deploy canary with controlled traffic patterns: single-threaded, batched, high QPS (queries per second) to isolate batching/config issues.
• A/B test one variable at a time: model weights, runtime version, batcher config, instance SKU.

Root-cause isolation checklist

• If profiling shows compute hotspot in new ops → model change (attenuation of layers, different kernel).
• If traces show queueing and CPU/GPU saturation → insufficient instance sizing or batching misconfig.
• If only some nodes slow → noisy neighbor, instance degradation, or scheduling issue.
• If network spans dominate → LB/mesh/regression.

Postmortem & prevention

• Capture minimal repro, exact commit, dependency versions; roll back if not already.
• Add regression tests: microbenchmarks for p50/p95/p99, CI perf checks on representative hardware.
• Automate canary metrics gating and quicker rollback path; add continuous profiling for model deployments.

I’d communicate status frequently, prioritize rollback if customer impact grows, and deliver a concise RCA with remediation and tests to prevent recurrence.

Below is a multi-stage Dockerfile that prioritizes small image size, supports CPU or GPU PyTorch builds via build ARGs, copies a pre-downloaded model at /models/model.pt, creates a non-root user, exposes port 8080, and provides a simple healthcheck. Key choices are commented inline and summarized after the Dockerfile.

# syntax=docker/dockerfile:1
# Build args: BASE (runtime base image), TORCH_WHEEL (pip spec for torch, e.g. "torch==2.2.0+cpu" or "torch==2.2.0+cu121")
ARG BASE=python:3.11-slim
ARG TORCH_WHEEL="torch==2.2.0+cpu"  # override to GPU wheel when building for GPU runtime

# ---- builder: install dependencies and build wheels (keeps final image small) ----
FROM ${BASE} AS builder
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends build-essential ca-certificates wget git \
    && rm -rf /var/lib/apt/lists/*
# Install pip and wheel, then install dependencies to /opt/venv
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN pip install --upgrade pip setuptools wheel
# Install runtime Python deps (including torch) into venv
# Use --no-cache-dir to avoid pip cache
RUN pip install --no-cache-dir ${TORCH_WHEEL} fastapi uvicorn pydantic

# ---- final: minimal runtime image ----
FROM ${BASE} AS runtime
# Create non-root user
RUN groupadd --gid 1000 appgroup && useradd --uid 1000 --gid appgroup --shell /bin/false --create-home appuser
WORKDIR /app
# Copy virtualenv from builder to final to avoid reinstalling wheels
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
# Copy application code (assumes ./app exists) and model artifact into image
COPY app /app
# model is provided into build context at ./models/model.pt and placed at /models/model.pt
COPY models/model.pt /models/model.pt
# Ensure correct ownership
RUN chown -R appuser:appgroup /app /models && chmod -R 750 /models
USER appuser
EXPOSE 8080
# python:3.11-slim does NOT ship curl (or wget), and installing it in the final stage adds
# apt-managed packages (plus their own CVE surface) purely for a healthcheck - use the
# interpreter already in the image instead of adding a system package for this.
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
    CMD python -c "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=2).status==200 else 1)" || exit 1
# Start command (example using uvicorn for a FastAPI inference app)
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080", "--workers", "1"]

Short rationale:

• Multi-stage: build/install deps in builder, copy only venv to final to reduce layers and size.
• ARGs let CI choose CPU vs GPU wheel (use GPU base image and CUDA-enabled torch wheel when needed).
• slim base plus --no-cache-dir and removing apt lists reduces image size.
• Non-root user improves security.
• Healthcheck uses HTTP probe; expose 8080 and run uvicorn as example entrypoint.

Approach (brief)
Instrument model forward by wrapping each layer call to record start/end timestamps and peak GPU/CPU memory around execution. Aggregate per-layer latencies and memory, export JSON timeline, and report top-3 time/memory hotspots.

Implementation (Python, PyTorch-style pseudo-code)

import time, torch, tracemalloc

def instrument_model(model):
    """
    Attach forward hooks to every direct child module to record per-layer latency and peak
    memory. Returns the list of records (populated as the model runs) and the list of hook
    handles (call handle.remove() on each to detach cleanly).

    NOTE: overwriting a registered submodule attribute with a plain wrapper function (e.g.
    `setattr(model, name, wrapper_fn)`) raises TypeError, because nn.Module.__setattr__
    enforces that a name already tracked in self._modules stays a Module (or None) - it will
    not silently accept a callable in its place. register_forward_hook is the mechanism
    PyTorch actually provides for this and does not disturb the module tree.
    """
    records = []
    handles = []

    def make_hook(name):
        state = {}
        def pre_hook(module, args):
            if torch.cuda.is_available():
                torch.cuda.synchronize()
                state['start_mem'] = torch.cuda.memory_allocated()
            else:
                tracemalloc.start()
            state['t0'] = time.perf_counter()

        def hook(module, args, output):
            if torch.cuda.is_available():
                torch.cuda.synchronize()
                peak = torch.cuda.max_memory_allocated()
            else:
                _, peak = tracemalloc.get_traced_memory()
                tracemalloc.stop()
            t1 = time.perf_counter()
            records.append({
                'layer': name,
                'latency_ms': (t1 - state['t0']) * 1000,
                'peak_bytes': int(peak),
            })
        return pre_hook, hook

    for name, child in model.named_children():
        pre_hook, hook = make_hook(name)
        handles.append(child.register_forward_pre_hook(pre_hook))
        handles.append(child.register_forward_hook(hook))
    return records, handles

# usage:
# records, handles = instrument_model(model)
# with torch.no_grad(): model(input)
# for h in handles: h.remove()

Key concepts & reasoning

• Use CUDA sync to measure accurate GPU kernel time.
• Use torch.cuda.max_memory_allocated() for GPU peak; tracemalloc for CPU.
• Wrapping named children captures per-layer stats without modifying internals.

Output & analysis

• Export timeline JSON of records.
• Compute top-3 by latency and by peak_bytes.

CI integration

• Run profiler on fixed seed synthetic inputs in nightly/PR benchmarks.
• Store baseline metrics; compute regression if any layer latency or peak memory increases > threshold (e.g., 5% absolute or 50ms).
• Fail CI on significant regressions; include diffed timeline artifact and hotspot report for triage.
• Add statistical smoothing (median over N runs) to reduce noise and GPU warmup.