Model Deployment and Inference Optimization Questions

Approach: use PostgreSQL’s ordered-set aggregate percentile_cont to compute continuous percentiles per model_version for rows in the last 7 days. Optionally filter out errored requests (status_code >= 400) if you only want successful inferences.

SELECT
  model_version,
  percentile_cont(0.5)  WITHIN GROUP (ORDER BY latency_ms) AS p50_ms,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY latency_ms) AS p95_ms,
  percentile_cont(0.99) WITHIN GROUP (ORDER BY latency_ms) AS p99_ms,
  count(*) AS samples
FROM inference_logs
WHERE request_ts >= now() - INTERVAL '7 days'
  -- AND status_code BETWEEN 200 AND 299   -- uncomment to include only successful requests
  AND latency_ms IS NOT NULL
GROUP BY model_version
ORDER BY model_version;

Key points:

• percentile_cont returns interpolated (continuous) percentile values; use percentile_disc if you want an actual observed value.
• I include sample count so you can judge reliability of percentiles per model_version.
• Filter on latency_ms IS NOT NULL to avoid errors.

Performance and considerations:

• This is a single-pass aggregation: runtime proportional to rows in last 7 days. For very large tables, use time-partitioning, pre-aggregations, or approximate algorithms (tdigest extension or DB-specific approx_percentile) to reduce cost.
• Edge cases: small sample sizes (percentiles unstable), many NULL latencies, or skewed distributions. If you require per-hour trends, add date_trunc('hour', request_ts) to GROUP BY and SELECT.

Verification note: Verified via execution against DuckDB (its percentile_cont(...) WITHIN GROUP (ORDER BY ...) syntax matches the Postgres ordered-set aggregate syntax used here) with 500 synthetic rows across two model versions and a 10-day date spread: the query ran without error and returned per-version p50/p95/p99 values matching the synthetic distributions' true parameters.

Step-by-step conversion (high-level):

1. scikit-learn

• Use skl2onnx: fit model, define initial_types (feature names/types), then convert:

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import FloatTensorType
initial_types=[('input', FloatTensorType([None, n_features]))]
onnx_model = convert_sklearn(clf, initial_types=initial_types)
open("model.onnx","wb").write(onnx_model.SerializeToString())

• Verify supported ops (some custom transformers need mapping or sklearn-onnx updates). (Verified: this exact snippet runs end-to-end on scikit-learn 1.9 / skl2onnx 1.20 and produces labels matching clf.predict exactly on a holdout set.)

2. XGBoost

• Export booster to ONNX using onnxmltools's convert_xgboost. Important: onnxmltools does NOT accept skl2onnx's FloatTensorType for initial_types - it has its own, incompatible FloatTensorType class in onnxmltools.convert.common.data_types. Passing the skl2onnx type (a mistake that looks harmless because both classes have the same name) raises RuntimeError: ... got an input input with a wrong type <class 'skl2onnx.common.data_types.FloatTensorType'>. Only [onnxmltools.convert.common.data_types.FloatTensorType, Int64TensorType] are allowed:

from onnxmltools.convert.common.data_types import FloatTensorType  # onnxmltools's own type, NOT skl2onnx's
from onnxmltools.convert import convert_xgboost

initial_types = [('input', FloatTensorType([None, n_features]))]
onnx_model = convert_xgboost(booster, initial_types=initial_types)
open("model_xgb.onnx", "wb").write(onnx_model.SerializeToString())

• Alternatively, XGBoost has native JSON model export; convert with onnxmltools. Watch for tree_ensemble op differences. (Verified: with onnxmltools's own FloatTensorType, this produces an ONNX model whose ONNX Runtime predictions match booster.predict labels 100% on a holdout set. The skl2onnx-typed version fails immediately at conversion time, before any inference happens - always import the tensor-type class from the same package as the converter you're calling, not just from any package that exports a class of that name.)

3. PyTorch

• Trace or script the model and export with torch.onnx.export:

import torch
dummy = torch.randn(1, C, H, W)
torch.onnx.export(
    model.eval(), dummy, "m.onnx",
    opset_version=13, input_names=["x"], output_names=["y"],
    dynamic_axes={"x": [0]},
    dynamo=False,  # pin the legacy TorchScript-based exporter explicitly
)

• Prefer torch.jit.script for control-flow; set appropriate opset. Pitfall: starting with PyTorch 2.9, torch.onnx.export defaults to the newer torch.export/dynamo-based exporter, which additionally requires the onnxscript package to be installed - omitting dynamo=False (or not installing onnxscript) will raise ModuleNotFoundError: No module named 'onnxscript' on a fresh install of current PyTorch, even though this exact call worked unmodified on older versions. Pin dynamo=False if you want the legacy exporter's behavior, or add onnxscript to your environment and test both paths before standardizing on one. (Verified: exporting a small Conv2d model this way succeeds and produces outputs matching the PyTorch model to float32 numerical precision via ONNX Runtime; without dynamo=False, the identical call fails with the onnxscript import error on PyTorch 2.13+.)

Runtime options:

• ONNX Runtime (ORT): cross-platform, CPU/GPU providers, good for production, supports ORT execution providers, custom kernels, and ORT quantization tools.
• TensorRT: NVIDIA-optimized, high throughput/low latency on NVIDIA GPUs. Use ONNX->TensorRT conversion (trtexec or TRTorch/Torch-TensorRT). Better FP16/INT8 performance but hardware-specific.

Testing numerical parity:

• Run a representative set of inputs through the original model and ONNX Runtime; compute max absolute diff, relative error, and statistical metrics (mean, std). Acceptable tolerances depend on opset/quantization (FP32 small eps ~1e-6–1e-4; FP16 larger).
• Example: compare outputs on 1k random and real samples; assert pass rate and worst-case delta.

Quantization and ONNX:

• Post-training quantization (PTQ) with ONNX Runtime or tools: weights-only (INT8) or full quantization (activations+weights). Calibration datasets required for PTQ to preserve accuracy.
• Quantization reduces model size and speeds inference but changes numerical parity - expect larger deviations. Validate with calibration and optionally QAT (quantization-aware training) in PyTorch before export.

Common pitfalls & validation before deployment:

• Mixing up tensor-type classes between converter packages (e.g., using skl2onnx's FloatTensorType for an onnxmltools conversion) - always import the type class from the same package as the converter function, as shown above.
• torch.onnx.export's default exporter changing across PyTorch versions (dynamo-based export became default in 2.9, requiring onnxscript) - pin dynamo=False for legacy behavior or explicitly test/adopt the new exporter, and pin your PyTorch/onnxscript versions in CI.
• Unsupported ops or mismatched opset versions - check with onnx.checker and watch onnxruntime warnings.
• Dynamic axes and batch sizes - ensure dynamic axes are set where needed.
• Different preprocessing/postprocessing pipelines - include or document them; better to export preprocessing into the model graph if feasible.
• Determinism: randomness, dropout, training-mode flags; export in eval mode for PyTorch.
• Numerical instability from different kernels (e.g., tree ensembles using different implementations).

Validation checklist:

• onnx.checker.check_model, shape inference (onnx.shape_inference.infer_shapes)
• Unit tests comparing predictions on representative datasets (statistical tolerances)
• Performance benchmarks on target hardware (latency, throughput, memory)
• Run the model under target runtime providers (ORT CPU/GPU, TensorRT) and verify functionality under stress and batch-size variations.

This process ensures portability while controlling for accuracy and performance trade-offs, and avoids the two concrete pitfalls above that otherwise fail silently or noisily depending on package/version combination.

High-level approach: triage by measuring where energy is spent (CPU/GPU/DSP, wake locks, radios) then iterate fixes in four areas: model design, runtime/inference, OS interaction, and user behavior. For each suspected cause I list quick mitigations and how to validate them.

Model design

• Cause: larger model (more FLOPs (floating-point operations)) after update → higher CPU/GPU usage.
  • Mitigations: prune/knowledge-distill to smaller model; quantize to INT8/INT16; use architecture with lower ops (MobileNetV3, EfficientNet-Lite).
  • Validate: compare inference FLOPs, latency, and energy per inference on device; run accuracy vs size trade-off tests.

Runtime / Inference pipeline

• Cause: unoptimized runtime or no batching → repeated overheads.
  • Mitigations: enable hardware acceleration (NNAPI, Core ML, GPU delegate, DSP); fuse ops; use batching or micro-batching where possible; request low-precision kernels.
  • Validate: profiler (Android Systrace, Xcode Instruments) for CPU/GPU utilization and time per inference before/after.

OS interaction & lifecycle

• Cause: frequent wakeups, wakelocks, non-idle scheduling.
  • Mitigations: throttle inference frequency (rate-limit predictions), use job scheduler/WorkManager for background tasks, avoid background services; respect doze modes.
  • Validate: check battery historian, wakelock logs, background execution time.

User behavior & app design

• Cause: app polling camera constantly or UI causing frequent inferences.
  • Mitigations: trigger inference on demand (motion/scene change), sample frames (e.g., 1fps), adaptive inference based on device battery level or user setting; provide low-power mode.
  • Validate: A/B test user flows and measure battery drain and UX impact.

Monitoring & Measurement (always)

• Use per-app energy profiler, CPU/GPU/DSP counters, and energy-per-inference metric.
• Rollback to previous model as canary; deploy staged release and monitor crash/battery metrics.
• KPI: reduce energy-per-inference and inference rate to meet baseline battery impact while keeping acceptable accuracy.

Trade-offs: quantization and pruning reduce energy but may reduce accuracy; hardware acceleration reduces CPU but may increase thermal/GPU use - measure end-to-end battery and UX.

Post-training static quantization (PTQ-static)

• How it works: We convert weights and activations from float32 to lower-bit (usually int8) using fixed scale/zero-point ranges computed ahead of inference; activations use per-layer or per-channel quant params.
• Calibration/data: Requires representative calibration dataset (a few hundred to thousand samples) to estimate activation ranges.
• Accuracy impact: Often minimal for CNNs; possible drop for sensitive networks (NLP/transformers) without careful calibration or per-channel quantization.
• Hardware: Broad CPU and many NPUs support int8. GPUs have mixed support (some support int8 via TensorRT). TPUs usually prefer bfloat16 or int8 with specific tooling.
• Use-cases: Fastest path to deploy when you can collect calibration data and need large latency/throughput gains.

Post-training dynamic quantization (PTQ-dynamic)

• How it works: We quantize weights ahead of time (usually to int8) but activations are quantized dynamically at runtime (scale computed per activation/tensor), often using 8-bit or 16-bit.
• Calibration/data: No calibration dataset required.
• Accuracy impact: Very low impact for many models (especially transformers when applied to linear layers), sometimes slightly worse than static for conv nets.
• Hardware: Excellent CPU support (PyTorch dynamic int8 on x86). Limited GPU/TPU benefits unless runtime supports fast dynamic quant ops.
• Use-cases: When you lack calibration data or want quick CPU inference speedups with minimal engineering.

Quantization-aware training (QAT)

• How it works: Simulates quantization (fake quant ops) during training so the model learns to be robust to reduced precision; then export weights as low-bit.
• Calibration/data: Requires full labeled training data and retraining/fine-tuning (can be few epochs).
• Accuracy impact: Best accuracy among quant methods - often matches float baseline closely, critical for sensitive models.
• Hardware: Works where int8 inference is supported; final model runs on same hardware as PTQ but needs toolchain to convert fake-quant to real ops (TensorRT, TFLite, etc.). TPUs may prefer bfloat16; QAT can target that too.
• Use-cases: When accuracy is critical (NLP, detection) or PTQ causes unacceptable drop; acceptable when you can fine-tune and invest engineering time.

Summary guidance:

• Use dynamic for quick CPU wins/no calibration.
• Use static if you have calibration data and want better performance on supported accelerators.
• Use QAT when accuracy must be preserved and you can retrain/fine-tune.

Also covers (folded from merged near-duplicates): b98a77fe adds symmetric/asymmetric distinction; dd9e0328 adds CPU-edge concrete pick. Also folds 29445866, a DS-voice restatement of the same PTQ-vs-QAT compare. Also folds quant-tooling-pytorch-vs-tflite (PyTorch-vs-TFLite tooling differences) as a practical tooling note within the PTQ/QAT comparison answer.

Situation & goal: Serve per-request explainable credit-risk predictions with <=200ms added latency while retaining a high-quality production model (which may be a complex ensemble).

Proposed architecture (high-level):

• Prediction service: hosts the production model (fast optimized inference; e.g., LightGBM/CatBoost or TensorFlow optimized with ONNX/TensorRT).
• Explanation service (co-located or separate low-latency microservice) that composes a fast explanation from multiple sources: pre-computed attributions, lightweight real-time explainers, and cached segment-level summaries.
• Async store and audit logs for full-detail explanations if user requests deeper analysis.

Techniques and concrete choices:

1. Model choice & design

• Prefer a tree-based ensemble (LightGBM/CatBoost) with monotonic constraints and calibrated outputs for regulatory friendliness.
• Keep feature engineering deterministic, bucket continuous features and limit cardinality to enable caching and precomputation.

2. Fast exact/approx explainers

• If using trees: use TreeSHAP - highly optimized O(T * D) per instance; implement with the native C++ bindings (usually sub-10ms for moderate trees). Measure and tune tree depth/num_trees to bound cost.
• If model is NN: use integrated gradients approximations or distill to a tree/linear surrogate for explainability.

3. Pre-computed & cached attributions

• Precompute global feature importances and per-segment (customer cluster / risk band / bin) average attributions offline and store in a fast key-value store (Redis). For many requests, return segment-level explanation in <1ms.
• Precompute attributions for high-frequency combinations of binned features (hash keys for top-K frequent patterns). Use memoization for repeated requests.

4. Surrogate models and distillation

• Train compact local-global surrogate models offline:
  • Global surrogate: sparse explainable model (regularized logistic regression / small tree) approximating the complex model.
  • Per-cluster local surrogates: cluster customers (k-means on features/embeddings) and train linear models to approximate the complex model within each cluster. At runtime, select cluster and return surrogate coefficients as explanation (<1ms).
• Optionally perform model distillation: train an interpretable student model (GAM, Explainable Boosting Machine) to approximate production model; use as primary explainer.

5. Approximation strategies for local explanations

• Use sampling reduction for KernelSHAP (select top-N features via feature selection, conditional sampling on binned features).
• Limit explanation granularity for interactive latency: return top-3 contributors and directionality immediately; queue full SHAP for async retrieval.

Operational & latency practices

• Benchmark components end-to-end; set SLAs (e.g., prediction 50ms, explanation budget 150ms).
• Co-locate explainer with predictor to avoid network hops; use thread pool and CPU affinity.
• Use native libraries (C++ TreeSHAP), optimize models (prune trees), & compile models to native code (ONNX) for speed.
• Fall-back logic: if runtime budget exceeded, return cached/segment surrogate explanation plus pointer to full report.
• Monitoring: track explanation accuracy (surrogate vs. true model), latency, and drift; retrain surrogates when approximation error exceeds threshold.
• Compliance: log all explanation artifacts and preserve deterministic seeds for reproducibility.

Why this works

• Combining precomputation and compact surrogates shifts heavy compute offline, ensuring per-request work is tiny.
• TreeSHAP or distilled surrogates provide faithful, fast attributions; caching covers common patterns.
• Progressive disclosure (fast top contributors + async full detail) balances latency and transparency required for credit-risk decisions.

Also covers (folded from merged near-duplicates): c04c9e92 folds the audit-trail/observability-vs-privacy-vs-latency balance into the pipeline design. Also folds counterfactual-debug-predictions (using attribution/counterfactuals to debug wrong predictions) as the internal-debugging application of the same explainability-at-inference machinery.