NLP system deployment and efficiency Questions
Discuss deploying language models in production: batch vs. online serving, latency budgets, memory requirements, quantization and distillation for efficiency. Understand inference optimization for large models. Discuss monitoring and handling model degradation.
HardSystem Design
35 practiced
Design an architecture for on-device personalization of a language model under privacy constraints (no raw text leaves device). Consider approaches: local fine-tuning, adapters/LoRA updates, federated learning, encrypted aggregation, and how to merge per-device updates into global models. Discuss inference-time merging (e.g., posterior averaging, adapters) and safeguards against data poisoning.
Sample Answer
Requirements & constraints:- Personalization on-device only: no raw text leaves device.- Limited compute/memory, intermittent connectivity, heterogeneous data.- Improve user experience while protecting privacy and preventing poisoning.High-level architecture:1. Base model (server): large pretrained LM kept centrally.2. On-device personalization module: lightweight adapters/LoRA layers + private optimizer + local data store.3. Coordination: orchestration server for model metadata, update scheduling, and secure aggregation parameters (no raw text).4. Secure aggregation & analytics: encrypted aggregation (secure multi-party computation or homomorphic encryption) + differential-privacy budget accounting.5. Monitoring & defense: anomaly detection, robust aggregation, and rollback mechanisms.Local update strategies (on-device):- Prefer adapters/LoRA or prompt/adaptation vectors vs full fine-tuning to reduce compute, storage, and communication. Store only adapter weights or low-rank updates.- Optionally allow full fine-tuning for powerful devices, but convert to adapter format before sharing.- Training pipeline: local data preprocessing, gradient clipping, per-example DP noise addition (if sharing), sparsity/quantization of updates for efficient upload.Federated learning & encrypted aggregation:- Devices compute updates (adapter matrices or gradients), apply local clipping, then encrypt updates for secure aggregation (secure aggregation protocol or homomorphic encryption).- Aggregator computes model-agnostic weighted average of updates without seeing raw updates; use DP noise to bound leakage; return global adapter updates or update server-side base model.- Use adaptive weighting strategies (e.g., per-device reliability score) rather than naive averaging.Merging per-device updates into global models:- Server can aggregate adapters into a global adapter set (averaged LoRA weights) or integrate updates into the base model after validation.- Maintain per-cohort adapters: cluster devices by usage patterns and maintain multiple global adapters for different cohorts to avoid overgeneralization.- Use validation on held-out public data and shadow devices to test aggregated updates before deployment.Inference-time merging strategies:- Posterior averaging: ensemble predictions from base model and personalized adapter (logit interpolation or temperature-weighted averaging).- Adapter selection: dynamically select or interpolate between global adapter and device-local adapter based on context confidence.- Mixture-of-experts style routing: lightweight selector chooses which adapter(s) to use per request.- Prompting / context conditioning: prefer prompting or prefix-tuning on-device when low-resource.Safeguards against data poisoning and malicious devices:- Robust aggregation: - Byzantine-resilient aggregation (median, trimmed mean, Krum) instead of mean. - Norm clipping per-update and device-level contribution limits. - Outlier detection using update similarity to historical population (discard or down-weight anomalies).- Audit & validation: - Use server-side validation on synthetic or public holdout sets before accepting aggregated updates. - Rollback capability and staged rollout (canary cohorts).- Trust & accountability: - Reputation scores for devices, penalty for anomalous updates. - Limit personalization magnitude to prevent catastrophic shifts.Trade-offs & rationale:- Adapters/LoRA minimize bandwidth, compute, and privacy risk while enabling effective personalization—preferred default.- Secure aggregation + DP provides strong privacy but reduces utility; tune DP epsilon per product requirements.- Federated averaging scales but needs robust defenses to tolerate malicious clients.- Inference-time ensembling yields safer personalization (less risk of overfitting) while enabling immediate per-device behavior.Operational notes:- Provide SDK for on-device lifecycle (training loop, encryption keys, update scheduling).- Measure metrics: per-user utility lift, privacy budget consumption, convergence, and poisoning incidents.- Start with conservative adapters + posterior averaging + strong anomaly detection; iterate thresholds based on measured safety/utility.
HardSystem Design
33 practiced
Design a CI/CD pipeline for ML model deployments that ensures reproducibility, artifact provenance, schema checks, automated performance/regression tests, canary rollouts, and automatic rollback. Include tools you'd use for artifact storage (weights, tokenizer, config), data validation, test datasets, and how you'd gate promotion to production.
Sample Answer
Requirements & constraints:- Reproducibility: identical code, data, env, hyperparams -> deterministic builds- Provenance: trace weights/tokenizer/config -> who/what/when- Validation: schema & data checks pre-deploy- Automated perf/regression tests on holdout and synthetic test suites- Safe rollout: canary with metrics-based promotion and automatic rollbackHigh-level architecture:Git (PRs) → CI (GitHub Actions / GitLab CI / Tekton) → Build/Train → Artifact Store & Metadata → Staging cluster → Canary rollout → ProdCore components & tools:1. Source + CI- Git repo for code + infra + pipeline- CI: GitHub Actions / GitLab CI / Tekton pipelines to run lint, unit tests, model training orchestration triggers2. Data & feature/versioning- Data versioning: DVC or Delta Lake / lakehouse; record dataset commit IDs- Feature store: Feast for online features (versioned)3. Reproducible training & environment- Training runs in container (Docker) or K8s GPU pods (KubeFlow, Kubeflow Pipelines or Airflow/Tekton)- Environment pinned via conda lock / pip-lock + Dockerfile- Seed control and deterministic ops (CuDNN flags when possible)4. Artifact storage & provenance- Model artifacts: S3-compatible (AWS S3, MinIO) or artifact repo (MLflow Model Registry, Nexus/Artifactory)- Store: weights (.pt/.bin), tokenizer files, config/json, training dataset hash, code commit hash, environment lockfile, metrics, and model-card metadata- Metadata/tracking: MLflow or Weights & Biases to log runs, signatures, input/output schemas, and evaluation metrics; each artifact gets immutable URI and checksum- Signed releases: CI signs release artifacts (e.g., via GPG) and writes provenance into a metadata DB (Postgres/Glue + lineage in Data Catalog)5. Data validation & schema checks- Pre-training and pre-deploy: Great Expectations or TensorFlow Data Validation checks on schemas, distributions, missingness, feature drift thresholds- Gate fails CI if expectations not met; log anomalies to monitoring6. Test datasets & regression tests- Maintain canonical test datasets (unit/smoke/regression) stored versioned (DVC or S3 with dataset commit)- Automated test suite: - Unit tests for model I/O, tokenization, preprocessing - Functional tests: inference on small inputs - Performance/regression: full evaluation on holdout and benchmark datasets (latency, memory, accuracy, precision/recall, BLEU/ROUGE as applicable) - Adversarial / robustness tests & fairness checks- Use PyTest + custom evaluation scripts; CI fails on regressions beyond configurable deltas7. Staging, Canary rollout, and gating- Deploy candidate to staging namespace for integration tests (end-to-end with feature store and downstream services)- Canary: use Kubernetes + service mesh (Istio/Linkerd) plus Flagger for traffic shifting. Alternatively, use AWS SageMaker multi-model endpoints + traffic weights.- Canary strategy: start with 1-5% traffic; evaluate live metrics vs baseline for a time window- Metrics collection: Prometheus + Grafana, APM traces (Jaeger/New Relic), and model-specific monitors (EvidentlyAI or custom probes) for accuracy, latency, error rates, input distribution drift, and business KPIs- Promotion gating: automated policy checks: - No critical errors - Latency & error rates within threshold - Model quality metrics >= baseline or within acceptable delta - No drift alerts - Approval: automated if policies pass, else require manual sign-off in GitOps PR8. Automatic rollback & safety- Flagger (or custom controller) monitors metrics; on breach (e.g., 95th latency spike, accuracy drop) it automatically shifts traffic back to previous stable revision and raises alerts via PagerDuty/Slack- Maintain fast rollback path via immutable versioned deployments (K8s ReplicaSets / SageMaker model versions)9. Audit, governance & promotion gating- Promotion to production requires: - Passing CI + data/schema checks - Passing regression tests against canonical datasets - Signed artifact with MLflow/W&B entry, dataset hashes, code commit, environment lockfile - Security scan (container/image vulnerability scan) - Opt-in manual approval for major model changes (policy)- All promotions recorded in an audit log with timestamps, approver, metrics snapshotScalability & trade-offs:- Using MLflow + S3 scales and is simple; enterprise may prefer Artifactory + internal metadata store- Service mesh adds complexity but enables robust canary/routing features; serverless endpoints are simpler but less flexible for fine-grained traffic control- Strong gating reduces incidents but increases deployment latency — tune approval rules per risk level.Example flow (concise):1. PR merged → CI builds image, runs unit tests & schema checks2. Train job runs (if retrain) in reproducible container; logs to MLflow; artifact uploaded to S3 with manifest3. CI runs regression tests on canonical datasets; if pass, create release record4. Deploy to staging; run E2E tests5. Trigger canary in prod namespace with 5% traffic; monitor Prometheus + Evidently for 1 hour6. If pass policies -> promote to 100%; else automatic rollback to previous stable and page on-call.This pipeline ensures reproducible artifacts, traceable provenance, strict schema and data validation, automated regression testing, safe canary rollouts, and automatic rollback based on live metrics.
EasyTechnical
37 practiced
Explain the difference between data drift and concept drift in the context of an NLP classifier deployed in production. Give two concrete examples of each for an intent classifier and propose statistical tests or detectors you could implement to detect these drifts online.
Sample Answer
Data drift vs. concept drift (short):- Data drift: the input distribution p(x) changes over time while the mapping from input to label p(y|x) stays the same. The model sees different language/phrasing but intents remain defined the same.- Concept drift: the relationship p(y|x) itself changes — same text maps to a different intent or the intent definitions evolve.Concrete examples for an intent classifier (each 2):Data drift:1) New slang/abbreviations: users start writing "brb" or emojis to express intent; vocabulary and token distributions shift but intents unchanged.2) Channel shift: traffic moves from desktop web forms to voice transcripts—sentence length, disfluencies, ASR errors change p(x).Concept drift:1) Intent redefinition: product teams merge “account_help” and “billing_help” into one “support” intent; same utterances now map differently.2) Seasonal/behavioral change: during a promotion, “where is my order” is routed to refunds rather than shipping, changing labels for identical queries.Online detectors / statistical tests:- Data-drift detectors: - Population Stability Index (PSI) on features (token frequencies, embedding dims) between baseline and recent windows. - KL divergence or Jensen-Shannon between word/unigram or embedding distributions, with bootstrapped thresholds. - Embedding-based two-sample tests (e.g., MMD — Maximum Mean Discrepancy) on sentence embeddings.- Concept-drift detectors: - Monitor model confidence/calibration: sustained drops in predicted probability for top class or increase in entropy; use Page-Hinkley or CUSUM for change detection. - Label-feedback drift test: when true labels are available, track per-intent accuracy or confusion matrix shifts; use McNemar’s test or a Bernoulli CUSUM on error rates. - Proxy-label drift: clustering recent inputs and comparing cluster-to-predicted-intent mapping; if clusters map differently over time, flag possible concept drift.Operational notes:- Combine detectors and set thresholds via validation (false alarm trade-off).- Use sliding windows, stratify by channel, and instrument quick human-in-the-loop labeling to confirm drift and retrain or recalibrate as needed.
HardTechnical
39 practiced
Compare gRPC, HTTP/REST (JSON), and a custom binary protocol for high-throughput, low-latency model inference. Discuss serialization cost, streaming/token-by-token responses, connection overhead, load balancing, and how protocol choice affects client SDK design. If designing a new protocol, outline the key features and why they're necessary.
Sample Answer
High-level summary: For low-latency, high-throughput model inference, protocol choice is a tradeoff between latency (connection/setup, framing, parsing), throughput (serialization cost, binary packing, batching), and features (streaming, flow-control, load-balancing compatibility). Below I compare gRPC, HTTP/REST(JSON), and a custom binary protocol, then outline a recommended custom-protocol spec.Comparison- Serialization cost - gRPC (protobuf): compact, fast binary serialization with schema; low CPU, small wire size; supports zero-copy in many runtimes. - HTTP/REST (JSON): human-readable but high CPU parse cost and larger payloads; higher GC pressure in some languages; ok for small requests but poor at scale. - Custom binary: can beat protobuf if tailored (flatbuffers/Cap’n Proto/MessagePack-like): zero-copy, off-heap, fixed-layout for tensors reduces encode/decode.- Streaming / token-by-token - gRPC: native HTTP/2 streaming (unary, server, bidi). Good for token streaming with built-in backpressure and well-understood SDKs. - HTTP/REST: possible via chunked responses, Server-Sent Events, or WebSockets; more ad-hoc, client parsing varies. - Custom binary: can implement framed streaming with explicit flow-control for token-level latency and batching semantics.- Connection overhead - gRPC: uses HTTP/2, single long-lived connection with multiplexing — very efficient for many concurrent streams. - HTTP/REST: HTTP/1.1 has higher connection churn; HTTP/2 mitigates this but many REST stacks still use HTTP/1.1 or have intermediaries that break multiplexing. - Custom: should be built on a multiplexer (HTTP/2/QUIC) to minimize handshake and enable parallel streams.- Load balancing - gRPC: many LBs aware of HTTP/2 streams; however long-lived connections can complicate per-request LB (client-side channels + name resolver + pick-first or round-robin mitigate). Keepalive and connection draining matter. - HTTP/REST: stateless requests fit simple LB models; ephemeral connections simplify balancing but increase overhead. - Custom: must expose per-request routing (stream IDs) or use short-lived connections / client-side LB to avoid head-of-line issues.- Client SDK design effects - gRPC: SDKs provide generated stubs, strong typing, streaming primitives, and built-in retry/metadata hooks. SDK must manage channels, connection pooling, and backpressure. - HTTP/REST: SDKs simpler but must implement retry, streaming parsing, and efficient binary tensor transport (e.g., base64 or multipart), and handle varying server behavior. - Custom: SDK must encapsulate framing, flow-control, binary tensor buffers, version negotiation, and efficient memory management. SDK complexity increases but gives more optimization surface (zero-copy, pinned memory).Designing a new protocol — key features and why- Binary, schema-driven messages (like proto/flatbuffers): minimal serialization cost, forward/backward compatibility.- Built on a multiplexed transport: HTTP/2 or QUIC to avoid connection overhead and enable many concurrent inference streams.- Framing with explicit stream and message IDs: allow interleaving, cancelation, prioritization.- Server push / streaming primitives with backpressure: token-by-token streaming with credit-based flow control to avoid overwhelming GPU pipeline.- Zero-copy tensor descriptors and memory handles: allow passing GPU memory handles or DMA offsets to avoid host copies.- Batching primitives and hints: client can request micro-batching or server can advertise batch windows; priority & latency budgets in headers.- Lightweight negotiation/versioning and capability discovery: codec, compression, max frame size, supported precisions.- Security & observability: TLS + auth metadata, trace IDs, per-stream metrics, and error codes.- Retry and idempotency semantics: per-stream IDs and clear semantics for retries vs partial outputs.- Graceful connection draining and LB metadata: allow LBs to drain connections and signal to clients.Why necessary: these elements minimize round-trips/CPU work, enable true token-level low latency with safe flow-control, let servers exploit GPU zero-copy paths, and make load balancing predictable at scale while keeping SDKs able to provide simple ergonomics for developers.Practical recommendation: start with gRPC over HTTP/2 using a flatbuffers or proto schema, add GPU-memory pinning extensions and explicit batching/flow-control headers. If you outgrow gRPC (e.g., need QUIC, GPU handles), design a custom protocol with the features above and provide robust SDKs to hide complexity.
EasyTechnical
32 practiced
An online Q&A API shows high latency for repeated identical queries. Describe a practical caching strategy to reduce latency and GPU compute for this scenario. Specify: cache key design, where to place the cache (edge, application, model-server), TTL and stale-while-revalidate policies, invalidation rules when underlying models update, and how to handle cache consistency across multiple regions.
Sample Answer
Framework: treat the API as a read-heavy service where identical queries yield identical deterministic responses (or probabilistic with acceptably small variance). Aim: reduce GPU inference, lower tail latency, and keep freshness when models or prompts change.Cache key design:- Primary key = hash(request_fingerprint) where request_fingerprint = normalized_prompt + normalized_params (model name + version tag + temperature + max_tokens + system messages + user_locale + any relevant feature flags).- Include model_version and prompt_template_id in the key (not just model name) so keys invalidate automatically on model/prompts changes.- For non-deterministic outputs include a sampled_seed in the fingerprint only if you want to cache specific seeds; otherwise mark as non-cacheable.Cache placement:- Edge CDN (e.g., Cloudflare, Fastly) for global low-latency reads of popular queries.- Application-level cache (Redis/ElastiCache) as primary authoritative cache for misses and write-through updates.- Model-server local LRU cache for very hot queries to avoid network hop and GPU scheduling overhead.TTL & stale-while-revalidate:- Set short TTLs for high-change surface (e.g., 5–30 minutes) and longer for stable content (hours/days) depending on SLA.- Use stale-while-revalidate: serve stale cached response for up to X% of TTL (e.g., stale for 5 minutes) while asynchronously refreshing the cache by triggering an inference + write-back. That keeps tail latency low while keeping data fresh.Invalidation rules when models update:- Versioned model identifiers: whenever you deploy a new model or prompt template, increment model_version/prompt_template_id so old keys are automatically not matched.- For hot-fix patches that must invalidate existing cached outputs: publish an invalidation event to cache layer (Redis pub/sub + CDN purge API) to evict keys with affected model_version/prompt_template_id.- Provide a kill-switch header/query param to bypass cache for debugging.Cache consistency across regions:- Keep application cache (Redis) as primary with active-passive or clustered topology (Redis Cluster with geo-replication) and use edge caches per region reading from local app cache for misses.- For strong consistency on invalidation, push invalidation events via a global message bus (e.g., Kafka, Cloud Pub/Sub) that all regional app instances subscribe to; apply tombstones with short grace TTL to prevent stale reads during propagation.- If cross-region replication lag is unacceptable, prefer conservative TTLs and rely on model_version-based cache partitioning rather than trying to synchronously evict every regional cache.Additional considerations:- Cache only deterministic or high-reuse queries; respect privacy (don’t cache PII) and authorization (cache key includes user-scoped auth when responses are personalized).- Monitor cache hit-rate, GPU utilization, and tail latency; tune TTLs and stale-window based on metrics.- Auditability: log cache hits/misses with key and model_version for debugging.
Unlock Full Question Bank
Get access to hundreds of NLP system deployment and efficiency interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.