Architecture and Technical Trade Offs Questions

Centers on system and solution design decisions and the trade offs inherent in architecture choices. Candidates should be able to identify alternatives, clarify constraints such as scale cost and team capability, and articulate trade offs like consistency versus availability, latency versus throughput, simplicity versus extensibility, monolith versus microservices, synchronous versus asynchronous patterns, database selection, caching strategies, and operational complexity. This topic covers methods for quantifying or qualitatively evaluating impacts, prototyping and measuring performance, planning incremental migrations, documenting decisions, and proposing mitigation and monitoring plans to manage risk and maintainability.

HardSystem Design

28 practiced

Create an incremental migration plan to extract model serving and feature pipelines from a tightly coupled monolith into microservices with minimal user impact. Include which components to extract first, API/contract design, testing and verification strategy, data migration steps, rollout plan, and how to measure that migration maintains correctness throughout.

Sample Answer

Requirements & constraints:- Zero/low user-visible downtime, preserve decision correctness, support rollback, keep latency SLOs, cadence: incremental over weeks.

High-level incremental plan (phased):1. Prepare & decouple (1–2 weeks) - Inventory model/feature calls, data sources, dependencies, SLAs. - Add logging/tracing, unique request IDs, and idempotency keys. - Introduce schema registry (Avro/Protobuf) and versioned contracts.

2. Extract read-only feature service (feature-store facade) (2–4 weeks) - Provide a microservice that reads same backing store but exposes stable API for features (getFeatures(entity_ids, feature_list, timestamp?)). - Run in parallel with monolith; implement request shadowing (send traffic to both, ignore responses). - Validate parity via automated comparator.

3. Extract real-time feature pipeline (streaming) (2–4 weeks) - Migrate enrichment/aggregation into a streaming microservice (Kafka/Flink/Beam). - Keep monolith consuming original topic; produce to new topic with version tags. - Use backwards-compatible schema evolution.

4. Extract model serving (stateless inference) (2–4 weeks) - Serve model via REST/gRPC endpoint (predict(request_id, features) -> prediction + metadata). - Start with shadow mode, then canary, then full cutover. - Use model server (TorchServe/TensorFlow Serving) in k8s with autoscaling.

API / contract design:- Use versioned protobuf/gRPC for low-latency model calls; REST for external/backwards compatibility.- Idempotent endpoints: include request_id, model_version, feature_version.- Responses include metadata: model_version, feature_snapshot_version, confidence, explainability token.- Define consumer-driven contracts; store in contract repo; enforce with contract tests in CI.

Testing & verification strategy:- Unit tests for transformations; integration tests for end-to-end pipelines with synthetic data.- Contract tests: provider/consumer tests (PACT or similar).- Shadowing verification: compare outputs (predictions + features) between monolith and new services for large samples; compute statistical parity and sample diff logs.- Backfill tests: run batch inference with historical data and compare distributions and metrics.- Chaos & load tests: validate behavior under failures and peak traffic.

Data migration steps:- Keep single source-of-truth for data initially. Do not write dual-writers until validated.- Streaming: produce new versioned topics; consumers must support both v1/v2.- Batch: run backfill to populate new stores; tag records with feature_snapshot_version.- When dual-write unavoidable: implement fan-out write with write-side verification and eventual reconciliation jobs.

Rollout plan (safety-first):1. Local dev & staging full validation.2. Shadow traffic in prod (100% traffic replicated) — monitor for weeks.3. Canary: route small % (1–5%) live requests to new stack for real responses; gradually increase.4. Feature-flag final cutover; fallback to monolith on error.5. Full cutover, then decommission monolith components after observing stability for defined period.

Measuring correctness & observability:- Functional parity metrics: prediction match rate, feature value match rate, distributional KL-divergence, calibration drift.- Business metrics: downstream conversion/error rates, false positives/negatives.- Latency, error rate, resource usage, p95/p99.- Alerts for thresholds (e.g., >1% prediction mismatch, >10% latency regression).- Continuous data validation (Great Expectations), model performance monitoring, concept/data drift detectors, and automated rollback if SLOs or correctness metrics breach.

Rollback & risk mitigations:- Always keep monolith path available and toggle via feature flags.- Store audit logs linking request_id -> both responses for forensic analysis.- Small incremental changes, strong automated tests & canaries minimize blast radius.

This plan prioritizes low-risk, observable incremental extraction: extract read-only features first (lowest coupling), then streaming/real-time pipelines, then model serving — each validated via shadowing, contract tests, canaries, and monitored with parity and business-level metrics.

EasyTechnical

28 practiced

Explain canary deployment and describe how you would run a canary rollout for a new ML model version. Include which metrics to monitor (both system and model-quality), traffic ramp-up strategy, rollback criteria, and how to detect subtle correctness regressions during the canary.

Sample Answer

Canary deployment is rolling a new version to a small subset of production traffic to validate behavior under real conditions before full release. For an ML model, a safe canary verifies both system stability and model quality using real input distributions.

Rollout plan- Start with a shadow run and small routed traffic: 0% (shadow) → 1% → 5% → 20% → 50% → 100%. Wait at each step (e.g., 30–60 min for latency/ops signals, 24–48 hours for model metrics) and require passing checks before proceeding.- Use automated gates: if any metric crosses its threshold, halt and optionally rollback.

Metrics to monitor- System (SRE): request latency (P95/P99), error rate (5xx), CPU/GPU utilization, memory, throughput, request queue length, tail latency, deployment/serving failures.- Model-quality: primary business metric (e.g., conversion, CTR), offline metrics (accuracy, AUC, F1, precision/recall), calibration (reliability), prediction distribution drift, confidence score distribution, top-k correctness for ranking, loss, and per-slice performance (user cohorts, geos).- Data/feature drift: PSI/KL divergence per feature, missing-value rate, schema changes.

Detecting subtle correctness regressions- Shadow testing: run new model on 100% of traffic in parallel (no serving) and compare outputs to current model.- Per-slice analysis: monitor small cohorts (low-frequency but high-value users), minority classes, and edge cases.- Statistical tests: compute significance (e.g., bootstrapped confidence intervals) for metric deltas; track PSI/KL for feature shifts.- Explainability checks: compare feature attribution distributions (SHAP) for large shifts.- Backtesting: replay recent logged inputs and compare outcomes; raise alerts on distributional/behavioral drift.- Human-in-the-loop: sample and review predictions in ambiguous cases.

Rollback criteria (examples)- System: >2x baseline 95th percentile latency, error rate increase >0.5–1% absolute, resource saturation.- Model-quality: degradation in primary business metric beyond an agreed delta (e.g., >1–3%), statistically significant drop in AUC/F1, increase in false positives/negatives for critical slices, or large calibration drift.- Data issues: sudden feature schema mismatch or PSI > 0.2 for key features.

Additional best practices- Canary duration depends on traffic and event frequency—rare-event metrics need longer observation windows.- Automated monitoring + manual sign-off for critical releases.- Keep canary isolated with easy rollback (traffic switch or feature-flag), and store detailed logs for postmortem.

HardTechnical

38 practiced

Evaluate secure inference approaches when customers require that raw input data remain confidential: homomorphic encryption, secure enclaves (e.g., SGX), and client-side inference. Compare security guarantees, performance overhead, engineering complexity, deployment constraints, and which approach you'd recommend for ML models used in sensitive domains.

Sample Answer

Situation: A customer requires raw inputs remain confidential for ML inference. I'll evaluate homomorphic encryption (HE), secure enclaves (SGX), and client-side inference across security guarantees, performance, engineering complexity, deployment constraints, and a recommendation.

Security guarantees:- Homomorphic encryption: Strong cryptographic guarantee — server never sees plaintext; security relies on HE scheme (e.g., CKKS for approximate arithmetic). Provable under standard hardness assumptions, but careful parameter selection required to meet chosen security level.- Secure enclaves (SGX): Confidential computing — plaintext processed inside hardware-protected memory. Strong protection against OS-level attackers, but susceptible to microarchitectural side-channels (e.g., speculative execution, page-fault attacks) unless mitigations applied.- Client-side inference: Best confidentiality — data never leaves client device. Threat model shifts: protect model IP and ensure client environment integrity.

Performance overhead:- HE: Very high. CKKS inference latency often 10x–10,000x slower and much higher memory; limited supported ops (approximate arithmetic), large ciphertext sizes, batching helps but throughput still low for deep models.- SGX: Moderate overhead (10–50% typical) if model fits enclave; paging into enclave or large RAM needs cause big slowdowns. Network/IO overhead small.- Client-side: Lowest server-side latency; device compute capability dictates speed. Mobile GPUs/TPUs may be sufficient; for large models, may require model compression or on-device accelerators.

Engineering complexity:- HE: High — implement HE-friendly model (quantization, low-depth ops), use libraries (Microsoft SEAL, PALISADE), tune crypto params, manage ciphertext lifecycle.- SGX: High-medium — port runtime into enclave, manage trusted computing base, apply side-channel mitigations, handle attestation workflows.- Client-side: Medium — optimize/convert model (TensorFlow Lite, ONNX), protect model IP (obfuscation, watermarking), handle heterogeneous devices.

Deployment constraints:- HE: Works with untrusted servers and multi-tenant clouds; scales poorly; good for low-throughput or high-sensitivity use cases.- SGX: Requires cloud/hardware supporting enclaves and customer trust in vendor; attestation required; limited memory (enclave size) on some platforms.- Client-side: Requires client hardware capable of inference; updates and model rollback need robust distribution; may not be feasible if clients are thin or constrained.

Recommendation (sensitive domains):- If absolute cryptographic confidentiality with server-side scaling is required and workloads are low-throughput or model is small/HE-friendly: consider HE (CKKS) with careful performance expectations.- If you need stronger performance with server-side processing and can accept a hardware-trust model: use secure enclaves with side-channel mitigations and remote attestation.- For best privacy and latency when clients are capable: client-side inference with model optimization; combine with techniques to protect model IP (secure model partitioning, split inference, or watermarking).

Practical hybrid: use client-side for most cases; for advanced server-side needs use SGX for performance-sensitive paths and HE as a fallback for highest-assurance, low-throughput operations. Include threat-model documentation, monitoring, and regular security reviews.

HardTechnical

32 practiced

Architect a distributed training system for a transformer model that exceeds single-GPU memory and requires multi-node training. Compare data parallelism, model parallelism, and pipeline parallelism: discuss communication overhead, memory usage, hardware requirements, fault tolerance, and how you'd prototype and measure scaling behavior and bottlenecks.

Sample Answer

Requirements & constraints: large transformer that doesn't fit single GPU (model size > GPU memory), target multi-node training with acceptable throughput, <24–48h time budget, need for fault-tolerance and ability to scale to N nodes.

High-level options and comparison:

1) Data Parallelism (DP)- Communication: All-reduce grad sync each step (high bandwidth; scales well with NCCL/IB for many GPUs). Communication size ~model_params per step.- Memory: Replicates full model per GPU; per-GPU memory dominated by activations and optimizer state.- Hardware: High-bandwidth interconnect (RDMA/InfiniBand), many GPUs with enough memory to hold model.- Fault tolerance: Simple — checkpoint and restart; node failure requires re-launch or elastic frameworks.- Best when model fits single GPU but batch-parallelism needed.

2) Model Parallelism (Tensor/Operator Parallelism, TP)- Communication: Fine-grained (tensor slices) between pipeline stages or GPUs within a layer; latency-sensitive and frequent (all-gather/concat).- Memory: Splits parameters across devices — reduces per-device parameter memory but activations still can be large.- Hardware: Topology-aware placement; low-latency links between paired GPUs.- Fault tolerance: Harder; partial state on failed device complicates recovery.- Best for very large layers (e.g., huge embedding or FFN).

3) Pipeline Parallelism (PP)- Communication: Sends activations between stages; micro-batching reduces idle time but increases activation memory unless checkpointing used.- Memory: Each GPU stores subset of layers; activation memory can be reduced with activation checkpointing and recomputation.- Hardware: Balanced compute per stage and bandwidth between stage-adjacent GPUs.- Fault tolerance: Stage failure causes larger recompute; needs checkpointing and orchestration.

Practical hybrid: Use ZeRO (optimizer/state/shard) + tensor parallelism (for linear layers) + pipeline parallelism (stage partitioning) — this is what Megatron-LM/DeepSpeed do. ZeRO reduces optimizer & gradient memory enabling DP-like scaling without full replication.

Prototyping & measuring scaling:- Start single-node multi-GPU prototype: baseline throughput, memory per GPU, and backward/forward time breakdown (use PyTorch profiler + CUDA NVProf/Nsight, NCCL debug).- Measure strong and weak scaling: keep global batch constant (strong) and per-GPU batch constant (weak); plot throughput vs GPUs.- Instrument: per-step time, compute time, comm time (NCCL times), GPU utilization, PCIe/NIC utilization, memory headroom.- Bottleneck detection: if comm_time >> compute_time → optimize with overlap, gradient compression, larger batch, or better network; if compute_time >> comm_time → scale compute (tensor parallel), balance stages; if memory-bound → enable activation checkpointing, ZeRO stage 2/3.- Fault-tolerance tests: simulate node failure, verify checkpoint frequency and restart time; test elastic training (ray/torch.distributed.elastic).

Deployment considerations:- Scheduling (GPU topology-aware), reproducible deterministic seeds, mixed precision (AMP/FP16) to reduce memory, learning-rate scaling with batch size, and automated profiling dashboards.

This design balances memory, communication and hardware trade-offs and recommends iterating: prototype DP + ZeRO first, add tensor and pipeline parallelism when parameter size forces slicing.

EasyTechnical

37 practiced

What is an Architecture Decision Record (ADR) and why is it important for documenting ML architecture and trade-offs? Describe the key elements an ADR should include when recording a decision such as 'use centralized feature store X vs local on-service features' and how ADRs help future migrations.

Unlock Full Question Bank

Get access to hundreds of Architecture and Technical Trade Offs interview questions and detailed answers.

Join thousands of developers preparing for their dream job.