Evaluation of deep, domain specific technical knowledge relevant to the candidate's own team, product, or problem space, whatever that domain is. Candidates should demonstrate subject matter expertise in their problem space and be able to explain core concepts, architectures or system designs, domain specific algorithms or methods, and practical trade offs. The specific domain varies by role and industry: it might be recommendation systems and data platforms for a tech company, claims and underwriting systems for insurance, supply chain and logistics platforms, payment and settlement rails for fintech, clinical or health record systems for healthcare, or content and production pipelines for media. Expect questions on domain specific data flows and integration patterns, versioning and change management strategies, common customer or user workflows, typical pain points in that domain, and how domain constraints shape day to day priorities and decisions. For product facing roles, be ready to explain core product features, typical customer workflows, integration points, and how domain constraints influence product decisions. For engineering, platform, or delivery focused roles, describe how the domain shapes responsibilities and challenges, and outline an approach to initial discovery, diagnosis, and early improvements when picking up an unfamiliar part of that domain. This topic tests both conceptual depth in the candidate's actual domain and the ability to map that domain knowledge to concrete product and engineering decisions.
HardSystem Design
90 practiced
Design a compliance-first data platform for building and serving AI models on regulated financial data. Cover encryption and key management, role-based access control, anonymization/pseudonymization approaches, retention and erasure policies, immutable audit logs, explainability tools, and how to enable safe experimentation while minimizing exposure.
Sample Answer
Requirements & constraints:- Functional: train, validate, deploy AI models on regulated financial data; support experiments, feature stores, explainability, monitoring.- Non-functional: compliance with GDPR/GLBA/PCI-DSS, low-latency inference, strong confidentiality, tamper-evident audit, provable erasure, least privilege.High-level architecture:- Ingest → Secure Data Lake (encrypted at rest) → Data Preparation & Anonymization Service → Feature Store (encrypted) → Training Cluster (isolated) → Model Registry → Serving + Explainability & Monitoring.- Control plane: IAM/RBAC, Policy Engine, KMS, Audit Log Store (WORM), Compliance UI.Encryption & Key Management:- All data encrypted at rest and in transit (TLS 1.3). Use envelope encryption: data keys (DEKs) per dataset/tenant, encrypted by a central KMS.- KMS: enterprise HSM-backed (Cloud HSM or on-prem Thales) for root keys; automated key rotation, split-key multi-approval for destructive operations (rotation/expiration).- Key access audited; use short-lived session keys for compute nodes. For model artifacts, encrypt with model-specific DEKs.Role-Based Access Control:- Fine-grained RBAC + ABAC (attributes: purpose, environment, dataset sensitivity, user role).- Roles: Data Steward, Compliance Officer (read audit), ML Engineer (access pseudonymized data/feature read), Model Owner (deploy), Production Service Account (inference).- Enforce via gateway/middleware that checks signed requests and purpose tags; deny-by-default.Anonymization / Pseudonymization:- Two-tier approach: - Pseudonymization for most ML tasks: replace identifiers with consistent cryptographic pseudonyms (HMAC with per-dataset secret) so features remain linkable within scope but irreversible without key. - Anonymization / aggregation for high-risk outputs: k-anonymity + l-diversity + differential privacy for aggregates.- Provide a reversible pseudonymization escape hatch only to Data Stewards using key escrow with multi-party approval and strict audit.Retention & Erasure Policies:- Policy engine attaches retention metadata to datasets/models.- Erasure: implement authenticated deletion of DEKs for crypto-shredding (makes data unreadable) plus physical deletion workflows; log proof-of-deletion.- Versioned data: tombstone markers propagate to feature store and model inputs; periodic jobs remove expired artifacts.- Retention rules enforced automatically on snapshot lifecycle and model retraining pipelines.Immutable Audit Logs:- Write-only, append-only audit logs stored in WORM storage and replicated to an external ledger (blockchain-style or signed append-only stream).- Log access decisions, key operations, data joins, model training runs, inference requests, explanation requests, approval workflows.- Provide cryptographic signing and periodic root-of-trust notarization; allow compliance queries and tamper-evident proofs.Explainability Tools:- Integrated explainability service accessible only with proper authorization: local (SHAP/Integrated Gradients) and global explanations, counterfactual generation, feature importance dashboards.- Explanations sanitized to avoid leaking sensitive training data (use privacy-preserving explainability: limit granularity, apply DP noise).- Store explanation requests in audit logs; require justification tag for sensitive cases.Safe Experimentation while Minimizing Exposure:- Sandbox environments with synthetic and pseudonymized datasets. Synthetic data generator trained to match distributions but validated for non-reidentifiability.- Differentially private training options (DP-SGD) available; per-experiment privacy budget tracked centrally.- Shadow mode / canary deployments: inference runs in parallel without acting on real accounts.- Feature flags & access gates: experiments require tickets & automated risk assessment; risky experiments need approval from Data Steward + Compliance.- Resource isolation: ephemeral training clusters in VPCs with no external egress except to approved artifact stores.- Monitoring & automatic kill-switch: anomaly detectors on model drift, unusual data access patterns, or high explanation uniqueness score.Operational and governance practices:- Threat model and risk register reviewed annually; data classification matrix.- CI/CD guarded by policy-as-code (OPA) checks: dataset sensitivity, model lineage, privacy labels, required approvals.- Regular audits, penetration tests, and tabletop exercises. SLA for compliance requests (erasure, data subject requests).- Education: guardrails and templates for privacy-preserving model development; default to safest config.Trade-offs:- Strongest guarantees require additional latency and developer friction (HSM, approvals, DP noise). Use risk-based controls to balance productivity vs exposure.This design provides cryptographic isolation, auditable controls, privacy-preserving data handling, and safe experimentation paths while meeting regulatory expectations for financial data.
HardTechnical
56 practiced
Design a model governance framework for a company-wide AI platform. Include model risk tiers, approval gates, documentation and reporting checklists, performance thresholds, ownership and roles, auditing cadence, retirement criteria, and tooling suggestions to automate compliance enforcement.
Sample Answer
Requirements & scope:- Applicable to all company AI models (research, prod, third-party APIs); balance speed vs risk; support regulated lines (finance, healthcare).Model risk tiers (examples):- Tier 1 — High risk: decisions affecting safety, financial/legal outcomes, PII exposure, regulated domains.- Tier 2 — Medium risk: customer-facing recommendations, content generation with moderate impact.- Tier 3 — Low risk: internal analytics, prototyping, non-decision support tools.Approval gates & lifecycle checkpoints:- Design review (architecture, data provenance, threat model) — required for Tier 1/2.- Pre-deployment validation (robustness, fairness, explainability tests) — automated + manual signoff.- Security & privacy review — mandatory for Tier 1.- Production readiness (monitoring, rollback, SLAs) — signoff by Model Owner and Compliance.- Post-deployment periodic re-certification.Documentation & reporting checklist (must-have):- Model card: purpose, inputs/outputs, training data summary, lineage.- Data lineage & consent records.- Performance metrics: primary/secondary metrics, thresholds, confidence calibration.- Fairness & bias assessment results, remediation steps.- Threat model, adversarial testing results.- Explainability artifacts (saliency maps, SHAP, prompt templates).- Deployment plan, rollback criteria, owner contacts.Performance thresholds & monitoring:- Define SLOs per model (accuracy/F1, latency, hallucination rate, fairness delta).- Alert thresholds: e.g., >3% drop vs baseline or >2x increase in unknown input rate triggers incident.- Continuous monitoring: data drift, concept drift, input distribution, downstream business KPIs.- Canary + shadow deployments for Tier 1.Ownership & roles:- Model Owner (business accountable)- Model Lead (AI Engineer — technical owner)- Data Steward (data quality & lineage)- Compliance Officer (policy/approvals)- Security Engineer (threat model & access control)- QA/Validation Team (independent testing)- Audit Owner (periodic audits)Auditing cadence:- Tier 1: quarterly audits + continuous automated checks- Tier 2: semi-annual audits- Tier 3: annual lightweight review- Triggered audits after incidents or major data changesRetirement criteria:- Sustained underperformance vs thresholds for N days- Obsolescence: better model available with clear uplift- Regulatory changes or unacceptable risk profile- Lack of owner/maintenance for defined periodTooling & automation suggestions:- Model registry (e.g., MLFlow, Feast + metadata store) for lineage and versioning- CI/CD pipelines with automated tests (unit, fairness, robustness) using GitOps- Monitoring stack: Prometheus + Grafana + Seldon/Prometheus exporters or an MLOps platform (Databricks, Tecton, Cortex)- Drift & explainability automation: WhyLabs, Fiddler, Arize- Access control & secret management: Vault, IAM policies- Governance UI for approvals and audit trails (custom or tools like Domino, Evidently)- Automated policy engine to block deployments that fail required checksRationale & trade-offs:- Tiers concentrate governance effort where impact is highest, preserving agility for low-risk work.- Mix of automated enforcement and human sign-off balances speed and accountability.- Continuous monitoring + clear owner assignments minimize blind spots and support rapid incident response.Implementation approach:- Pilot with 2 Tier 1 models for 3 months, iterate checklists and automation, then roll out company-wide with training and playbooks.
MediumTechnical
51 practiced
Write the key elements of an incident runbook for when model performance drops in production. Include detection signals, triage checklist, immediate mitigation actions (rollbacks, feature toggles), criteria for escalation, communication templates, and post-incident analysis steps.
Sample Answer
Summary: A concise incident runbook for production model-performance degradation covering detection, triage, mitigation, escalation, communications, and post-incident analysis.Detection signals:- Automated alert thresholds: sudden drop in key metrics (accuracy, F1, AUC, BLEU, RMSE) beyond X% or outside SLA.- Input-distribution drift: population stats, feature drift (KL divergence > threshold), increased OOD rate.- Latency/throughput anomalies and error spike (nan/inf predictions).- Business KPIs: conversion, revenue, complaint rate.- Canary/blue-green test failures and shadow-run discrepancies.Triage checklist:- Confirm alert validity: reproduce on recent logs and canary dataset.- Scope: percentage of traffic affected, models/services, regions, versions.- Root-candidates: data drift, upstream feature change, label skew, model weight corruption, infra (gpu/memory), config/feature-flag change, package dependency update.- Collect: model version, commit, serving logs, recent deploys, input samples, metrics timeline.Immediate mitigation actions:- Rollback to last known-good model/version (automated if safety-critical).- Activate feature toggles to route traffic to canary/backup or degrade to simpler rule-based logic.- Rate-limit or pause non-essential inference pipelines (batching mismatch).- Enable shadowing to compare or divert a fraction while keeping production stable.- If root is data: apply input sanitization or filtering rules.Criteria for escalation:- Business impact exceeds threshold (revenue loss, legal/SLA breach).- >X% users affected or >T minutes unresolved with no mitigation.- Data leak, PII exposure, or safety-critical mispredictions.- Need for cross-org support (infra, data, SRE, legal, PM).Communication templates:- Initial alert (to Slack/pager, subject): “URGENT: Model degradation detected — Model: [name], Impact: [metric drop], Scope: [traffic%], Action: Investigating. Ops lead: [name].”- Status update (every 30–60 min): “Status: Investigating / Mitigated / Rolled back. What we know, what we did, next steps, ETA.”- Incident resolved: “Resolved: Root cause, actions taken, user/metric impact, immediate mitigations, next steps.”- Stakeholder summary (post-mortem): one-paragraph impact + timeline + remediation plan.Post-incident analysis:- Timeline reconstruction: alerts → triage → mitigation → resolution with timestamps.- Root cause analysis using logs, data snapshots, and model artifacts.- Quantify business impact and user scope.- Remediation: automated tests (canary, synthetic tests), better monitors (drift detectors, on-prem holdouts), guardrails (pre-deploy data checks, checksum for model artifacts), improved rollback runbooks, runbook rehearsal.- Action items with owners and deadlines; verify fixes in staging then in staged rollouts.- Share postmortem (blameless) with metrics, lessons learned, and follow-up verification plan.
MediumSystem Design
61 practiced
Design API contracts and access controls for an internal model-as-a-service platform used by multiple internal teams. Cover authentication/authorization, per-tenant rate-limiting, model version routing, SLA tiers, billing/quotas, multi-tenant isolation, and observability endpoints you would expose to consumers.
Sample Answer
Requirements (short): internal Model-as-a-Service for multiple teams supporting per-tenant auth/authorization, rate-limits, model-version routing, SLA tiers, billing/quotas, strong multi-tenant isolation, and consumer-facing observability.High-level approach: provide REST/gRPC API gateway + auth service + tenant metering/billing + model router + model execution pool. Tenant metadata (SLA tier, quota, allowed models/versions, billing account) stored in config DB. Use centralized policy engine (OPA) for authorization.API contract (examples):- POST /v1/infer - Headers: Authorization: Bearer <JWT>, X-Tenant-ID, X-Model: <model-name> (optional), X-Model-Version: <v1|traffic:weight> - Body: { input: {...}, request_id?: string, explain?: boolean } - Response: { request_id, model, version, output, meta: {latency_ms, cost_units} }- GET /v1/models - Query: ?tenant_id= - Response: allowed models + versions + default version- GET /v1/usage - Returns tenant's usage (tokens, compute_seconds), current spend, quota_remaining- POST /v1/admin/route - For ops: set version routing weights: { model, routes: [{version, weight}] }Authentication & Authorization:- Auth: OAuth2 JWT issued by internal identity provider. JWT includes tenant_id, scopes, SLA_tier.- Short-lived service tokens; support mTLS for high-trust tenants.- Authorization: RBAC + attribute-based checks via OPA: - Validate tenant_id in JWT matches X-Tenant-ID - Check scope for actions (infer, admin, billing) - Enforce model-level allow-list per tenantPer-tenant rate-limiting & quotas:- Rate-limit strategy: hierarchical token-bucket - Global per-tenant QPS and burst (based on SLA) - Per-model soft limits - Throttling response: 429 with Retry-After and current bucket state- Quotas tracked in metering service: compute_seconds, input_tokens, output_tokens; quotas reset monthly or rolling window.- Grace/overage: allow configurable overage for higher SLA with billing multiplier; otherwise hard block.Model version routing:- Default routing set per-tenant or global with weights.- Client can request version via header; if not allowed, gateway honors default routing.- Support canary weights: traffic splitting at router level using consistent hashing + weight table.- Sticky routing for session affinity via request_id cookie/header.SLA tiers & behavior:- Bronze: best-effort, lower QPS, higher latency SLOs, no overage allowed.- Silver: guaranteed baseline QPS, reduced latency SLO, limited overage at multiplier.- Gold: high QPS, priority scheduling, dedicated model pool options, 99th-percentile latency SLOs, cost-center billing with surge protection.- Service responds with SLA metadata headers: X-SLA-Tier, X-SLO-Latency-msBilling / quotas:- Metering service emits events per request: tenant_id, model, version, compute_ms, tokens_in/out, cost_units.- Pricing rules defined per model and per SLA tier; cost_units aggregated to monthly invoice.- Expose billing endpoints: GET /v1/billing/preview, POST /v1/billing/alerts to set thresholds.- Real-time alerts via webhook when quota thresholds hit.Multi-tenant isolation:- Logical isolation by tenant namespace: separate model permissions, configs, per-tenant routing rules.- Strong data isolation: encrypt data in transit and at rest; ephemeral storage in execution node; scrub logs of sensitive inputs unless tenant opts-in.- Compute isolation: schedule tenants to separate Kubernetes namespaces/node pools or use cgroups/gpu isolation for high-SLA tenants.- Audit logs per-tenant with RBAC access; option for tenant-owned VPC peering for Gold customers.Observability endpoints exposed to consumers:- GET /v1/observability/metrics?granularity=1m&range=7d - Returns time-series: requests, latency percentiles, error rates, model-specific throughput, cost_units- GET /v1/observability/traces?request_id= - Returns trace and model routing path (router, backend node, model version)- GET /v1/observability/explanations?request_id= (if explain enabled) — model outputs + deterministic metadata (tokens, attention info) subject to privacy rules- Webhooks/streams: push usage and anomaly alerts (latency spikes, elevated error-rate)- Prometheus-compatible metrics endpoint for internal dashboards (read-only)- SLAs: /v1/health returns component health and SLA burn rateObservability best-practices:- Correlate logs/traces/metrics by request_id and tenant_id.- Expose cost/unit attribution per request to help teams optimize.- Retention/PII: default scrub of inputs; opt-in retention policy for debugging.Trade-offs & reasoning:- JWT + OPA gives flexible fine-grained authorization without per-call DB hits (cache policies).- Token-bucket handles bursts well; hierarchical limits allow global enforcement and per-model nuance.- Routing weights in control-plane enable safe canaries; sticky routing avoids cold-start variability.- Logical isolation reduces resource overhead vs full physical separation; offer dedicated pools for strict isolation at higher cost.- Metering per-request provides accurate billing but adds latency; implement async batch exporter to billing with idempotency.Edge cases:- Network partition between gateway and router: return 503 with explanatory headers.- Model upgrade rollback: router supports quick weight rollback and traffic mirroring for offline validation.- Denial-of-service: combine rate-limits with anomaly detection and automated tenant throttling/quarantine.This design balances security, tenant fairness, observability, and operational flexibility while enabling per-tenant SLAs, usage-based billing, and safe model rollout.
MediumTechnical
59 practiced
Describe a cost-optimized training pipeline for large transformer fine-tuning or pretraining that uses spot instances, mixed precision, gradient accumulation, sharded checkpoints, and autoscaling. Focus on checkpoint frequency, resume strategies, fault tolerance, and how to balance lower cost vs longer wall-clock time.
Sample Answer
Requirements & goals:- Minimize cloud cost using spot/preemptible GPUs while keeping training progress safe and reasonable wall‑clock time.- Maintain reproducible resume, low recovery overhead, and autoscaling responsiveness.Pipeline design:- Use mixed precision (AMP) + gradient accumulation to keep per‑step memory low and maximize GPU utilization on smaller spot VMs. Batch = microbatch * accum_steps tuned to keep stable BatchNorm/optimizer behavior.- Run on spot instances behind a cluster manager (Kubernetes + Karpenter / AWS Batch / GCE Managed Instance Groups) with node groups for GPU types and fallback to on‑demand for critical master/parameter servers.Checkpointing & frequency:- Two-tier checkpoints: 1) Frequent lightweight optimizer/partitioned state snapshots every N steps (e.g., 100–500 steps) saved as sharded checkpoints (per-rank) to distributed object storage (S3/GCS). These are small and allow quick resume. 2) Less frequent full consistent checkpoints (model + optimizer + RNG + tokenizer/state) every M*N steps or hourly (e.g., hourly or every 5–10GB written) that merge shards and validate integrity.- Save async: each rank writes its shard in parallel; a coordinator writes manifest (epoch/step/RNG hashes) to enable atomic resume.Resume strategies & fault tolerance:- On preemption, autoscaler replaces node; training job attempts fast resume from latest lightweight shard. Use a short grace period / preemption hook (if available) to trigger immediate checkpoint. Maintain a WAL of last applied gradient step for idempotency.- Use training framework support (DeepSpeed ZeRO, FairScale, PyTorch FullySharded) to load sharded checkpoints directly to new ranks, avoiding full merge. Implement coordinator logic to remap shards to new ranks using rank-to-shard mapping persisted in manifest.- Validate checkpoints via checksum; if latest is corrupt, fallback to previous one automatically.Autoscaling & balancing cost vs wall‑clock:- Prefer high spot utilization with policy: allow up to X% spot capacity; keep a small on‑demand anchor (1–2 nodes) for checkpoint coordination and leader election to avoid full job loss.- Tune checkpoint frequency vs cost/time: more frequent checkpoints increase I/O cost and slightly longer step time, but reduce wasted compute on preemption. Use expected spot interruption rate to set N: target expected wasted work << cost of extra checkpointing. Example: if average spot lifetime is 12 hours, checkpoint hourly; if 1–2 hours, checkpoint every 5–15 minutes.- Use longer gradient accumulation when spot nodes are heterogeneous or small to amortize communication overhead; but watch learning dynamics — scale learning rate accordingly.- Use autoscaler warm pools and fast provisioning (instance templates, GPU images) and prefer instance types with lower preemption rates.Operational best practices:- Monitor interruption metrics, I/O throughput, and checkpoint latency; adapt N dynamically.- Test full recovery regularly.- Compress or use multipart uploads for large shards; TTL prune older checkpoints.- Cost controls: spot bidding strategy or use provider spot fleets with capacity-optimized allocation.Trade-offs:- Aggressive checkpointing -> higher storage/I/O cost, lower wasted compute.- Sparse checkpointing -> lower overhead, higher recomputation after interruption, possible longer wall time.- Anchor on‑demand nodes increase cost but improve resilience and reduce recovery complexity.This approach yields a cost‑optimized, fault‑tolerant pipeline that leverages mixed precision, gradient accumulation, sharded checkpoints, and autoscaling while controlling the tradeoff between lower cost and longer wall‑clock time.
Unlock Full Question Bank
Get access to hundreds of Domain and Product Technical Knowledge interview questions and detailed answers.
