Evaluation of deep, domain specific technical knowledge relevant to the candidate's own team, product, or problem space, whatever that domain is. Candidates should demonstrate subject matter expertise in their problem space and be able to explain core concepts, architectures or system designs, domain specific algorithms or methods, and practical trade offs. The specific domain varies by role and industry: it might be recommendation systems and data platforms for a tech company, claims and underwriting systems for insurance, supply chain and logistics platforms, payment and settlement rails for fintech, clinical or health record systems for healthcare, or content and production pipelines for media. Expect questions on domain specific data flows and integration patterns, versioning and change management strategies, common customer or user workflows, typical pain points in that domain, and how domain constraints shape day to day priorities and decisions. For product facing roles, be ready to explain core product features, typical customer workflows, integration points, and how domain constraints influence product decisions. For engineering, platform, or delivery focused roles, describe how the domain shapes responsibilities and challenges, and outline an approach to initial discovery, diagnosis, and early improvements when picking up an unfamiliar part of that domain. This topic tests both conceptual depth in the candidate's actual domain and the ability to map that domain knowledge to concrete product and engineering decisions.
HardTechnical
62 practiced
Design personalization under EU GDPR constraints: discuss technical product choices to minimize personal data collection while preserving personalization value, including on-device vs server-side models, anonymization, and consent flows.
Sample Answer
Requirements & constraints:- Maximize personalization utility (CTR/engagement lift) while complying with GDPR: data minimization, purpose limitation, consent, right to be forgotten, DPIA where needed.- Prefer designs that avoid storing identifiable data centrally.Approach (high-level):1. Default to privacy-by-design: collect only features strictly necessary; prefer ephemeral, aggregated, or transformed representations.2. Use on-device personalization when feasible; fall back to privacy-preserving server-side methods when needed.Technical choices and patterns:- On-device models: - Small footprint models (e.g., distilled transformers, light FM) trained centrally and pushed as non-identifying weights. Personalization happens locally using raw user signals (clicks, time spent). Pros: minimal personal data leaves device, aligns with GDPR. Cons: limited cross-user learning, model update plumbing. - Update methods: periodic model pushes or federated learning (FL) for continuous improvement.- Federated Learning + Secure Aggregation: - Clients compute gradients/updates; server aggregates encrypted updates. Combine with differential privacy (DP) noise to bound disclosure (epsilon budget). Use secure aggregation to prevent per-client recovery.- Server-side with strong minimization: - Store only pseudonymized IDs (rotating salts) and hashed features; limit retention windows; compute cohort-level signals (cohort-ID) rather than per-user profiles. - Use k-anonymity/cohorting for segmentation; treat cohorts as unit for personalization to reduce identifiability.- Anonymization & transformations: - Feature hashing, binning, quantization; remove direct identifiers; apply DP to aggregates and analytics. - For logs/telemetry, collect only necessary event counts or sketches (HyperLogLog, count-min) to reduce data fidelity.Consent & UX:- Granular, purpose-linked consent: separate consent for personalization, analytics, model improvement. Default to non-personalized or contextual recommendations until opt-in.- Provide clear in-app controls to toggle personalization, view used data, and request erasure. Implement consent-aware pipelines: if no consent, route to non-identifying paths (on-device-only or cohort-based).- Record provenance and consent scope; honor withdrawals by deleting local/ server-side personal artifacts and recomputing cohorts without the user.Governance & metrics:- DPIA and threat modeling for personalization features.- Monitor utility vs privacy: track lift (A/B) per privacy setting, and privacy budget consumption (DP epsilon).- Periodic audits, retention enforcement, and automatic key rotation for pseudonyms.Trade-offs:- On-device/FL reduces risk but can limit model complexity and convergence speed. Adding DP improves privacy but reduces utility—tune epsilon with stakeholder input and A/B tests.- Cohorting reduces identifiability but may be less relevant for hyper-personalized signals.Example: mobile news app- Default: contextual, non-personalized feed.- If user opts-in: local model adapts to reading behavior; gradients sent under FL + secure aggregation + DP for global improvement.- Server stores only cohort stats and model versions; user can revoke consent to wipe local model and exclude their updates from future aggregates.
EasyTechnical
55 practiced
Define model drift and data drift. Provide a simple approach to detect drift for a production classification model, describe what signals you'd monitor, and explain first-line automated actions to take when drift is detected.
Sample Answer
Model drift vs data drift- Data drift (covariate shift): inputs' distribution p(x) change over time (e.g., sensor calibration, seasonality).- Model drift (concept drift): relationship between inputs and labels p(y|x) changes (model no longer maps x→y correctly), often seen as degraded performance.Simple detection approach1. Baseline: store reference distributions and recent labeled performance window.2. Statistical tests per feature: - Continuous: KS-test or Population Stability Index (PSI). - Categorical: Chi-squared or PSI on binned frequencies.3. Prediction/label checks: - Monitor distribution of model scores (softmax/confidence) with KS/PSI. - For labeled samples, track AUC/accuracy, calibration (Brier), and confusion matrix shifts.4. Use rolling windows (e.g., last 1k examples vs reference) and require multiple signals (feature + performance) to reduce false positives.Signals to monitor (real-time + batch)- Feature distributions and schema (missingness, types).- Prediction distribution and confidence.- Label distribution and latency-to-label.- Key performance metrics (AUC, precision/recall) on recent labeled subset.- Data volume, inference latency, and upstream pipeline errors.First-line automated actions- Alerting: send incidents when thresholds/p-value rules triggered.- Canary fallback: route small % to previous stable model while investigating.- Data validation: block or quarantine inputs failing schema checks.- Automated increased labeling: ramp up sampling rate for human review or active learning.- Retrain trigger: if sustained drift + performance drop, kick off automated retrain pipeline (with guardrails).- Rollback: if retrain fails or drift severe, revert to baseline model and restrict new deployments.Rationale: combine distributional tests (fast, unsupervised) with performance checks (ground-truth) to distinguish harmless covariate shifts from harmful concept drift, and apply conservative automated mitigations to maintain availability while gathering evidence for full remediation.
HardSystem Design
59 practiced
Design a production ML platform for multiple teams that includes a multi-tenant feature store, catalog, access controls, cost attribution, and model isolation. Describe high-level architecture, security boundaries, and the operational model for onboarding teams.
Sample Answer
Requirements (clarify):- Functional: multi-tenant feature store (online + offline), central catalog, RBAC/ABAC, model isolation (per-team infra & data), cost attribution per team/workload.- Non‑functional: low-latency reads (ms), scalable writes, strong data lineage, auditability, compliance.High-level architecture:- Ingress: ETL & streaming pipelines (Airflow/KF/Prefect + Kafka) → Features ingestion service.- Feature Store: - Offline store: columnar data lake (Delta/Parquet on S3) for training. - Online store: low‑latency key-value DB (Redis/DynamoDB/Cockroach) with TTL. - Serving layer: standardized Feature API (gRPC/REST) with per-tenant namespaces.- Feature Catalog & Metadata: - Metadata DB (Postgres + Elasticsearch) storing schemas, lineage, freshness, owners, tags, cost center. - UI + API for discovery, dataset docs, sample code snippets.- Access Control & Security: - Central AuthN via SSO (OIDC), AuthZ via centralized policy engine (OPA) implementing RBAC + ABAC policies tied to tenant, team, environment. - Per-tenant encryption keys (KMS) and network isolation (VPCs / namespaces). - Audit logs to immutable store (WORM) and SIEM integration.- Model Isolation & Serving: - Per-team model infra using namespaces in Kubernetes or separate AWS accounts; standardized model-serving stack (KFServing/Triton) with network policies limiting feature access. - Feature access tokens scoped to model identity; mutual TLS between serving and feature API.- Cost Attribution & Observability: - Metering layer emitting usage events (reads, writes, compute) tagged with tenant/team and cost center to billing pipeline (Kafka → billing DB). - Monitoring: Prometheus + Grafana, SLOs for latency/freshness, alerting + incident runbooks.- Data Flow & Lineage: - Ingestion → store; catalog updated automatically with lineage from orchestration metadata; training jobs reference features via catalog IDs to ensure reproducibility.Operational model for onboarding teams:1. Pre‑onboard: Template repo, onboarding checklist (data sensitivity, compliance), tenant creation (namespace, KMS keys, default RBAC).2. Hands‑on onboarding (1–2 weeks): workshop to define canonical features, register initial feature set in catalog, connect pipelines, run sample training using feature API.3. Guardrails: automated tests (schema, freshness, PII scanning), policy checks (via OPA), and review by platform steward.4. Production promote: stomps include canary rollout for online features, quota limits, cost alert thresholds.5. Ongoing: quarterly reviews, cost reports, and playbooks for incident + deprecation.Trade-offs:- Strong isolation (separate accounts) increases operational overhead but maximizes security; namespaces reduce overhead with slightly higher blast radius.- Centralized catalog improves reuse but requires strict governance to prevent leakage.This design balances self‑service for Data Scientists with governance, observability, and per-team isolation needed in production multi‑tenant environments.
HardTechnical
69 practiced
Choose an architecture for a real-time fraud-detection system requiring <10ms inference latency, 99.99% availability, nightly model updates, and discuss feature selection, model architecture (lightweight vs complex), serving infra, and fallback strategies if the model is unavailable.
Sample Answer
Requirements & constraints:- <=10ms end-to-end inference (including feature fetch), 99.99% availability, nightly model updates, high correctness for fraud detection (low false negatives), operational simplicity for audits.High-level architecture:- Edge/API Gateway -> Feature Store (hybrid) + Enrichment services -> Model Serving cluster (low-latency) -> Decision Router -> Logging/Async scorer/Alerting -> Nightly retrain pipeline.Feature selection & engineering:- Favor precomputed, materialized features in a low-latency online feature store (Redis/KeyDB/Feast with Redis backend). Keep per-request joins minimal. Use coarse-grained, high-signal features: user risk score, device fingerprint hash, rolling aggregates (1m/1h/24h), recent behavioral embeddings precomputed offline and refreshed nightly or hourly. Avoid heavy per-request feature computation (no large embeddings or graph traversals synchronously).Model architecture (lightweight vs complex):- Primary production model: lightweight but expressive (e.g., gradient-boosted trees with Treelite or a small optimized feedforward neural net). These run in <1ms when compiled/native and are interpretable-ish for audits.- Secondary/ensemble offline model: heavier (GNNs or larger DL) that runs asynchronously for enrichment, labeling, and to identify drift; used to update nightly model and for investigations.- Justification: GBDTs or tiny nets hit latency and reproducibility targets; complex models add marginal accuracy but break SLOs and availability.Serving infrastructure:- Use compiled model artifacts (Treelite / ONNX Runtime with CPU optimizations) deployed in Kubernetes with node affinity to high-frequency instance types. Autoscale based on request rate plus HPA/Cluster Autoscaler. Place models behind a lightweight inference proxy (Envoy) and a canary rollout pipeline for nightly updates.- Co-locate feature-store cache and model pods in same AZ for minimal network latency; multi-AZ replication for 99.99% availability.- Use persistent logging to Kafka for every request (features, score, decision) for replay and offline retraining.Nightly updates:- Nightly training pipeline produces validated artifact + calibration + explainability report + tests (accuracy, latency, fairness, A/B). Use blue-green deployment with traffic shadowing and gradual promotion after automated checks.Fallback strategies:1. Model unavailable (timeout/failure): fail-open vs fail-closed policy depending on business—prefer fail-secure: apply deterministic rule-based engine (simple rule set: block if high-risk IP or high amount + velocity) or score via cached last-known-model outputs (last-night model) kept in memory. These execute <1ms.2. For partial feature-store outage: substitute with coarser features (global priors, last-known aggregates) or default conservative scores increasing fraud suspicion to be safe.3. Circuit breaker + bulkhead isolation to prevent cascading failures; metrics trigger immediate rollback to previous model and alert on-call.Monitoring & ops:- Real-time SLO dashboards (latency P99, error rate), shadow traffic comparisons, data-drift detectors + feature-distribution alerts, automatic rollback on detection.- Regular audit logs for compliance and an explainability interface for investigators.Trade-offs:- Prioritize latency and availability over marginal accuracy from heavy models. Invest in high-quality feature engineering and deployment/validation to recover most performance cost-effectively.
HardTechnical
51 practiced
You need to measure causal effect of a new recommendation algorithm on revenue. Describe how you'd design an experiment or causal pipeline (randomization, assignment unit, uplift modeling, covariate balance checks), how you'd handle interference, and how to surface trustworthy causal estimates to PMs.
Sample Answer
First, clarify the estimand: average treatment effect (ATE) of the new recommender on incremental revenue (per-user or per-visit) over a pre-specified horizon (e.g., 28 days). Then design an experiment to identify it.Experiment design- Assignment unit: randomize at the user level if exposure is per-user. If recommendations are shared (household, session, or feed-level) or there is strong spillover, randomize at the smallest non-interfering cluster (household, device, or browser-cookie) — cluster randomization.- Randomization: stratified randomization by key covariates (past revenue decile, recency, device) to improve precision. Use blocked random assignment with a fixed seed and log treatment assignment.- Treatment definition: intent-to-treat (ITT) = users assigned to algorithm; track exposure dosage (how many recommended items shown & clicked).Covariate balance & diagnostics- Pre-check balance on historical revenue, session counts, demographics using standardized mean differences and KS tests. Report balance tables and cumulative distribution plots.- Pre-register primary metric (incremental revenue) and secondary metrics (CTR, add-to-cart, return rate). Use pre-specified variance estimator accounting for clustering.Interference & SUTVA violations- If interference likely (e.g., recommendations affect product popularity), use cluster randomization or design network-aware experiments (graph cluster randomization). Measure spillovers via partial population experiments: vary treatment saturation across clusters and estimate direct and indirect effects.- For marketplace-wide demand shifts, include time-series controls and run difference-in-differences (DiD) comparing treated vs. control pre/post.Analysis & causal pipeline- Primary estimate: ITT using regression adjustment (ANCOVA) to improve precision: revenue ~ treatment + pre-period revenue + strata fixed effects; cluster-robust SEs.- If compliance/exposure varies, estimate Local Average Treatment Effect (LATE) using instrument: assignment as instrument for actual exposure.- Heterogeneous effects: train uplift models (causal forests / meta-learners) with honest splitting to estimate conditional average treatment effects; validate with cross-fold calibration and policy risk (expected revenue under decision rule).- Robustness: bootstrap, permutation tests, sensitivity to unobserved confounding (Rosenbaum bounds), alternate specifications.Surface results to PMs- Executive summary: concise ATE with 95% CI, percent lift, p-value, sample size, and time window.- Visualization: uplift curve, cumulative incremental revenue over time, distribution of individual treatment effects, calibration plot for uplift model.- Actionable guidance: recommended rollout thresholds (e.g., target segments with positive CATE), expected revenue impact at scale, risks (spillovers, cannibalization), and recommended monitoring metrics and safety guardrails (realtime drift detection).- Transparency: provide pre-registration, randomization logs, data pipeline lineage, and reproducible analysis notebook so PMs and engineers can trust estimates.
Unlock Full Question Bank
Get access to hundreds of Domain and Product Technical Knowledge interview questions and detailed answers.
