HardSystem Design

84 practiced

As a senior Data Scientist at Google, you're asked to define a 12-month data science roadmap for improving search relevance using machine learning across multiple product teams. Outline a multi-phase roadmap that includes hypothesis discovery, R&D, experimentation at scale, data collection and labeling strategy, feature-store design, infrastructure investments, cross-team coordination, measurable success metrics, and risk mitigation strategies.

Sample Answer

Requirements & constraints (clarify): improve overall search relevance across multiple products (web, mobile, verticals) within 12 months while minimizing user disruption, preserving latency SLAs, and enabling reproducible experiments and cross-team reuse.

High-level 12‑month, 4-phase roadmap

Phase 0 – Discovery & Hypothesis (Months 0–2)- Run stakeholder workshops across product teams to collect pain points, KPIs, traffic slices, and constraints.- Conduct data audit: freshness, coverage, label availability, feature gaps, instrumentation gaps.- Produce prioritized hypothesis backlog (e.g., better query understanding, recency modeling, personalization signals) scored by expected impact × feasibility.- Quick-win pilots: offline evaluation of top 3 hypotheses using historical logs and counterfactual metrics.

Phase 1 – R&D & Labeling Strategy (Months 2–6)- For each prioritized hypothesis run focused R&D: small-model prototypes (BERT, lightweight ranking nets) and ablation studies.- Labeling strategy: - Use hybrid approach: existing implicit feedback (clicks, dwell), targeted human labels for ambiguity, and weak supervision for scale. - Employ active learning to prioritize labeling high-uncertainty examples. - Build annotation guidelines and inter-annotator agreement targets (Cohen’s kappa > 0.7).- Define offline metrics aligned to online success (NDCG@k, MRR, calibrated CTR lift, latency impact).

Phase 2 – Experimentation at Scale & Feature-Store Design (Months 6–10)- Feature store: - Centralized, versioned feature store (online + batch) with clear ownership and schemas. - Support derived features, freshness guarantees, and lineage metadata. - Tech choices: BigQuery/Spanner for batch, Redis/Bigtable for low-latency online reads; integrate with TF/TFX or Vertex AI pipelines.- Experimentation platform: - Instrumentation for safe online experiments: randomized bucket routing, user-level consistency, stratified sampling, and logging for counterfactual evaluation. - Implement ramping & monitoring: sequential testing, risk-aware rollouts, automatic rollback thresholds. - Run large-scale A/B and interleaving experiments across verticals; use multi-armed bandits for rapid exploration where appropriate.

Phase 3 – Productionization & Cross-Team Rollout (Months 10–12)- Harden models: model governance, CI/CD for models, canary deployments, feature drift detection.- Cross-team coordination: - Establish cross-functional working group (DS, infra, SRE, PMs, UX) with weekly syncs and shared OKRs. - Documentation, SDKs, and onboarding playbooks so teams can adopt feature store and experimentation primitives.- Measure and iterate: evaluate success metrics, scale up proven models, retire low-impact hypotheses.

Measurable Success Metrics- Primary business metrics: relative NDCG@10 uplift, net-user engagement (dwell time), query success rate, and query-to-click conversion.- Safety & quality: latency within SLA, percentage of experiments that pass predefined quality gates, annotation agreement.- Operational: feature store adoption (teams using it), model deployment frequency, mean time to rollback.

Infrastructure investments- Versioned feature store (batch + low-latency online), streaming ETL (Beam/Dataflow), model training infra (TPU/GPU clusters, TFX/Vertex), experiment telemetry pipeline (low-latency logging + BigQuery), monitoring/dashboards (Grafana/Looker), and labeling platform integrated with active learning.

Risk mitigation- Causal confounding & feedback loops: use counterfactual evaluation and interleaving to prevent positive-feedback bias.- User harm / safety: conservative rollout with sentinel test queries, human-in-loop review for high-risk changes.- Data drift & model staleness: automated drift detectors, periodic retraining schedules, and fallback to baseline ranker.- Cross-team alignment risks: clear SLAs, API contracts, and a governance board to arbitrate priorities.

Why this approach- Sequential but overlapping phases let us deliver value early (quick wins), de-risk with offline validation, scale experiments safely, and institutionalize assets (feature store, infra, runbooks) so improvements are sustainable across teams.

MediumTechnical

101 practiced

Google encourages experimentation and learning from failure. How do you structure a blameless post-mortem for an ML model failure so the team learns effectively, prevents recurrence, and shares outcomes cross-functionally? Include what artifacts you produce, metrics to capture, and how you track remediation.

HardTechnical

90 practiced

Design a program to increase the overall impact of data science at Google by systematically surfacing high-leverage opportunities (for efficiency, revenue, or quality). Describe the signals and metrics you'd use to identify opportunities (for example volatility, revenue sensitivity), the tooling to enable discovery and rapid experimentation, governance to prioritize, and KPIs to measure program success.

Sample Answer

Goal: create a repeatable program that systematically finds, prioritizes, and executes high-leverage DS opportunities across Google to increase efficiency, revenue, or quality.

Discovery signals & metrics- Business sensitivity: estimated revenue/MTTR/MAU exposure per unit change (dollars or % per 1% metric shift).- Volatility & error surface: time-series volatility, anomaly frequency, and outage correlation — high volatility often hides leaks.- Impact density: number of affected users/systems × effect size.- Cost-to-serve: compute/storage/person-hours per pipeline; high cost + high variance = optimization candidate.- Decision frequency & automation gap: decisions made manually or daily where automatable models yield ROI.- Observability signals: drift rate, model performance decay, alert rate, and downstream KPI coupling.

Tooling to enable discovery & rapid experimentation- Central Opportunity Catalog: searchable index (signals, owners, tags, impact estimate, maturity).- Automated scanners: pipelines that compute the signals above weekly; produces ranked candidate lists.- Lightweight experiment platform: templated A/B and canary workflows, feature toggles, online/offline eval harness, standardized metric dashboards.- Reusable DS primitives: shared feature store, model infra (TFX), monitoring libs, pre-built uplift estimators.- Notebook-to-prod templates and one-click deployment + rollback.

Governance & prioritization- Cross-functional Opportunity Council: product, eng, finance, legal; meets weekly to triage top N.- Prioritization rubric: expected value (EV) = impact × confidence − cost; penalize technical debt and compliance risk.- Time-boxed incubation lanes: Quick wins (2–4 weeks), Mid (1–3 months), Research (3+ months).- Readiness criteria for production: reproducible experiment, fairness/compliance check, monitoring hooks.

KPIs to measure program success- Net impact: cumulative realized revenue/efficiency/quality delta attributable to program.- Conversion rate: % candidates that reach production from discovery.- Time-to-impact: median time from discovery to measurable impact.- ROI per project: impact / total cost.- Coverage: % business domains scanned and instrumented.- Model health: avg drift reduction and sustained performance gains.

Example flow1. Automated scanner ranks a spike in ad auction variance with high revenue sensitivity.2. Catalog flags owner; Council approves quick experiment.3. DS team uses templates to run an A/B model improvement, deploys via one-click canary.4. Monitor shows 0.8% revenue lift; program records ROI, documents artifact in catalog, and scales.

This program combines automated signal generation, low-friction experimentation, and a governance loop focused on expected value and delivery velocity to systematically increase DS impact.

HardTechnical

86 practiced

Propose design improvements for an experimentation platform to support sequential and adaptive experiments (for example multi-armed bandits) at Google scale while preserving statistical validity and guardrails. Discuss allocation algorithms, stopping rules, handling multiple comparisons, deterministic logging for reproducibility, and developer-facing tooling you would provide.

Sample Answer

Requirements & constraints:- Support sequential/adaptive allocation (contextual & non-contextual bandits) at Google-scale: millions of users, low-latency online allocation, offline reproducibility, strong Type I error & business guardrails, multiple concurrent experiments.

High-level design:- Hybrid architecture: online low-latency allocator (stateless microservice + cached policies) + offline experimentation engine for evaluation, logging, and long-running analysis. Central metadata/catalog for experiments, versions, and deterministic seeds.

Allocation algorithms:- Offer pluggable policy library: epsilon-greedy, Thompson Sampling (Beta/Bernoulli, Gaussian), Contextual LinUCB/NeuralBandit. Default to Thompson Sampling with conservative priors + forced exploration schedule. Provide “safe” variants (conservative baseline guarding, risk-aware utility functions) and constraints API (traffic caps, per-cohort limits).

Stopping rules & statistical validity:- Use pre-specified adaptive-analysis framework: group-sequential boundaries or alpha-spending (e.g., Lan-DeMets) with correction for adaptive allocation. For Bayesian policies, translate posterior probability thresholds to frequentist guarantees via simulation-based calibration and pre-computed operating characteristics. Mandate pre-registration of primary metric & stopping plan.

Multiple comparisons:- Hierarchical test registry: declare families and hypotheses; use FDR control (Benjamini-Hochberg) for many related tests; for many adaptive experiments, use multiplicity-aware p-value combination or sequential BH procedures. Provide permutation-based or bootstrap null estimation when assumptions fail.

Deterministic logging & reproducibility:- Log deterministic decision inputs: user id hash, experiment config id & version, RNG seed, policy snapshot, feature vector, allocation decision, exposure time, and outcomes. Persist logs in append-only immutable store with retention and snapshotting. Provide tooling to replay allocations offline by loading policy snapshot + deterministic RNG to reproduce historic assignments.

Developer-facing tooling:- UI for experiment design, pre-registration, simulation sandbox (inject traffic, run many synthetic environments to estimate regret, false-positive rates), automatic operating-characteristics reports, rollout manager (safeguards, kill switches), cohort management, and visualization dashboards. Provide SDKs (py/scala/java) with client-side deterministic hashing and built-in telemetry. Auto-generate recommended priors, exploration schedules, and sample-size / stopping guidance from historical data.

Governance & guardrails:- Enforce hard caps on business-critical metrics; require canary rollout for high-risk features; automated alerts for metric degradation with causal attribution. Require approvals for policy changes affecting allocation fairness.

Trade-offs:- Conservative statistical controls reduce speed of detection; simulation-driven calibration and safe defaults balance speed vs. risk. Complexity is increased but modular design keeps online path simple and auditable.

This design provides scalable adaptive experimentation while preserving reproducibility and statistical rigor, plus developer tools to safely iterate.

MediumTechnical

79 practiced

You find a change in upstream ETL that will bias a core product metric used for weekly executive reporting. Explain how you would triage the issue, quantify the impact, communicate the risk to stakeholders, coordinate a mitigation plan with data engineering, and adjust analytics until the upstream pipeline is fixed.

Sample Answer

Situation: While preparing weekly executive metrics I discovered a sudden drift in a core KPI (weekly active retention) after an upstream ETL change — counts dropped ~12% vs. trend with a concurrent change in a join key used by the pipeline.

Task: My goal was to triage root cause, quantify business impact, notify stakeholders clearly, coordinate a fix with Data Engineering, and provide an accurate interim metric for reporting until the pipeline was corrected.

Action:- Triage & quantify - Reproduced the issue in a sandbox by running the new ETL vs. previous snapshot; confirmed discrepancy tied to a changed column type and null handling. - Measured impact: 12% absolute drop in metric, concentrated in 3 of 8 regions; estimated effect on KPIs (e.g., forecasted weekly revenue attribution down 4%). - Validated with raw source data and downstream aggregates to rule out dashboard/reporting bugs.- Communicate risk - Wrote a concise incident summary (what changed, evidence, estimated impact, confidence level) and held a 30-min stand-up with Product, Analytics Manager, and Exec sponsor. - Provided recommended messaging: mark this week’s metric as “under investigation” with an annotated footnote and avoid decisions until corrected.- Coordinate mitigation with Data Engineering - Opened a priority ticket with reproducible test script and sample bad rows; paired with an engineer to propose fixes (restore prior null handling, add backward-compatible casting). - Agreed SLAs: hotfix within 24 hours, full regression pipeline test and deploy within 72 hours.- Interim analytics adjustments - Built a temporary backfill script to recalculate the KPI using previous ETL logic for affected window; exposed both “reported” and “adjusted” values in the executive deck. - Added automated checks: unit tests on join keys, row-count deltas, and alerting if metric deviates >5% from 7-day rolling trend. - Documented the incident, root cause, and post-mortem actions.

Result: Executives received an annotated metric with an adjusted value for immediate decisions; Data Engineering deployed the hotfix in 18 hours and backfilled data. Post-mortem reduced similar incidents by adding CI checks and saved an estimated weekly reporting risk of $150k in misinformed decisions. Learned to prioritize reproducible artifacts, clear risk communication, and automated monitoring to prevent recurrence.

Career Motivation & Google Alignment Questions

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Sample Answer

Unlock Full Question Bank