Data Collection and Instrumentation Questions

Designing and implementing reliable data collection and the supporting data infrastructure to power analytics and machine learning. Covers event tracking and instrumentation design, decisions about what events to log and schema granularity, data validation and quality controls at collection time, sampling and deduplication strategies, attribution and measurement challenges, and trade offs between data richness and cost. Includes pipeline and ingestion patterns for real time and batch processing, scalability and maintainability of pipelines, backfill and replay strategies, storage and retention trade offs, retention policy design, anomaly detection and monitoring, and operational cost and complexity of measurement systems. Also covers privacy and compliance considerations and privacy preserving techniques, governance frameworks, ownership models, and senior level architecture and operationalization decisions.

HardTechnical

44 practiced

You are the head data scientist and must prioritize an instrumentation backlog across eight competing product requests. Propose a prioritization framework that balances business impact, observability risk, implementation cost, and ML value. Include a scoring rubric and an example decision for two hypothetical requests.

HardTechnical

37 practiced

You observe a 10% drop in conversion in production analytics for a specific country. Describe a minimal set of additional instrumentation and analysis steps you would deploy immediately to diagnose root cause within 24 hours. Include events, feature flags, cohort definitions, funnel checkpoints, and quick checks for data integrity.

Sample Answer

Immediate goal: triage, isolate scope (product vs data), and identify likely regressors within 24h. Prioritize cheap, high-signal instrumentation and analysis.

1) Quick scope & hypotheses- Ask: Is drop real (analytics pipeline) or behavioral (users)? Hypotheses: release bug, feature flag rollout, third‑party change (payment/ads), data loss, cohort-specific issue (new users, device/OS, country-specific gateway).

2) Minimal instrumentation to deploy now- Add an integrity heartbeat event emitted at ingestion points (server, client) with schema-version, event-timestamp, country, user_id, session_id.- Log feature_flag_state event whenever flags are evaluated for users (flag_name, value, user_id, reason).- Capture critical funnel events with explicit IDs if not present: page_view, add_to_cart, begin_checkout, payment_attempt, payment_success (include country, user_agent, app_version, client_time, server_time).- Instrument error/exception events: payment_error, validation_error with error_code.

3) Cohorts to generate immediately- New users last 14d vs returning- App version groups (latest vs previous)- OS/device (iOS vs Android), browser versions- Traffic source (organic, paid, affiliate)- Flag variants (flag=on/off)

4) Funnel checkpoints and analyses- Compute conversion rate week-over-week and hourly for country by cohort across checkpoints: page_view → add_to_cart → begin_checkout → payment_attempt → payment_success.- Identify where the largest relative drop occurs (e.g., 10% overall but 25% between payment_attempt→payment_success suggests payments).- Run A/A on adjacent countries to rule out global issues.

5) Quick data integrity checks (SQL examples)

sql

-- event volume check
SELECT date_trunc('hour', event_time) hour, count(*) FROM events
WHERE country='X' AND event_time >= now() - interval '48 hours'
GROUP BY 1 ORDER BY 1;

-- unique users vs previous week
SELECT count(distinct user_id) FROM events
WHERE country='X' AND event_time BETWEEN now()-interval '7 days' AND now();

- Check for schema changes, null spikes, duplicated events, time skew between client/server timestamps.- Verify ingestion pipeline logs for errors, backpressure, or dropped batches.

6) Feature-flag impact analysis- Compare conversion per flag variant using propensity-weighted or exact-match cohorts. If flag correlates with drop, roll back or disable immediately.

7) Rapid experiments and actions- If pinpointed (e.g., payment provider failing): rollback to previous provider/config, kill recent deploy, or route traffic to fallback.- If data pipeline issue: serve alert and fall back to raw logs for manual aggregation.

8) Decision criteria & next steps- If conversion loss localized to a funnel step → fix that service or UX.- If tied to cohort (app version) → fast rollback or targeted fix.- If analytics integrity uncertain → pause trusting aggregated metrics until integrity checks pass.

This plan balances immediate observability (events + flags), focused cohort/funnel analysis, and fast mitigations to restore conversion or isolate root cause within 24 hours.

MediumSystem Design

28 practiced

A schema change requires adding a new critical field used by ML features. Describe a backfill strategy to populate the new field historically for three months of data with minimal downtime and avoiding double-processing of downstream consumers. Include orchestration, idempotency, validation, and monitoring steps.

Sample Answer

Requirements:- Populate a new critical feature for past 3 months with minimal downtime.- Ensure downstream consumers (real-time and batch) don’t double-process.- Orchestrate, ensure idempotency, validate, and monitor.

High-level approach:1. Prep & schema/versioning - Add nullable field + feature-store version tag. Keep producer writes unchanged. - Add feature-store/backfill metadata table (job_id, window, status, watermark).

2. Orchestration - Use Airflow (or Prefect) DAG: steps: extract -> transform -> validate -> load -> promote. - Parallelize per-day or per-hour partition to speed backfill.

3. Idempotency & avoiding double-processing - Writes are upserts keyed by (entity_id, feature_name, feature_ts). Use conditional upsert that only writes if incoming feature_ts > stored_ts or if job_id not in processed_jobs for that partition. - Maintain per-partition checkpoint/lock in metadata table to allow safe retries. - Consumers read features by version tag or “as_of” timestamp; set contract so consumers ignore records with backfill_job_id until job is COMPLETE, or use a separate staging namespace until validation passes.

4. Validation - Unit checks (nulls, ranges), statistical checks (distribution vs online feature), row counts per partition, and sample spot-checks. - Run full-scan reconciliation comparing aggregate metrics vs source (e.g., counts by day, uniques).

5. Monitoring & alerting - Emit metrics: processed_rows, error_rate, lag per-partition, write_conflicts. - Dashboards and alerts for error_rate > threshold or mismatch in reconciliation. - Post-backfill: run integration tests with downstream model scoring on a shadow path and compare outputs.

6. Cutover - After successful validation, flip feature-store alias/version so consumers automatically read new field, or notify consumers to read new version. - Keep metadata for audits and safe rollback (retain staging data for X days).

Trade-offs:- Staging namespace reduces risk but adds temporary storage cost.- Per-partition parallelism speeds process but increases coordination complexity.

This plan ensures minimal downtime, idempotent writes, clear orchestration, robust validation, and observable monitoring to prevent double-processing.

HardTechnical

33 practiced

Design an approach to produce conversion attribution when events have been deduplicated and users may exist across multiple devices. Propose deterministic linking (login, email hash), probabilistic matching, graph-based identity resolution, and describe how to quantify confidence and propagate uncertainty into downstream revenue attribution.

Sample Answer

Situation/Goal: We need an identity-resolution pipeline that links deduplicated events across devices to produce accurate conversion attribution and propagate uncertainty into downstream revenue figures.

Approach overview:1. Deterministic linking (high precision, baseline)- Match on stable identifiers: logged-in user_id, email hash, phone hash, device_id when tied to account, cookieIDs after consent. These create deterministic edges with confidence = 0.99–1.0.- Always prefer deterministic edges and collapse entities immediately.

2. Probabilistic matching (for anonymous cross-device links)- Features: IP/time co-occurrence, user-agent similarity, geo trajectories, behavioral vectors (click patterns, browsing fingerprints), app install timestamps, frequency patterns.- Model: Gradient-boosted tree or logistic regression trained on labeled pairs (from deterministic positives, and sampled negatives). Use calibration (Platt or isotonic) to map model scores to probabilities P(same_user).- Thresholding: set three regimes — accept (P>0.95), reject (P<0.05), review/soft-link (0.05–0.95) stored as weighted edge.

3. Graph-based identity resolution- Build a bipartite/heterogeneous graph: nodes = identifiers (email hash, device_id, cookie), edges = deterministic or probabilistic links (edge weight = probability).- Resolve connected components and perform probabilistic aggregation: compute marginal posterior that each node belongs to a latent user using belief propagation or iterative network flow (message passing). Optionally run hierarchical clustering with edge-probabilities.- Enforce transitive consistency by combining probabilities: for nodes A-B (p1) and B-C (p2), infer A-C probability ≈ p1*p2 adjusted by network priors; use log-odds summation to avoid underflow.

4. Quantify confidence- For each resolved identity (cluster), compute: - Cluster confidence = 1 - Π(1 - edge_prob adjusted) or use posterior from graphical model. - Per-event attribution confidence: combine cluster confidence with time-decay and click-to-conversion model probability.- Calibrate using holdout deterministic ground truth and measure ROC, precision@k, Brier score.

5. Propagate uncertainty into revenue attribution- Attribution model assigns credit to events; instead of hard assignment, distribute revenue probabilistically: - For each event e linked to conversion c via path with probability p_e→c, assign revenue share = p_e→c * credit_rule (last-click/position-based). - Sum expected revenue per channel = Σ events (p_e→conversion * revenue * channel_share).- Compute variance: treat assignments as Bernoulli with mean p; propagate variance to produce confidence intervals for revenue (Var = Σ p(1-p)*(revenue*share)^2 plus covariance terms if needed).- Report expected revenue ± 95% CI and flag channels where uncertainty materially changes decisions.

Operational considerations:- Incremental updates: stream edges and update graph via online belief propagation.- Privacy & compliance: hash identifiers, limit feature retention, and respect do-not-track.- Monitoring: alert on drift (model score distribution), and periodically retrain with new deterministic labels.- Evaluation: A/B test downstream decisions using stochastic attribution vs. deterministic to measure business impact.

This pipeline balances precision (deterministic), recall (probabilistic), principled aggregation (graph inference), and transparent uncertainty propagation into revenue metrics so stakeholders can make risk-aware decisions.

HardTechnical

33 practiced

You have only a sampled subset of user interactions but need to train a conversion prediction model. Describe how to compute sample weights to correct for sampling, how to estimate the variance increase from sampling, and how you would validate that model performance on weighted sampled data generalizes to the full population.

Sample Answer

Approach summary- Use inverse-probability weighting (IPW): each sampled interaction i gets weight wi = 1 / p(xi), where p(x) is the (possibly conditional) sampling probability. If sampling was unconditional uniform by strata, p is the known inclusion probability for that stratum. Use normalized weights for optimization stability: w̄i = wi * (Nsample / sum_j wj) so the mean weight = 1.

How to compute weights- If sampling is known by strata S: p(S) = n_S_sample / N_S_pop → wi = 1/p(S).- If sampling is conditional on features and you only have sample: estimate p(x) with a propensity model (logistic regression or GBM) predicting sampled vs not-sampled using a labeled dataset of sampled + a small labeled population sample or known selection log. Then wi = 1 / p̂(x).- Clip or winsorize extreme weights (e.g., cap at 99th percentile or max W) and/or add small ridge to p̂ to avoid huge variance.

Estimating variance inflation from sampling- Compute effective sample size (Neff): Neff = (sum_i wi)^2 / sum_i wi^2. This is the usual design-effect based measure.- Variance inflation factor (VIF) ≈ n / Neff. So standard errors scale by sqrt(n / Neff). For model coefficients or AUC variance, use weighted-sample variance formulas or survey package estimators that account for weights.- For predictions, use bootstrap with resampling proportional to weights (or weighted bootstrap / Rao-Wu bootstrap) to empirically measure variance increase.

Validating generalization to full population1. If you can, hold out or obtain an unbiased small population test set (even few percent) and evaluate weighted-model predictions vs true outcomes — compare metrics unweighted on the full-pop holdout.2. Perform internal checks: - Calibration by strata: compare predicted conversion rates (weighted) to empirical full-pop rates where available (strata means). - Distributional checks: reweight sample to population using weights and compare feature marginals (means, quantiles) to known population statistics; large mismatches indicate propensity model misspecification. - Sensitivity analyses: vary clipping thresholds, alternate propensity models, and compare performance stability.3. Use simulation: generate synthetic population by upweighting sample according to estimated p̂, then draw simulated full-pop labels (or use importance-weighted metrics) to check bias/variance.4. Bootstrap-based confidence intervals for performance metrics using weighted resampling to reflect increased uncertainty.

Practical tips and trade-offs- Prefer stable propensity estimation with regularization; avoid huge weights by combining strata or smoothing p̂.- When training models, use the normalized weights in the loss function and consider focal loss / sample reweighting in batches.- Report Neff and weight diagnostics with model results so stakeholders understand increased uncertainty.- If weights are very variable, consider doubly-robust approaches (IPW + outcome model) to reduce bias.

Unlock Full Question Bank

Get access to hundreds of Data Collection and Instrumentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.