Feature Analysis and Launch Evaluation Questions
Designing and applying evaluation frameworks to measure feature success and inform launch decisions. Topics include defining success metrics, experimentation design and basic A over B testing concepts, setting evaluation timeframes, identifying confounding factors, cohort and funnel analysis, instrumentation requirements, and how to iterate based on results. Candidates should be able to propose metrics, describe trade offs in evaluation design, and explain how launch evaluation influences product prioritization.
HardTechnical
76 practiced
Design an algorithm or approach to detect experiment contamination caused by the same user seeing both control and treatment across devices using hashed identifiers. Describe privacy-respecting methods for matching, false-positive/negative trade-offs, and how you would scale detection on large event streams.
Sample Answer
Approach (summary)- Goal: detect users who appear in both control and treatment across devices while preserving privacy (only hashed identifiers available).- Use a two-stage, privacy-respecting matching: (1) secure deterministic hashing for join keys, (2) approximate streaming detection with bounded state to scale.- Key ideas: HMAC with experiment-specific secret (rotated), per-day keyed hash to limit linkability; use Bloom filters / Count-Min sketches or HyperLogLog for approximate cross-group membership; flag contamination when same keyed-hash maps to both arms within a window.Algorithm + example implementation (streaming pseudocode)1. Hashing: client/devices send stable IDs hashed with HMAC_SHA256(secret || id). Secret stored in secure KMS; rotate periodically.2. In stream processors (Flink/Beam), use keyed windows on hashed_id_key.3. Maintain minimal state per key: bitmask {saw_control, saw_treatment} + last_seen timestamp. On event: - if state becomes both bits set within experiment window, emit contamination event (experiment_id, hashed_key, timestamps, devices_count estimate).4. For large scale, replace exact per-key state with approximate membership: - per-partition Bloom filters for control and treatment; periodically compute intersections via bitwise AND (fast) to estimate contamination rate.5. To limit storage, evict keys not seen for window W.Example (Python-like pseudocode for batch detection)Privacy-respecting details- Use HMAC (server-held secret) not plain hash; rotate secret periodically to limit long-term linkability.- Namespace by experiment/date so hashes can’t be trivially joined across experiments/time without server access.- For stronger privacy, run matching inside a secure environment (Trusted Execution Environment / Secure Enclave) or use cryptographic PSI for cross-party matching.- When exposing contamination metrics, report aggregated rates with differential privacy (add calibrated noise) and thresholds (only release if count > k).False-positive / false-negative trade-offs- Exact keyed matching (HMAC) => low FP if secret safe, but FNs due to different identifiers per device (no deterministic ID across devices) or missing signals (cookie loss).- Bloom filters and sketches => space-efficient but introduce false positives (depending on parameters) and no false negatives for membership; intersection estimates may inflate contamination rate. Tune Bloom filter size and hash count to balance FP rate.- Time-windowing affects FN/FP: too-wide window may count sequential exposures as contamination (FP for causality); too-narrow misses cross-device exposure (FN). Choose window by experiment exposure lifetime and user behavior.Scalability- Stream processing: use Apache Flink/Beam with keyed state per hashed_id; windowing and TTL evictions ensure bounded state.- For extreme scale, use approximate structures: - Maintain per-shard Bloom filters for control and treatment; compute shard-wise intersections to estimate contamination rate O(shards * bloom_size / word_size). - Use HyperLogLog to estimate distinct contaminated users without per-key state.- Storage & compute: partition by hash prefix to balance load; use RocksDB state backend for Flink to store large keyed state efficiently.- Monitoring: track ingestion latencies, state sizes, false-positive estimates (from held-out ground truth where available).Complexity- Exact per-key detection: O(N) time, O(U) state where U distinct hashed IDs in window.- Approximate (Bloom/HLL): O(N) ingest with O(1) per-event update, fixed memory M (configurable), intersection computation O(M/word).Edge cases & mitigations- Shared devices (family tablets): may produce legitimate contamination—flag and surface to analysts rather than auto-exclude; consider device-type heuristics.- Salt leakage: rotate secrets and audit access; if leaked, rehash/roll experiments.- Sparse identifiers: combine multiple hashed signals (hashed_email, hashed_phone, advertising_id) with privacy rules; use weighted scoring to reduce FPs.- Verify pipeline with labeled sample (privacy-compliant) to estimate FP/FN and tune parameters.Resulting practice- Implement HMAC-scoped keys, stream detection with TTLed keyed state for correctness; for scale switch to Bloom/HLL approximations; protect outputs with aggregation thresholds and DP noise. This balances detection accuracy, privacy, and operational cost.
python
import hmac, hashlib, time
SECRET = b'secure_rotating_secret_v1'
def keyed_hash(id_str, experiment_id):
# include experiment_id to scope linkage; secret rotated periodically
return hmac.new(SECRET, (experiment_id + '|' + id_str).encode(), hashlib.sha256).hexdigest()
# process events (event: {id, experiment, arm, ts})
from collections import defaultdict
state = defaultdict(lambda: {'control':False,'treatment':False,'last':0})
contaminations = []
WINDOW = 7*24*3600
for ev in events:
key = keyed_hash(ev['id'], ev['experiment'])
s = state[(ev['experiment'], key)]
s[ev['arm']] = True
s['last'] = ev['ts']
if s['control'] and s['treatment']:
contaminations.append((ev['experiment'], key, ev['ts']))
# evict oldHardTechnical
142 practiced
Propose a repeatable iteration framework after an experiment completes: include follow-up experiments, feature branching, holdouts, and how experiment learnings feed into product prioritization and roadmap. Specify who owns each step and how to measure whether iteration improved outcomes.
Sample Answer
Situation: After an experiment completes we want a repeatable, measurable iteration loop so learnings turn into stronger product outcomes and the roadmap improves continuously.Framework (repeatable 6-step loop):1. Synthesize results (owner: Data Scientist + Analytics): produce one-page summary—primary metric lift, CIs/p-values, risk signals, segment heterogeneity, inferred causal mechanisms, and recommended next hypotheses. Deliverable within 3 business days.2. Decide action (owner: Product Manager + Stakeholders): triage into three outcomes—(A) ship as-is, (B) iterate with follow-ups, (C) rollback/holdout extension. Use a decision rubric: effect size, robustness, engineering cost, strategic fit.3. Design follow-up experiments (owner: Data Scientist + PM): specify hypothesis, primary/secondary metrics, sample size, duration, and guardrails. Include feature-branch designs: parallel branches for alternative implementations (A/B/C) and planned comparisons vs original.4. Implement & isolate (owner: Engineering + Experimentation Platform): use feature branches and dark-launch where possible; create code branches per variant, flag-controlled rollout, and automated CI tests. Reserve a statistically-powered holdout cohort (e.g., 5–10%) untouched for long-term validation.5. Run & monitor (owner: Data Scientist + SRE/Monitoring): real-time health dashboards, automated anomaly alerts, pre-registered analysis plan. Stop or adapt per sequential testing rules.6. Integrate into roadmap (owner: PM + Product Strategy, informed by DS): when a variant passes pre-defined success thresholds and holdout checks, schedule ship and allocate maintenance; if iterating, schedule next experiment in roadmap backlog with priority = expected value uplift × confidence / engineering cost.Ownership summary:- Data Scientist: analysis, experiment design, monitoring, final recommendation- Product Manager: decision, prioritization, roadmap integration- Engineering: implementation, feature branches, CI/CD- Experimentation Platform Team: allocation, holdouts, tooling- Analytics/BI: dashboards and long-term trackingMeasuring iteration improvement:- Short-term: delta in primary metric vs control; statistical significance and effect size- Medium-term: holdout comparison at 30/90/180 days to detect fade or downstream impacts (retention, revenue)- Process metrics: time from experiment end to decision, decision accuracy (fraction of shipped features that sustain lift at 90 days), number of follow-up iterations per idea, deployment lead time- Business ROI: uplift × exposed users — compare cumulative ROI of iterated features vs one-shot shipsExample: initial experiment showed +3% engagement but heterogeneity by cohort. DS recommended two follow-ups (personalization branch, simplified UI branch) and a 10% holdout. Engineering implemented branches behind flags; after 60 days the UI branch showed sustained +5% in holdout comparison; PM prioritized ship next quarter. Process metrics tracked showed decision-to-ship reduced from 8 to 5 weeks after adopting this loop.Why this works:- Clear owners avoid handoff delays- Feature branches + holdouts prevent regression and detect delayed effects- Pre-registered plans and process metrics ensure iteration improves both outcomes and discovery efficiency, not just noise.
EasyTechnical
88 practiced
List a minimal pre-launch instrumentation checklist you would use to validate that an A/B experiment is safely measurable in production. Include specific tests, sanity checks, and smoke tests you would run in staging and in the first hours after launch.
Sample Answer
Minimal pre-launch instrumentation checklist to validate an A/B experiment is safely measurableStaging (before launch)- Feature gating & deterministic assignment: verify experiment ID and deterministic bucketing produce stable treatment for given user IDs (unit tests + integration test).- Traffic split & targeting: simulate traffic to confirm percent allocation and inclusion rules.- Event instrumentation: confirm all primary/secondary metric events fire with required properties (user_id, experiment_id, cohort, timestamp). Use synthetic users.- Schema & contract checks: validate event payloads against schema (types, required fields).- Deduplication & idempotency: send repeated events to ensure server-side dedupe works.- Logging & observability: ensure experiment events flow to downstream pipelines (stream, warehouse) and monitoring dashboards show data.- Power/sample calc: confirm expected daily sample and time-to-detect given baseline and MDE.First hours in production (smoke + sanity)- Smoke test (0–15 min): hit feature with test accounts; confirm assignment and events appear end-to-end (frontend → event bus → raw topic).- Data arrival & latency (15–60 min): verify median ingest latency < SLA, no backlog in stream processors.- Sanity checks (hourly, first 4 hours): - Traffic sanity: actual treatment/control split ≈ planned (± tolerance). - User uniqueness: count(distinct user_id) consistent across raw and aggregated layers. - Identity join: experiment_id joins correctly to user profile/backfill. - Metric invariants: overall CTR/engagement not NaN; control group metrics stable vs historical baseline. - No huge lift/drop: flag any effect > 5× expected SD.- Alerts & rollback criteria: automatic alerts for missing events, schema errors, large deviation in traffic, pipeline failures; pre-approved rollback plan.Quick SQL checks (examples)- Traffic split: SELECT cohort, COUNT(DISTINCT user_id) FROM experiment_events WHERE day = CURRENT_DATE GROUP BY cohort;- Event presence: SELECT COUNT(*) FROM raw_events WHERE experiment_id = 'exp_123' AND event_type IN ('view','click');This checklist ensures assignment correctness, data integrity, observability, and fast recovery.
HardTechnical
91 practiced
You observed a small short-term revenue uplift from a personalization feature. Build a concise ROI forecast that combines short-term lift and projected lifetime value (LTV) to estimate net present value (NPV) of launching the feature. List assumptions you must make and how you would test key assumptions.
Sample Answer
Approach summary:- Combine observed short-term incremental revenue lift with projected incremental LTV per new/retained user, discount future cash flows, subtract rollout & operating costs to compute NPV. Present a simple model and sensitivity tests.Model (steps & formulas):1) Short-term incremental revenue (ΔR0) = observed lift * baseline revenue in test population scaled to target users.2) Incremental LTV per affected user (ΔLTV) = sum_{t=1..T} (Δr_t * retention_t) / (1+discount)^t.3) Total incremental lifetime value = #affected_users * ΔLTV.4) NPV = ΔR0 + Total incremental lifetime value - Implementation_cost - Ongoing_costs_discounted.Example (concise):- Observed 2% uplift on $10M monthly = ΔR0 = $200k.- Affected users = 100k; incremental monthly revenue per user = $0.02.- Assume monthly retention decay 5%, T=24 months, discount 8% annual (~0.64% monthly).- Compute ΔLTV per user ≈ sum_{m=1..24} 0.02*(0.95)^{m-1}/(1+0.0064)^m ≈ $0.38 → Total LTV ≈ $38k.- Implementation $150k, ops $5k/month discounted ≈ $100k → NPV ≈ 200k + 38k - 250k = -12k (borderline).Key assumptions to list:- Scalability: effect size scales from test to population linearly.- Causality: uplift is due to the feature (no confounding).- Retention/decay rates and per-user incremental revenue persistence.- Discount rate, time horizon T.- Implementation and O&M costs.How to test assumptions:- Rollout experiment: phased A/B or stepped-wedge to validate scale and externalities.- Cohort analysis: track per-cohort incremental revenue and retention over time to estimate ΔLTV.- Sensitivity analysis / Monte Carlo: vary uplift, retention, costs, discount to produce probability of positive NPV.- Causality checks: regression adjustment, difference-in-differences, covariate balance, funnel metrics to spot compensating behaviors.- Instrumentation: feature flags, revenue attribution, logging to measure downstream effects and operational load.Recommendation:- If NPV sensitive, run larger rollout with monitoring and stop/go thresholds; prioritize instrumentation and cohort LTV measurement before full launch.
EasyTechnical
74 practiced
Explain statistical power in plain language. Describe three levers you can change to increase power in an experiment and an example trade-off for each lever (for example, longer test duration vs time-to-decision).
Sample Answer
Statistical power is the probability an experiment detects a real effect when one exists — i.e., the chance you’ll correctly reject the null hypothesis. Intuitively: higher power means less risk of missing an important change (fewer false negatives).Three levers to increase power and trade-offs:1) Increase sample size- Why: More data reduces random noise and makes real effects clearer.- Trade-off: Cost and time. Collecting more users or longer test duration increases expense and delays decisions (longer time-to-decision).2) Reduce variance / improve measurement quality- Why: Cleaner signals (better instrumentation, more precise metrics, filtering irrelevant noise) increase the signal-to-noise ratio so the same effect is easier to detect.- Trade-off: Implementation effort and potential bias. Instrumentation changes or stricter filters can be time-consuming and might change what you measure (less generalizable).3) Raise significance threshold or use a one-sided test / increase effect size via stronger treatment- Why: Using a higher alpha (e.g., 0.10 vs 0.05) or a one-sided test makes it easier to reach significance; designing a stronger intervention amplifies the true effect.- Trade-off: Higher alpha increases false positives (more Type I errors). Stronger treatments may be less realistic or riskier for users/business.In practice balance these: choose sample size and measurement improvements first, and only relax statistical thresholds or change treatment strength with clear justification.
Unlock Full Question Bank
Get access to hundreds of Feature Analysis and Launch Evaluation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.