Streaming Platform Data Analysis Scenarios Questions

Data analysis scenarios for subscription video streaming platforms, covering streaming and engagement metrics (DAU/MAU, watch time, completion rate, session length), user retention and cohort analysis, content consumption patterns, recommendation system evaluation, A/B test design and analysis, and data visualization and storytelling in the streaming domain.

HardTechnical

86 practiced

A new personalization model shows +5% AUC offline but online A/B test shows a small decrease in watch-time and an increase in short sessions. As the ML engineer on-call, outline a prioritized diagnostic plan to investigate and pinpoint root causes, including instrumentation, experiment slicing, and quick remediation strategies.

Sample Answer

Clarify goals & constraints- Confirm primary metric (watch-time) and safety constraints (no regressions >X%), traffic fraction, rollout windows, and whether online inference path differs from offline evaluation (features, freshness, logging).

Priority diagnostic plan (ranked)

1) Verify instrumentation & data fidelity (immediate, high priority)- Confirm experiment assignment integrity (randomization, bucketing, overlap).- Check logging completeness for impressions, clicks, play starts, play duration, session start/end, and reasons for short sessions (e.g., skip, autoplay stop).- Reconcile counts: compare experiment event counts vs. expected traffic; look for dropped or duplicated events.

2) Fast experiment slicing (parallel)- User segments: new vs returning, geography, device type, app version, subscription tier.- Content slices: genre, length, thumbnail type, recommended vs. home feed, autoplay vs. click-to-play.- Context slices: time-of-day, network quality (cell/Wi‑Fi), session source (push, home, search).- Compute metric deltas per slice with statistical significance & sample size checks.

3) Offline-vs-online feature & distribution checks- Compare feature distributions (input features, embedding indices, availability/freshness) between training/offline eval and production inference windows.- Check for feature drift, missing values, or different preprocessing (e.g., tokenization, normalization) in serving.- Validate model code parity: same weights, same inference code path, same beam/decoding/hyperparams.

4) Causal and behavior tracing- Inspect top ranked items for users in treatment vs control; sample user sessions to see if recommendations changed in obvious ways (e.g., too many short clips).- Run counterfactual logging: store model scores and features for a sample to replay offline and reproduce ranking.

5) System & latency checks- Ensure model latency and timeouts aren’t causing fallbacks to default ranking that bias short sessions.- Check resource throttling, stale caches, or partial failures.

Quick remediation strategies (safety-first)- Rollback or ramp down traffic to control if impact exceeds threshold.- Patch: apply a conservative scoring multiplier to reduce aggressive reranking (e.g., blend new model with baseline with weight α).- Hotfix feature differences (serve default feature values) if missing features found.- Whitelist unaffected cohorts while investigating.

Next steps & longer-term fixes- Implement deterministic counterfactual logging for future experiments.- Add automated slice monitoring and alerting (watch-time, short-session rate).- Postmortem: update offline eval to include metrics correlated with watch-time (e.g., short-session penalty) and include production-like feature staleness in validation.

Expected outputs in 24–72 hours- Instrumentation verification report, top 5 harmful slices, root-cause hypothesis (e.g., model over-optimizes short-content), and either rollback or blended deployment with monitoring.

MediumSystem Design

84 practiced

Design an anomaly detection system to surface sudden regional increases in buffering or drops in bitrate for Netflix. Describe telemetry to collect, real-time aggregation strategy, anomaly detection algorithms you would use, alerting thresholds, and how you would triage false positives.

Sample Answer

Requirements & constraints:- Detect sudden regional spikes in buffering or bitrate drops within minutes to allow ops/OSS to act; low false positive rate; scalable to millions of streams; support drill-down to ISP/CDN/device-level.

Telemetry to collect:- Per-play events (timestamped): session_id, region (geo + CDN edge), ISP, device type, player version, content_id, bitrate, buffer_start/stop events, rebuffer_duration, throughput estimate, playback_position, QoE_score. Include sampling tags and ingestion metadata (producer timestamp, server timestamp).

Real-time aggregation strategy:- Use a streaming pipeline (Kafka → Flink/Beam) to compute per-region, per-minute aggregates: playback_count, median/95th-percentile bitrate, total & avg rebuffer_rate (fraction of sessions with rebuffer > threshold), mean rebuffer_duration, throughput distribution. Maintain sliding windows (1m, 5m, 1h) and baseline windows (same hour-of-day, day-of-week) in a low-latency state store (RocksDB). Emit metrics to TSDB (Prometheus/Influx) and feature store for ML.

Anomaly detection algorithms:- Multi-stage approach: 1. Statistical baseline + seasonality: compute adaptive baseline using EWMA and seasonal decomposition (hour-of-day, day-of-week). Flag deviations beyond k sigma (k adaptive via historical variance). 2. Change-point detection: streaming Bayesian online change point or CUSUM on aggregated timeseries for quick detection of abrupt shifts. 3. Probabilistic model: use a Bayesian hierarchical model (region ← ISP ← device) to share strength across sparse regions and compute posterior probability of degradation. 4. Optional ML: a streaming isolation forest or an online supervised classifier trained on past incidents (features: delta from baseline, percentile shifts, sudden ISP-specific spikes) to reduce false positives.

Alerting thresholds & policies:- Multi-tier alerts: - WARN: relative increase > 2σ and sustained for 2 consecutive windows OR 10% relative jump in rebuffer_rate. - CRITICAL: posterior prob of degradation > 0.95 AND absolute impact > impact_threshold (e.g., >1k affected sessions or >5% regional rebuffer increase).- Include suppression rules: during planned experiments/deploys, or if sample count < minimum (e.g., <50 sessions) to avoid noise.- Alert payload includes top contributing ISPs, CDNs, device types, recent trend charts, raw sample counts, and suggested runbook steps.

Triage & reducing false positives:- Automated enrichment: on alert, run automated drill-down jobs (join with CDN logs, edge health, rollout flags, monitoring of AWS/infra; check player version rollouts and recent deploys).- Corroboration checks: cross-validate with independent signals (server-side throughput, CDN error rates, downstream infra metrics). If multiple signals agree, escalate.- Feedback loop: operators mark alerts as true/false; store labels to retrain supervised model and tune thresholds.- Noise reduction: require persistence (e.g., 3 windows) and impact thresholds; use hierarchical models to ignore single-ISP anomalies unless they affect large user counts.- Postmortem automation: when false positive confirmed, auto-adjust baseline sensitivity for that region/time (short-term) and log feature impacts.

Operational considerations:- Latency: aim for end-to-end detection within ~2–5 minutes.- Scalability: partition streams by region; use compact state stores; autoscale.- Observability: dashboards, alert audit logs, and daily false-positive analytics.This design balances speed, statistical rigor, hierarchical modeling to handle sparse data, and human-in-loop triage to keep false positives low while surfacing actionable regional degradations.

HardTechnical

72 practiced

Develop a hierarchical probabilistic forecasting approach for regional hourly viewing hours for capacity planning. Explain model choices (hierarchical time series reconciliation, probabilistic forecasts like quantile regression or parametric distributions), how to handle intermittent demand for small regions, and how to evaluate probabilistic accuracy (e.g., CRPS).

Sample Answer

Situation & objective: We need hourly probabilistic forecasts of viewing hours across a geographic hierarchy (global → country → region → city) for capacity planning, preserving coherence (aggregates = sum of children), handling many small regions with intermittent demand, and producing well-calibrated uncertainty.

Approach overview:- Produce base probabilistic forecasts at the lowest practical level (e.g., region/hour) and reconcile them to ensure hierarchical coherence using MinT (shrinkage) or Bayesian reconciliation.- Use models that output full predictive distributions or quantiles: quantile regression forests / Gradient Boosting Machines with quantile loss (LightGBM/XGBoost), DeepAR/Transformer-based probabilistic time-series (for complex seasonality and covariates), or parametric state-space (ETS/Prophet-like) with appropriate error distributions (log-normal or negative binomial for counts).

Modeling details:- Features: hour-of-week, holiday/streaming events, content releases, promo flags, device mix, latency-impact signals, recent lagged viewing, weather if relevant.- Probabilistic choices: - Quantile regression: yields set of quantiles (5%,10%,...,95%) robust and nonparametric; train separate quantile models or a single QR forest/GBM. - Parametric: fit log-normal or Gamma for continuous hours; negative binomial / zero-inflated NB if counts and overdispersion present. Use likelihood-based training for full predictive distribution. - Deep probabilistic models (DeepAR, TFT): learn conditional predictive distributions and covariate effects; scale across many series via shared parameters.- Reconciliation: - Compute base forecasts (mean and covariance/quantile vectors) at each node. - Apply MinT (optimal linear reconciliation) using shrinkage estimator of the forecast error covariance Σ to project base forecasts onto the summation subspace; for quantiles, reconcile in probability space (e.g., reconcile sampled paths or reconcile means and then adjust quantiles via sorted samples). - Alternative: sample-based reconciliation—draw Monte Carlo samples from child predictive distributions, sum to get aggregates, then fit quantiles—preserves dependencies if joint sampling uses learned cross-series correlations (copula or multivariate Deep models).

Handling intermittent demand (small regions):- Use hierarchical shrinkage: borrow strength via pooled / multi-task models (global/local embeddings in DeepAR/TFT or hierarchical Bayesian pooling) so small regions inherit patterns from similar regions.- Use zero-inflated or hurdle models when zeros dominate; Croston’s method or intermittent-optimized models (TSB) can be baselines.- Aggregate-up forecasting: for very sparse series, forecast at a higher aggregation (country) then disaggregate probabilistically using historical proportions with modeled variability.- Regularization and feature-rich models to avoid overfitting noise.

Evaluation (probabilistic accuracy & capacity planning utility):- CRPS: primary aggregate and per-node metric—measures distance between predictive CDF and observed value.- Quantile (pinball) loss per quantile; check P50, P90 for SLO-aligned capacity planning (e.g., provision for 95th percentile).- Calibration: PIT histograms or reliability plots across quantiles; sharpness: compare predictive interval widths.- Backtest on rolling windows with event stratification (holidays, releases). Evaluate at multiple aggregation levels (node, parent, global) because reconciliation can change behavior.- Business metrics: forecasted tail (95–99%) error rate for capacity breach, expected cost of over/underprovisioning, and SLA violation probability—use a cost function to pick operating quantile.

Operational considerations:- Train models on GPUs/TPUs for deep models; use streaming retraining or frequent minibatch updates for nonstationarity (weekly retrain + daily warmstarts).- Log predictive samples and evaluate drift; implement explainability for capacity decisions (feature attributions).- Monitor calibration and recalibrate (isotonic or Platt-style) if needed.

Trade-offs:- Quantile models are flexible and robust; parametric models are sample-efficient and interpretable.- MinT reconciliation needs good covariance estimates; sample-based reconciliation preserves nonlinearity but is computationally heavier.- For capacity planning prioritize well-calibrated upper quantiles (P90–P99) and operational cost alignment over minimizing mean error.

This pipeline (feature engineering → probabilistic base models → reconciliation → evaluation on CRPS/quantile loss + business metrics → deployment + monitoring) balances coherence, uncertainty, and robustness for regional hourly viewing-hour forecasts used in capacity planning.

EasyTechnical

88 practiced

You have two tables: users(user_id, signup_date) and plays(user_id, play_date). Using pandas or SQL, write the steps or code to compute 7-day retention for weekly signup cohorts: for each cohort_week (users whose signup_date is in that week) compute percent of users who played in week 1 (signup week) through week 4. Provide an example using cohort_week = 2025-01-06 and sample play dates.

Sample Answer

Approach (brief): assign each user to a signup cohort_week (week start), compute for each play whether it falls in cohort week + 0..3 weeks, count distinct users per cohort who played in each week, divide by cohort size to get retention %.

SQL (Postgres-style):

sql

-- cohort size
WITH cohorts AS (
  SELECT user_id, date_trunc('week', signup_date)::date AS cohort_week
  FROM users
),
plays_weeks AS (
  SELECT p.user_id,
         date_trunc('week', p.play_date)::date AS play_week
  FROM plays p
  JOIN cohorts c USING (user_id)
),
week_offset AS (
  SELECT c.cohort_week, pw.user_id,
         EXTRACT(epoch FROM (pw.play_week - c.cohort_week))/604800 AS week_num
  FROM cohorts c
  JOIN plays_weeks pw USING (user_id)
)
SELECT
  cohort_week,
  COUNT(DISTINCT CASE WHEN week_num = 0 THEN user_id END) * 100.0 / COUNT(DISTINCT c.user_id) AS week0_pct,
  COUNT(DISTINCT CASE WHEN week_num = 1 THEN user_id END) * 100.0 / COUNT(DISTINCT c.user_id) AS week1_pct,
  COUNT(DISTINCT CASE WHEN week_num = 2 THEN user_id END) * 100.0 / COUNT(DISTINCT c.user_id) AS week2_pct,
  COUNT(DISTINCT CASE WHEN week_num = 3 THEN user_id END) * 100.0 / COUNT(DISTINCT c.user_id) AS week3_pct
FROM cohorts c
LEFT JOIN week_offset w ON c.cohort_week = w.cohort_week
GROUP BY cohort_week;

Pandas (example with cohort_week = 2025-01-06):

python

import pandas as pd
users = pd.DataFrame({
 'user_id':[1,2,3,4],
 'signup_date':pd.to_datetime(['2025-01-06','2025-01-07','2025-01-10','2025-01-20'])
})
plays = pd.DataFrame({
 'user_id':[1,1,2,3,4,4],
 'play_date':pd.to_datetime([
   '2025-01-06','2025-01-13','2025-01-08','2025-01-20','2025-01-20','2025-02-03'])
})
users['cohort_week'] = users['signup_date'].dt.to_period('W').apply(lambda p: p.start_time.date())
plays['play_week'] = plays['play_date'].dt.to_period('W').apply(lambda p: p.start_time.date())
df = plays.merge(users[['user_id','cohort_week']], on='user_id', how='left')
df['week_num'] = ((pd.to_datetime(df['play_week']) - pd.to_datetime(df['cohort_week'])).dt.days // 7).astype(int)
# keep offsets 0..3
df = df[(df['week_num']>=0)&(df['week_num']<=3)]
pivot = (df.groupby(['cohort_week','week_num'])['user_id']
           .nunique()
           .unstack(fill_value=0))
cohort_sizes = users.groupby('cohort_week')['user_id'].nunique()
retention = pivot.div(cohort_sizes, axis=0).fillna(0) * 100
print(retention)

Example result for cohort_week 2025-01-06 (from sample data):- cohort size = 3 users (user_ids 1,2,3 signed that week)- week0 (signup week): users 1 and 2 played -> 66.7%- week1: user 1 played -> 33.3%- week2: user 3 played (played 2025-01-20 is week2) -> 33.3%- week3: 0 -> 0%

Notes:- Use distinct users (nunique) so multiple plays count once.- Align week start convention (ISO weeks or Sunday-start) consistently between signup and play.

MediumTechnical

65 practiced

You want to detect whether a UI change caused an uplift in engagement but only a subset of users received the change non-randomly in historical logs. Sketch a pragmatic approach to estimate causal effect from observational logs including model-based adjustments and a plan to validate findings with a prospective randomized test.

Sample Answer

Situation: We have historical logs where a UI change (treatment) was rolled out non-randomly to a subset of users. We need a pragmatic causal estimate from observational data and a plan to validate it with a prospective randomized test.

Observational approach (stepwise):1. Define estimand and metrics: average treatment effect on engagement (e.g., DAU, session length, retention at 7 days). Specify window and units (user-day).2. Build rich covariates: user demographics, prior engagement history (lagged metrics, trends), device/OS, region, time-of-day, experiment exposure, and any rollout rules that determined assignment.3. Exploratory checks: compare covariate balance between treated/untreated, plot pre-treatment trends, identify obvious confounders and time-varying patterns.4. Model-based adjustments: - Propensity score model (e.g., gradient boosted trees) to estimate P(treatment | covariates). - Use propensity for matching (nearest-neighbor with caliper) and inverse probability weighting (IPW) to estimate ATE. - Fit outcome regression (covariate-adjusted model) predicting engagement with treatment + covariates. - Use a doubly-robust estimator (e.g., augmented IPW) to combine both and reduce bias if one model is misspecified. - Consider time-series models (difference-in-differences) if rollout is staggered and parallel trends hold; or synthetic control for group-level rollout.5. Diagnostics & sensitivity: - Check post-adjustment covariate balance (standardized mean differences). - Run placebo tests: assign fake treatment dates or use pre-period to ensure no pre-treatment effect. - Negative controls: outcomes or cohorts known to be unaffected. - Conduct sensitivity analyses for unmeasured confounding (e.g., Rosenbaum bounds, tipping-point analysis).6. Quantify uncertainty: bootstrap or robust SEs; report point estimates with CIs and assumptions.

Prospective randomized validation:1. Design: randomize at the appropriate unit (user or bucket) with stratification/blocking on key covariates (e.g., region, prior engagement decile) to ensure balance.2. Power/sample-size: calculate based on minimum detectable effect, baseline variance, desired power and alpha; account for clustering if randomizing at higher level.3. Metrics & analysis plan: pre-register primary/secondary metrics, pre-specify handling of multiple comparisons, and use pre-specified analysis (intent-to-treat). Include subgroup analyses but label exploratory.4. Monitoring & guardrails: set early stopping rules for safety (e.g., big negative impact), run short interim checks but avoid peeking bias.5. Rollout: start with a small randomized pilot (smoke test), then scale to full powered RCT.6. Reconcile: compare observational estimates to RCT ATE. If they differ, re-examine model specification, unmeasured confounding, or interference. Use RCT to recalibrate observational models for future counterfactual estimation.

Why this is pragmatic:- Uses flexible ML models for propensity/outcome while retaining causal estimators (DR/IPW) that have theoretical guarantees.- Emphasizes diagnostics and sensitivity tests to assess plausibility.- Confirms findings with an RCT, which provides the gold-standard validation and protects product/metrics before full deployment.

Unlock Full Question Bank

Get access to hundreds of Streaming Platform Data Analysis Scenarios interview questions and detailed answers.

Join thousands of developers preparing for their dream job.