Streaming Platform Data Analysis Scenarios Questions

Data analysis scenarios for subscription video streaming platforms, covering streaming and engagement metrics (DAU/MAU, watch time, completion rate, session length), user retention and cohort analysis, content consumption patterns, recommendation system evaluation, A/B test design and analysis, and data visualization and storytelling in the streaming domain.

HardTechnical

76 practiced

Offline AUC improved after a model update but online watch-time decreased. Propose a systematic diagnostic plan using logs, offline simulations, and experiments to identify root causes. Include checks for distribution shifts between training and production, calibration changes, feature availability mismatch, position bias, and per-segment heterogeneity.

Sample Answer

High-level goal: systematically test whether the offline AUC→online watch-time gap is due to data, model, or serving differences, quantify where the drop occurs, and run targeted experiments to isolate causes.

1) Quick sanity & logging audit- Confirm model version, feature pipeline, and serving timestamps in logs. - Compute online vs training sample counts, dropped-request rates, and latency spikes. - Log feature-level missingness and imputation occurrences in production.

2) Distribution-shift checks- Compare P_train(X), P_online(X) and P_train(Y|X) vs P_online(Y|X) using: - Univariate KS / JS divergence for continuous features, chi-square for categoricals. - Train a binary “production vs training” classifier (ROC/AUC > .7 indicates shift); inspect top features driving it.- If heavy shift, stratify by time/region/device and surface upstream causes (feature rollout, data-source change).

3) Calibration & ranking checks- Recompute calibration on holdout production-labeled data: reliability diagrams, Brier score, and expected calibration error. - Check predicted score → watch-time mapping: compare average watch-time per score bucket between offline holdout and production. AUC can improve while calibration worsens.

4) Feature availability / pipeline mismatch- Simulate serving pipeline with logged production feature values: run the offline model code on production feature dumps and compare outputs to served predictions. - Check for differences from feature formatting, normalization, unseen categories, stale feature stores, or missing upstream signals (e.g., delayed events).

5) Position / exposure bias- Analyze watch-time as function of position/rank: compute CTR/watch-time conditioned on rank. - Re-evaluate offline metrics using position-aware evaluation (IPS, inverse propensity weighting) or stratify AUC by position. Model improvements that change ranking distribution can expose items to worse positions reducing watch-time.

6) Per-segment heterogeneity- Slice performance by user cohort (new vs returning), content type, device, geography, time-of-day. - For each slice compute offline AUC, predicted score distribution, conversion to watch-time, and online delta. Identify segments where offline gains don’t translate.

7) Offline simulation experiments- Replay production traffic with: - A) New model predictions injected but served with same ranking logic; B) New ranking end-to-end. - Use counterfactual simulations (logged bandit data / IPS) to estimate expected watch-time under new model before full rollout.

8) Online experiments- Run an A/B test with careful logging: - Allocate small % to new model; instrument intermediate metrics: exposure distribution, position changes, time-to-first-play, completion rate. - Add micro-randomization: keep ranking constant but swap relevance scores to isolate ranking vs score-calibration effects.- If viable, run factorial test: new model score + old ranker vs new ranker vs both.

9) Root-cause triage & remediation- If calibration drift: recalibrate (Platt/isotonic) using recent production labels and retest. - If feature mismatch: fix pipeline or simulate imputations; consider feature-robust retraining. - If position bias: add position-aware loss or train with IPS / causal correction. - If segment-specific harm: roll back for affected slices or train specialized models.

10) Monitoring & guardrails- Add pre-deploy checks: production-simulated predictions, P(X) drift thresholds, calibration tests, and a small-canary A/B with throttled exposure and automated rollback if watch-time degrades.

This structured path — logging → shift detection → simulation → targeted experiments — lets you isolate whether the issue is data drift, calibration/ranking change, feature mismatch, position bias, or per-segment effects and choose the minimal, safe fix.

MediumTechnical

74 practiced

You are tasked with building features from viewing history and metadata (genre, device, session info) for a 30-day churn prediction model. Propose a comprehensive feature set grouped by category (behavioral, temporal, content-based, device), example transformations (e.g., ratios, rolling means), and describe how to avoid leakage when constructing labels and features.

Sample Answer

Behavioral features (session & user-level)- Total sessions in lookback (30d), avg session length, median session length- Total play time, avg play time per session, play_time / session_time ratio- Number of distinct titles consumed, repeat-watch count, proportion of resumed vs completed views- Max consecutive days active, days-since-last-session- Interaction counts: searches, adds-to-list, likes, skips, rebuffer eventsExample transforms: counts, ratios (repeats/unique_titles), rolling mean(7d)/std, exponential decay weights

Temporal features- Activity cadence: sessions per day, weekend vs weekday activity ratio- Hour-of-day / time-slot distribution entropy (preference for evenings)- Trend features: slope of weekly playtime (linear regression), change-rate (t / t-1)- Seasonality flags (holiday window), days since signup

Content-based features- Genre exposure: fraction of minutes per genre, top-3 genres, diversity (Shannon entropy)- New-release ratio (newly released titles / total), avg content age- Language preference, maturity-level distribution- Weighted affinity scores per genre (time-weighted)

Device & technical features- Device type distribution (TV, mobile, web), primary device- Cross-device ratio (num_devices / sessions)- App version churn proxy (recent app upgrades), OS version- Avg bitrate / rebuffer rate, crashes per session

Feature engineering patterns & transforms- Aggregations (sum, mean, median, max, percentiles), ratios, normalized counts per active-day- Rolling windows: 7d, 14d, 30d rolling means and stds; exponentially-weighted moving averages- Interaction terms: playtime * device_type_TV, genre_entropy * days_since_last_session- Binning/quantiles for skewed variables; target encoding for high-cardinality categorical (with smoothing)

Avoiding leakage when constructing labels & features- Strict temporal split: define an observation window (features) and a disjoint label window (e.g., predict churn in days 31–60) — never use events after feature window end.- Label definition: churn = no active sessions in prediction window; ensure minimum post-window follow-up to confirm churn.- Use blackout buffer between feature window and label start (e.g., 1–7 days) to avoid using signals that are immediate precursors to churn.- When computing aggregates (e.g., “days since last session”), compute relative to feature-window end timestamp.- In cross-validation, use time-based CV (rolling-origin) to mimic production and prevent temporal leakage.- For target encoding, compute encodings using training folds only; apply smoothing and add noise if necessary.

Practical considerations- Handle sparsity: fallback defaults for new users, use hierarchical features (e.g., cohort-level averages)- Monitor feature drift and perform feature importance / SHAP to validate signals- Assess collinearity, regularize or remove redundant features before modeling.

EasyTechnical

64 practiced

List and explain key privacy and compliance risks when analyzing granular viewing behavior (e.g., precise timestamps, content titles) at Netflix. For each risk propose mitigation strategies (technical, policy, and product-level), and mention regulatory frameworks to consider (e.g., GDPR).

Sample Answer

Situation: As a data scientist working with granular viewing logs (precise timestamps, titles, device IDs), key privacy and compliance risks and mitigations include:

1) Re-identification risk- Why: Combination of timestamps + titles + device identifiers can uniquely identify users (even if userID removed).- Technical mitigations: aggregate or bin timestamps (hour/day), remove/replace rare title events, apply k-anonymity or differential privacy when releasing datasets, use strong pseudonymization with rotating salts.- Policy/product: restrict use to approved projects, require privacy review/DPIA for high-risk analyses; UI: avoid exposing “watch history” in downloadable reports.- Regs: GDPR (art.4 personal data, Recitals on identifiability), CCPA.

2) Sensitive profiling & automated decisioning- Why: Inferring sensitive attributes (e.g., political/religious views from content) creates risk of prohibited profiling.- Technical: detect and flag models using sensitive labels; block training on inferred sensitive attributes unless explicit lawful basis.- Policy/product: prohibition or strict oversight for targeting based on sensitive inferences; transparency to users.- Regs: GDPR (special categories, art.9), ePrivacy.

3) Purpose creep & lack of lawful basis- Why: Using logs beyond original purpose (billing/recommendations) violates data minimization.- Technical: metadata tagging of dataset purpose; enforce data access checks in pipeline.- Policy/product: require documentation of lawful basis (consent, contract, legitimate interest) and purpose; limit retention.- Regs: GDPR (purpose limitation, art.5), PDPA/LGPD.

4) Data breaches / unauthorized access- Why: Logs are attractive targets.- Technical: encryption at rest/in transit, role-based access, audit logging, anomaly detection, tokenized test sets.- Policy/product: least-privilege access, periodic access reviews, breach response plan.- Regs: GDPR breach notification (72 hrs), CCPA security obligations.

5) Data subject rights (access, erasure, portability)- Why: Users can request deletion/portability; granular logs complicate honouring requests.- Technical: build deletion/propagation pipelines, maintain mapping from pseudonyms to deletion tokens, implement export tools that redact others’ data.- Policy/product: SLA for handling requests, UX to allow history control.

Additional considerations: international transfers (SCCs/adequacy), kid-directed content (COPPA), logging retention policies. Applying DPIAs, privacy-by-design, and partnering with Legal/Privacy teams ensures compliant, responsible analytics.

EasyTechnical

86 practiced

Given a dataset with columns: user_id, content_id, genre, watch_start TIMESTAMP, watch_end TIMESTAMP, device_type, country — outline a prioritized exploratory data analysis plan and list 6 visualizations you would produce. For each visualization specify: chart type, x/y axes or aggregation, segmentation/facets, and the question that visualization answers. Also mention sampling strategies for very large datasets and preprocessing steps you would apply before plotting.

Sample Answer

Prioritized EDA plan (brief):1) Clarify goals (engagement, churn, device/country differences). 2) Data quality & preprocessing. 3) Summary stats & distributions. 4) Time-based patterns and session lengths. 5) Cohorts and retention. 6) Device/country/genre segmentation and anomaly detection. 7) Feature ideas for modeling (e.g., session_length, hourly_bin, active_days).

Preprocessing steps:- Parse timestamps, compute session_length = watch_end − watch_start (sec/min); drop or flag negative/zero lengths.- Normalize categorical fields (lowercase, map variants), handle missing country/device.- Remove duplicates, dedupe by user_id+watch_start+content_id.- Cap outliers or log-transform session_length for plots.- Convert timestamps to derived fields: date, hour, weekday, month.

Sampling strategies for very large data:- Stratified sampling by country or genre to preserve distribution.- Time-based sampling (e.g., sample weeks/months) to keep seasonality.- Reservoir sampling for streaming data.- Aggregate at user-day or content-day level to reduce cardinality.

Six visualizations:

1) Daily active users and total watch time (time series)- Chart: Dual-axis line chart- X: date; Y1: unique users/day; Y2: total watch_time/day- Facets/segmentation: country (small multiples) or device_type- Question: How does engagement vary over time and across markets/devices?

2) Distribution of session lengths (density + boxplot)- Chart: Violin or boxplot with overlaid histogram- X: session_length (log scale); Y: density or count- Facets: genre- Question: What’s the typical session length and how does it vary by genre?

3) Hour-of-day heatmap for starts (calendar/heatmap)- Chart: Heatmap- X: hour (0–23); Y: weekday; value: count of watch_starts- Facets: device_type- Question: When do users start sessions by day/time and device?

4) Top genres by watch time and conversion (bar chart)- Chart: Horizontal stacked bar- X: total_watch_time or avg_watch_time per session; segments: % by device_type- Y: genre (ranked)- Question: Which genres drive the most watch time and on which devices?

5) Retention cohort chart (cohort analysis)- Chart: Line or heatmap cohort table- X: weeks since first watch; Y: cohort start week; value: % users active- Segmentation: country or sign-up channel if available- Question: How does user retention evolve over time and across regions?

6) Country vs. device adoption scatter with size by avg session- Chart: Bubble scatter- X: % sessions on mobile; Y: avg session_length; bubble size: total watch_time; color: region- Question: Which countries have high mobile adoption and how does that relate to engagement?

For plotting: aggregate to appropriate level (user-day, content-day) before plotting; use efficient libraries (datatable/dask/pyspark) to compute aggregates, then sample for point-level plots. Validate plots with confidence intervals or bootstrapped uncertainty when sampling.

MediumTechnical

69 practiced

Design an interactive dashboard for the content team to support renewal decisions. Describe which KPIs (e.g., total minutes, unique viewers, retention of viewers, acquisition source), visualizations (trend lines, cohort retention heatmap, demographic breakdowns), filters/drill-downs, and confidence/uncertainty indicators you would include. Explain how early indicators for new shows should be surfaced.

Sample Answer

Requirements & goal: help content team decide renew vs. cancel by surfacing lifetime and early-signals for shows, with actionable confidence and drill-downs.

Key KPIs (what to show)- Consumption: total minutes watched, avg minutes per viewer, completion rate (per episode)- Reach & engagement: unique viewers, repeat viewers (% returning within 7/30 days), sessions per viewer- Retention: day/week retention curves (D1, D7, D30), cohort survival rates- Acquisition & conversion: top acquisition sources, trailer CTR, search lift, social referral lift- Business impact: estimated ARPU uplift, churn reduction attributed to show- Content health: viewers-per-episode, drop-off points (time-of-drop), critic/ugc sentiment score

Visualizations (how to present)- Overview: KPI cards with sparklines + % change vs. prior period- Trend lines: rolling 7/28-day for minutes, uniques, completion with shaded confidence bands- Cohort retention heatmap: cohorts by release week showing retention over time (color = % retained)- Funnel / waterfall: impression → trailer view → episode 1 started → completed → returned- Episode-level timeline: stacked bar for minutes by episode, overlaid with completion %- Demographic breakdowns: stacked bars or sunburst (age, gender, region, device)- Drop-off violin / density plot: common abandonment points within episodes- Geographic choropleth and top markets table- Sentiment/time-series from social/critics alongside view trends

Filters & drill-downs- Global filters: show/season/episode, release week, region, device, subscription tier, acquisition source, promo/campaign- Drill-downs: from show → episode → cohort → individual anonymized viewer journey- Compare mode: side-by-side shows (benchmarks) and ad-hoc cohort comparisons- Time window selector and alignment options (calendar vs. “week since release”)

Confidence & uncertainty indicators- Show 95% confidence/credible intervals on trend lines (bootstrap or Bayesian)- Early-forecast uncertainty band for projected lifetime minutes & unique viewers- Data freshness tags (last ingest time) and sample size badges (e.g., n viewers in window)- Statistical significance markers when week-over-week changes exceed noise- Explainable model features for predictions (e.g., "High early completion rate + trailer CTR → +X% projected retention")

Surfacing early indicators for new shows- Early-Window Dashboard (first 7–14 days) with a composite Early Success Score combining: D1 completion, D7 retention, trailer CTR, repeat viewers, social lift, and paid acquisition conversion. Use weighted, explainable model (logistic / gradient-boosted) that outputs probability of reaching renewal thresholds.- Show drivers panel: feature contributions (SHAP or feature importance) explaining why score is high/low.- Alerting: automated thresholds (e.g., Prob(renew)>70% or D7 retention <10%) send emails/Slack with context and recommended actions (boost marketing, adjust episode order, A/B test thumbnail).- Continuous updating: Bayesian updating or Kalman-filter forecasts that narrow uncertainty as more data arrives.

Implementation notes (Data Scientist lens)- Back-end: precompute cohort aggregates + event-level retention in a data warehouse; serve time-series via materialized aggregates for fast dashboard load.- Models: survival analysis for long-term retention, classification for early success probability; use calibration and backtest on historical shows.- Validation: include model-backtesting view (predicted vs actual for past releases) so content team trusts forecasts.- Tooling: implement in Tableau/PowerBI for visuals; use Python for modeling and scheduled score writes to DB.

Unlock Full Question Bank

Get access to hundreds of Streaming Platform Data Analysis Scenarios interview questions and detailed answers.

Join thousands of developers preparing for their dream job.