Data and Trend Analysis with Pattern Recognition Questions
Analyzing quantitative and qualitative data to identify patterns, trends, correlations, and meaningful insights. Skills assessed include descriptive statistics, time series and trend analysis, visualization and dashboarding, hypothesis generation and testing, identifying seasonality and structural changes, distinguishing signal from noise, and synthesizing findings into clear recommendations. For qualitative inputs candidates should demonstrate coding, theme extraction, categorization, and synthesis of transcripts or survey responses. Emphasis is on choosing appropriate methods, validating patterns, avoiding common pitfalls such as confounding and spurious correlation, and communicating insights effectively to stakeholders.
MediumTechnical
21 practiced
Explain Granger causality: what hypothesis it tests, main assumptions, and limitations. Given two economic time series (advertising spend and sales), describe the data preparation steps (stationarity, differencing), how to select lag order, how to run the test in Python (statsmodels), and how to interpret p-values and the direction of Granger-causality.
Sample Answer
Granger causality tests whether past values of one time series contain information that helps predict another series beyond what the latter’s own past provides. Formally the null hypothesis is: "X does not Granger-cause Y" (lags of X have zero coefficients in a predictive regression for Y). Rejecting the null suggests predictive causality, not true structural causation.Main assumptions:- Stationarity of the series (or cointegration handled appropriately).- Linearity of the lagged relationships (standard test).- No omitted confounders that drive both series.- Errors are serially uncorrelated and homoskedastic (or adjusted).Limitations:- Granger causality = predictive precedence, not mechanistic cause.- Sensitive to nonstationarity, lag choice, sample size.- Fails for nonlinear relationships unless specialized tests used.- Contemporaneous effects or common drivers can produce spurious results.Data preparation (advertising spend A_t and sales S_t):1. Visualize and plot A_t, S_t; check for trends/seasonality.2. Test stationarity with Augmented Dickey-Fuller (ADF) / KPSS. If nonstationary, difference (ΔA_t, ΔS_t) or use VAR in differences; if both I(1) and cointegrated, use VECM instead.3. Remove seasonality or include seasonal dummies if needed.4. Align frequency, handle missing data, transform (log) to stabilize variance if appropriate.Lag selection:- Fit a VAR on the prepared (stationary) series and choose lag order by AIC/BIC/HQIC; also inspect partial autocorrelations.- Ensure enough observations remain after lagging.Python example (statsmodels):Interpreting results:- For each lag the test prints F-tests and p-values. If p-value < alpha (e.g., 0.05) at one or more relevant lags, reject the null that X does not Granger-cause Y.- Direction: run tests both ways: (sales, ad_spend) tests whether ad_spend → sales; (ad_spend, sales) tests sales → ad_spend. Significance only in one direction suggests uni-directional predictive causality; significance both ways suggests bidirectional feedback.- Consider consistency across multiple lag lengths and robustness checks (different lag selections, controlling for other variables, subsamples).
python
import pandas as pd
from statsmodels.tsa.stattools import adfuller, grangercausalitytests
from statsmodels.tsa.api import VAR
# assume df has columns 'ad_spend' and 'sales' indexed by time
# 1) stationarity test
for col in ['ad_spend','sales']:
print(col, adfuller(df[col].dropna())[:2]) # test statistic, p-value
# 2) difference if needed
df_diff = df.diff().dropna()
# 3) select lag via VAR
model = VAR(df_diff)
sel = model.select_order(maxlags=8)
print(sel.summary())
# 4) Granger causality test (maxlag = selected lag)
grangercausalitytests(df_diff[['sales','ad_spend']], maxlag=sel.aic, verbose=True)
# To test whether ad_spend causes sales, supply [sales, ad_spend] order.HardTechnical
19 practiced
Design a monitoring and remediation strategy for production ML models predicting customer churn where input distributions and label relationships may drift. Describe drift detection (univariate and multivariate), retraining triggers and cadence, canary deployments and rollback, feature-store/versioning, validation pipelines, and how to signal and involve stakeholders when model performance degrades.
Sample Answer
Requirements & constraints:- Real-time or batch scoring? (Assume daily batch + near-real-time for alerts)- Business tolerance for false positives/negatives, SLA for detection/remediation, data latency.High-level strategy:- Continuous monitoring pipeline that computes input-distribution metrics, joint/label metrics, business KPIs; automated alerts, automated canary rollout with safety gates, and human-in-the-loop remediation.Drift detection- Univariate: track feature histograms, population stability index (PSI), KL divergence, mean/std z-scores by segment. Maintain baseline rolling window (e.g., last 30/90 days).- Multivariate: use multivariate two-sample tests (MMD), training vs. serving embedding distances (e.g., PCA/autoencoder reconstruction error), density-ratio estimation or classifier-based drift detector (train a domain classifier to distinguish train vs. live).- Label relationship drift: monitor calibration (Reliability diagrams, Brier score), conditional error rates (by deciles and by cohort), precision/recall, lift, and monotonicity checks vs key features.- Statistical thresholds: use bootstrap to set confidence intervals and control false alarm rate. Differentiate transient spikes vs sustained drift (e.g., require drift persisting >3 days or >n batches).Retraining triggers & cadence- Automatic retrain if: (a) sustained drift beyond threshold on multiple detectors; (b) significant drop in business KPIs (churn lift, revenue impact); (c) label feedback indicates concept drift (model error rises >X%).- Scheduled cadence: weekly or biweekly retrain using newest labeled data with backtesting window (time-based cross-validation), but gated by validation pipeline. Keep manual override for emergency retrains.- Candidate selection: keep last N checkpoints; use rolling-training vs full-retrain experiments to compare.Canary deployments & rollback- Canary: deploy model to small % traffic (1–5%) or to representative cohorts. Route a held-out slice, compare online metrics vs control for MAPE, calibration, and business KPIs over short window (24–72h).- Safety gates: automated statistical tests for key metrics; fail-open or fail-back to previous model if worse than baseline by pre-set delta with high confidence.- Rollback: automated immediate rollback on critical failures (latency, obvious degradation), and manual rollback workflow with incident ticketing for non-critical degradations.Feature store & versioning- Centralized feature store with feature code, materialization pipelines, and lineage metadata. Store feature versions, preprocessing code, and schema (with hashes).- Snapshot training datasets, label joins, and pipeline DAGs. Use immutability for historical experiments and pointer-based references in production.- Enforce schema checks (types, null rates, cardinality) at ingestion.Validation pipelines- Offline validation: model-level tests (AUC, calibration, subgroup fairness), backtest on holdout windows, drift-aware validation (time-split), stress tests on synthetic shifts.- Pre-deploy CI: reproducible training run, unit tests for features, integration tests for scoring, canary smoke tests.- Post-deploy shadow testing: run new model in parallel (no serving) to compare outputs on live data.Stakeholder signaling & involvement- Alerting tiers: automated Slack/email paging for severity levels; include dashboards with context (feature drift, model metrics, business KPIs) and suggested next steps.- Runbooks: step-by-step remediation playbooks (investigate feature pipeline, label lag, rollback, retrain).- Weekly stakeholder summary: trends, leading indicators, experiments roadmap; monthly business review for model refresh cadence.- Cross-functional response: involve data engineering for data issues, ML infra for deployment, product/ops for business impact and decisions on intended behavior.Example quick flow:1) Monitor shows PSI>0.2 on top-3 features and classifier drift p<0.01 for 4 days → alert to on-call DS.2) Run automated validation & shadow evaluation; if performance drop confirmed (>2% AUC drop or business KPI hit), trigger canary deploy of retrained model to 2% traffic.3) If canary passes safety gates over 72h, ramp to 25% then 100%. If fails, rollback and create incident; begin offline retraining with labeled recent data.Trade-offs & notes:- Balance sensitivity vs fatigue: tune thresholds and ensemble detectors.- Prefer shadow/canary + human-in-the-loop before full automation for high-impact models.- Invest in observability and lineage to shorten MTTI/MTTR.
MediumSystem Design
30 practiced
Design a weekly churn analytics dashboard for a SaaS product with 1M users. Specify required metrics and how they are computed (e.g., active definition, churn window), data refresh cadence (near-real-time vs daily), pre-aggregation strategies, handling data latency, key visualizations for executives vs analysts, and recommended technology stack (warehouse, ETL/ELT, BI tool).
Sample Answer
Requirements & clarifications:- Weekly churn dashboard for a 1M-user SaaS (B2B/B2C? assume subscription-based with event logs + billing data).- Goal: monitor retention/leakage, identify cohorts at risk, enable action (marketing/CS).Metrics & definitions:- Active user: used product (defined events) at least once in the week OR had a successful billing for the period (choose one and keep consistent). Use both flags for cross-checks.- Churn window: user considered churned if no activity AND no paid renewal within 28 days after subscription period end (configurable).- Weekly Active Users (WAU): unique users with activity in past 7 days.- Weekly Churn Rate: (churned users in week) / (users active at start of week and eligible to churn).- Retention Rate (1/4/12 week): cohorts by signup/week, percent still active each week.- Net Revenue Retention (NRR), MRR churn, reactivation rate, CLTV over rolling 90d.- Cohort-level survival curves; feature-based churn propensity score (model output).How metrics computed:- Use event stream + billing table. Cohorts by signup_week. Churn evaluated with lookback window (28d) after last billing/end-of-period.- Compute rolling windows with event timestamps; label churn when condition satisfied.Data cadence & latency:- Near-real-time for user-level alerts and churn propensity scores (streaming inference): latency <5min.- Daily batch for authoritative weekly aggregates and financial metrics (overnight ETL), weekly materialized views for the dashboard. Executive views can be daily; analysts need raw user-level access.Pre-aggregation strategies:- Materialized daily aggregates: daily_active_by_user, daily_billing_status, user_last_activity, weekly_cohort_aggregates.- Precompute cohort-week retention, MRR by cohort, churn labels, and model scores nightly.- Use rollups and partitioning by date and cohort to reduce query cost.Handling data latency & correctness:- Mark dashboard with data currency timestamp and “incomplete” flags for last 48h.- Use watermarking for event ingestion; backfill jobs to reconcile late-arriving events.- Maintain event-versioning and reconcile billing vs events with audits.Key visualizations:- Executive (single-page): WAU trend, weekly churn %, MRR & NRR trend, top 3 at-risk cohorts, one-sentence insight & recommended action.- Analyst (multi-tab): cohort retention tables (heatmap), survival curves, funnel (activation → trial → paid), segmentation by product/plan/device, user-level drilldowns, features importance from churn model, sliceable by dates/region.- Alerts/notifications: spike in churn, MRR drop >X%, cohort deterioration.Tech stack (recommended):- Event ingestion: Kafka / Confluent or Kinesis- Streaming processing & enrichment: Flink or Spark Structured Streaming (for near-real-time labels/feature calc)- Data lake / warehouse: Snowflake or BigQuery (costs: choose based on org)- ETL/ELT: dbt for transformations, Airflow for orchestration- Feature store & model serving: Feast + Torch/TensorFlow + KFServing or Seldon for online scores- BI: Tableau or Looker for exec dashboards; Jupyter / Superset for ad-hoc analyst exploration- Monitoring & observability: Datadog / Prometheus; Data quality: Great ExpectationsTrade-offs:- Near-real-time adds complexity; prioritize daily batch for authoritative metrics, streaming for critical alerts and model inference.- Balance pre-aggregation granularity vs storage cost; keep raw event store for reprocessing.Implementation plan (phased):1) Instrument authoritative definitions; capture events + billing.2) Build nightly ETL + cohort aggregates + dashboard MVP (executive + analyst).3) Add churn model, reactivity, streaming alerts, and incremental pre-aggregations.
HardTechnical
21 practiced
An executive asks you to tell them definitively whether the loyalty program 'caused' a revenue increase. The data is noisy and several regions had concurrent marketing. How would you scope the analysis, choose methods, manage expectations, produce actionable recommendations, and communicate uncertainty and confidence to the executive team?
Sample Answer
Framework / Scope- Clarify the question and stakes: is the ask binary (“did the program cause revenue increase?”), or to estimate effect size and ROI for future decisions?- Define outcome(s) (net revenue, ARPU, retention), treatment (enrollment vs exposure), population (regions, customer cohorts), timeframe, and known concurrent campaigns.Analysis plan / Methods1. Aim for causal identification hierarchy: - Randomized experiment (ideal): recommend a rollout A/B if feasible for future validation. - Quasi-experimental methods given concurrent marketing: - Difference-in-Differences (DiD) with region × time fixed effects, controlling for region-specific trends. - Synthetic Control for regions where rollout was staggered. - Customer-level causal inference: propensity score weighting / doubly robust estimators to adjust for observables. - Instrumental variables if a valid instrument exists (e.g., eligibility threshold or rollout schedule).2. Combine approaches and run robustness checks (placebo periods, varying covariate sets).3. Model noise explicitly: hierarchical (mixed-effect) models to pool information across regions and quantify heterogeneity.Data & Implementation- Gather data: customer transactions, enrollment timestamps, marketing exposures, promo spending, seasonality, macro indicators.- Preprocessing: align timestamps, define pre/post windows, impute missingness, construct covariates capturing concurrent campaigns.- Run primary model (e.g., DiD with region-time FE + covariates), then validate with synthetic control and PSM-DR at customer level.Managing expectations & communicating uncertainty- Executive summary: state best estimate (effect size and 95% CI), level of confidence, and main caveats (confounding by concurrent marketing, data limitations).- Use visual aids: event-study plots, synthetic vs actual trajectories, coefficient plots with CIs, heterogeneity heatmaps.- Be explicit: “Given current data and methods, we estimate loyalty program increased revenue by X% (95% CI: a–b). Key uncertainty arises from concurrent marketing in regions A/C which could bias upward by Y%.”- Translate uncertainty to decisions: probability statements (e.g., P(effect > 0) = 0.87), expected ROI distribution.Actionable recommendations- Short term: run targeted A/B rollout in remaining regions; allocate an experiment budget to disentangle marketing overlap (e.g., factorial design).- Medium term: reweight marketing spend attribution to net out program effects; prioritize segments with strongest estimated lift.- Monitoring: deploy dashboard tracking cohort revenue, enrollment, and marketing exposures with alerts for divergence.- If immediate decision needed: provide risk-adjusted recommendation (e.g., continue program in regions where uplift > threshold and pause where indistinguishable from zero).Why this approach- Multiple causal methods and robustness checks reduce reliance on any single assumption.- Hierarchical models and heterogeneity analysis inform where program truly works, optimizing business action.- Clear quantification of uncertainty and decision-focused metrics (probabilities, ROI distributions) allow executives to act while understanding risk.Closing communication tone- Be concise, honest about limits, provide clear next steps (experiment plan + timeline), and offer to present visuals and drill into region/customer-level results in follow-up.
MediumTechnical
21 practiced
You have 3 years of daily revenue data with weekly seasonality and promotions spikes. In Python (statsmodels or pmdarima), describe step-by-step how to model and forecast with SARIMAX including: identifying orders, handling exogenous regressors for promotions, model diagnostics (residuals, ACF/PACF), and evaluating forecast accuracy using rolling-origin cross-validation. Include practical tips for prediction intervals.
Sample Answer
Approach summary: Deseasonalize/diagnose, identify SARIMA(p,d,q)(P,D,Q,s) via ACF/PACF and stationarity tests, include promotions as exogenous regressors (and lags), fit SARIMAX (statsmodels), run diagnostics, then evaluate with rolling-origin CV. Below is a compact, practical recipe with code snippets.1) Prep and identify orders- Test stationarity (ADF); if weekly seasonality s=7, consider D=1 seasonal difference on period=7 or use seasonal_decompose to inspect.- Examine ACF/PACF of differenced series to pick (p,q) and (P,Q).- Create promotion regressors: binary flag, promotion intensity, and lagged versions if effect is delayed.2) Fit SARIMAX (statsmodels)- Choose tentative orders from ACF/PACF; you can also use pmdarima.auto_arima with seasonal=True to suggest orders, but verify.3) Diagnostics- Residuals should be white noise: plot residuals, ACF of residuals, run Ljung-Box, histogram/QQ for normality.4) Forecasting & prediction intervals- To forecast, provide future exog for promotions. Use get_forecast to obtain mean and conf_int. If exog future is uncertain, scenario-based exog inputs or bootstrap residual simulation helps widen intervals.Practical tip: If residuals show non-normality or heteroskedasticity, simulate future paths by resampling residuals (bootstrap) and refitting/simulating to produce empirical prediction intervals.5) Rolling-origin cross-validation- Use expanding window: train on 0:t, test on t+1:t+h, move forward. Compute MAPE/RMSE aggregated.Evaluate multiple candidate orders and exog variants; choose model with best CV performance and sensible diagnostics.Final tips:- Always ensure future exog forecasts or scenarios are realistic.- Penalize overly complex seasonal models; prefer parsimony.- Use bootstrap simulation for robust prediction intervals when assumptions are violated.
python
import pandas as pd
import numpy as np
from statsmodels.tsa.stattools import adfuller
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
# assume df with columns 'date','revenue','promo'
df.set_index('date', inplace=True)
y = df['revenue']
X = df[['promo']].astype(float)
# stationarity
print(adfuller(y.dropna())[:2])
# seasonal diff
y_sdiff = y.diff(7).dropna()
plot_acf(y_sdiff); plot_pacf(y_sdiff)
# create lagged promo
for lag in (0,1,7):
X[f'promo_lag{lag}'] = X['promo'].shift(lag).fillna(0)python
from statsmodels.tsa.statespace.sarimax import SARIMAX
model = SARIMAX(y, exog=X, order=(1,1,1), seasonal_order=(1,1,1,7), enforce_stationarity=False, enforce_invertibility=False)
res = model.fit(disp=False)
print(res.summary())python
residuals = res.resid.dropna()
plot_acf(residuals); plot_pacf(residuals)
from statsmodels.stats.diagnostic import acorr_ljungbox
print(acorr_ljungbox(residuals, lags=[7,14], return_df=True))
residuals.plot(); import scipy.stats as st; st.probplot(residuals, dist="norm", plot=plt)python
# prepare X_future (length h)
fc = res.get_forecast(steps=h, exog=X_future)
mean = fc.predicted_mean
ci = fc.conf_int(alpha=0.05) # 95% PIpython
from sklearn.metrics import mean_absolute_percentage_error, mean_squared_error
def rolling_cv(y, X, order, sorder, initial_train, horizon):
errors=[]
for start in range(initial_train, len(y)-horizon+1):
train_y = y.iloc[:start]
train_X = X.iloc[:start]
test_y = y.iloc[start:start+horizon]
test_X = X.iloc[start:start+horizon]
m = SARIMAX(train_y, exog=train_X, order=order, seasonal_order=sorder).fit(disp=False)
pred = m.get_forecast(steps=horizon, exog=test_X).predicted_mean
errors.append(mean_squared_error(test_y, pred, squared=False)) # RMSE
return np.mean(errors)Unlock Full Question Bank
Get access to hundreds of Data and Trend Analysis with Pattern Recognition interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.