Machine Learning and Forecasting Algorithms Questions
An in-depth coverage of machine learning methods used for forecasting and time-series prediction, including traditional time-series models (ARIMA, SARIMA, Holt-Winters), probabilistic forecasting techniques, and modern ML approaches (Prophet, LSTM/GRU, Transformer-based forecasters). Topics include feature engineering for seasonality and trend, handling non-stationarity and exogenous variables, model evaluation for time-series (rolling-origin cross-validation, backtesting, MAE/MAPE/RMSE), uncertainty quantification, and practical deployment considerations such as retraining, monitoring, and drift detection. Applies to forecasting problems in sales, demand planning, energy, finance, and other domains.
MediumSystem Design
99 practiced
Design a monitoring plan for production forecasting models. Specify what data and model metrics you would monitor (for example distribution of residuals, coverage of prediction intervals, seasonal-adjusted error rates), feature drift indicators, alert thresholds, automated responses (retrain, degrade to baseline, notify), and a triage process to determine whether issues are data, model, or concept drift related.
Sample Answer
Requirements & objectives:- Detect data issues, feature drift, model degradation, and concept shift for time-series forecasting; minimize false alerts; enable automated safe responses and a human triage workflow.High-level components:1. Data monitors (ingest layer)2. Feature drift monitors (feature store / pipeline)3. Model performance monitors (predictions vs. truth)4. Alerting & automation engine (rules + playbooks)5. Triage dashboard + runbook (owner, steps, logs)What to monitor (metrics & signals)- Data quality & freshness: - Ingestion rate, missingness %, delayed partitions, duplicate keys, schema changes - Thresholds: >1% missing or >1 partition delay triggers warning- Feature drift indicators: - PSI or population stability index per feature (weekly rolling) — alert if PSI > 0.2 - KL divergence and 2-sample KS test (p < 0.01) for continuous features - Categorical shift: chi-square / JSD- Model performance (rolling windows: 7d, 30d, 90d): - Point metrics: MAE, RMSE, MAPE (seasonality-adjusted MAPE: remove expected seasonal component) - Probabilistic metrics: CRPS, Prediction Interval Coverage Probability (PICP) vs nominal (e.g., 90% PI) — alert if coverage deviates by >5 percentage points - Residual diagnostics: residual distribution mean ≠ 0, autocorrelation (ACF), heteroskedasticity tests - Calibration: PIT histogram uniformity, sharpness- Business KPIs / effect: - Destination metrics (inventory stockouts, capacity planning errors)- Operational: - Model latency, throughput, resource usageAlert thresholds & logic- Two-tier alerts: - Warning: metric crosses soft threshold (e.g., PSI 0.1–0.2, MAE +10% above baseline) — no action, email to owners - Critical: hard threshold (PSI >0.2, MAPE+30%, PICP off by >10%) or combined triggers (data + performance) — page on-call, activate automation- Use control-chart logic (rolling mean + 3σ) and baseline percent change (relative + absolute).Automated responses- Safe fallback: route predictions to baseline/simple model (last-year value, exponential smoothing) and mark downstream consumers with “degraded-mode” flag.- Auto-retrain: if drift is gradual and data integrity is OK, start scheduled retrain on latest N weeks with validation; require successful CI tests & backtest pass before promotion.- Block deployment: if schema mismatch or missing features, disable new predictions and fallback.- Automated rollback: revert to last validated model if new candidate fails post-deployment tests.- Notifications: include diagnostics snapshot, recent metric deltas, top drifting features, sample inputs/outputs.Triage process (playbook)1. Surface: dashboard shows which monitors fired + timestamps and affected entities.2. Data check (owner: data engineering / pipeline): - Confirm ingestion, latency, schema changes, upstream system incidents. - If data broken → fix upstream, replay, re-score if possible; mark incident resolved.3. Feature check (owner: ML engineer): - Examine feature distributions, PSI/KL; check transformations, scaling, categorical encodings. - If feature pipeline bug → patch and reprocess; if feature legitimately shifted, continue to Step 4.4. Model check (owner: data scientist): - Evaluate residuals, retrain candidate model on recent window; compare via backtest: holdout, rolling origin, forecast horizons. - If retrained model restores metrics → promote via canary rollout.5. Concept drift check (owner: product/analyst + DS): - Correlate with external events (promotions, holidays, policy changes); if concept drift, update model architecture/features or adjust targets.6. Remediation & postmortem: - Apply fixes (retrain, feature engineering, pipeline fixes), track RCA, update thresholds and monitoring rules to reduce recurrence.Practical notes & best practices- Multi-horizon monitoring: evaluate per forecast horizon (1d, 7d, 30d).- Granularity: monitor per segment (region, product) to catch local drift.- Use simulated and labeled anomaly scenarios to tune thresholds; avoid alert fatigue by aggregating similar alerts and enforcing rate limits.- Log inputs, predictions, ground truth and full feature snapshots for reproducibility.- Automate canary evaluation and CI checks (data tests, backtests, fairness checks) before any automated promotion.This plan balances automated safety (fallbacks, retrain), statistical detection (PSI/KL, residual analytics, PICP), and a clear human triage path that isolates data vs feature vs concept issues and returns the system to a validated state.
HardSystem Design
86 practiced
Architect a production forecasting platform that supports 100k time series, nightly or on-demand retraining, API serving of forecasts, backtests and lineage storage, a model registry, and automated drift detection. Describe components such as data ingestion (ETL), feature store, training orchestration, model registry, serving layer, storage schema for forecasts/backtests, scalability concerns, and strategies for auditability and backward compatibility.
Sample Answer
Requirements & constraints (clarify briefly): 100k independent time-series, nightly or ad‑hoc retrain, low-latency API for forecasts, backtesting and lineage traceability, model registry, automated drift detection, high scalability, auditability and backward compatibility.High-level architecture:- Data ingestion (ETL): event/batch sources → Kafka (streaming) + Airflow (batch). Raw landing into object store (S3 / GCS) in partitioned Parquet/Delta format. Use CDC where needed.- Feature store: Feast or custom store backed by OLAP (BigQuery/ClickHouse) for historical features and Redis/RedisGears or DynamoDB for online lookups. Feature definitions stored with versioned schemas.- Training orchestration: Kubeflow Pipelines / Airflow + Spark or Ray for distributed training. Orchestrator supports scheduled nightly runs and on‑demand runs via API; keeps resource isolation via Kubernetes namespaces and GPU autoscaling.- Model registry & metadata: MLflow or custom registry that stores model binary, run_id, hyperparams, training data snapshot pointers, feature versions, git commit, container image, evaluation metrics, and lineage graph edges. Registry enforces semantic model versioning.- Serving layer: Two-tier serving: - Batch/large-horizon: Precompute forecasts nightly and store results. - Real-time API: TF-Serving / TorchServe / FastAPI microservice behind Kubernetes with autoscaling. Cache recent forecasts in Redis and use sharded serving by ts_id ranges for scale. Support both synchronous single-ts queries and bulk batch endpoints.- Storage schema (forecasts/backtests): - forecasts table (Parquet/Delta): - ts_id, forecast_date (when forecast made), target_date, horizon, quantile (optional), point_forecast, upper, lower, model_version, run_id, feature_versions, created_at - backtests table: - ts_id, backtest_window_start, backtest_window_end, metric_name, metric_value, model_version, run_id, seed, params - lineage/metadata tables: data_snapshot pointers (S3 path + hash), feature_spec_id, code_commit, pipeline_run_id- Automated drift detection: - Data drift: run nightly checks comparing feature distributions (PSI, KS, population stability) per ts_id or shard. Use streaming monitors for sudden changes. - Concept drift: monitor prediction residuals/accuracy via rolling windows; run statistical tests and track metrics in registry. - Alerting + automatic triggers: if drift thresholds exceeded for N series or key segments, enqueue retrain job (partial or full) and mark model state as “stale” in registry.- Scalability considerations: - Shard by ts_id/hashing across worker pools for training and serving. Use distributed training frameworks (Spark, Ray) with per-shard parallelism. - Store precomputed forecasts to avoid heavy on-demand computation; keep TTL-based caching for near-term horizons. - Use columnar storage and partitioning by date/ts_id to optimize reads. - Autoscaling clusters and spot instances for cost efficiency; distributed feature computation with incremental updates to avoid full recompute.- Auditability & lineage: - Immutable storage: write-once S3/Delta with hashes; retention policies. - Metadata capture at every step: ingestion job id, data snapshot checksum, feature version id, pipeline run_id, model run_id, container image+tag, model signature (input schema), and evaluation results. Presentable as graph (Neo4j or MLMD). - Signed artifacts and access logs for compliance.- Backward compatibility & model contract: - Model signature: enforce input/output schema with explicit feature names/types and defaulting rules. Register signatures in model registry. - Canary & shadowing: deploy new model as shadow for a subset of ts or traffic; compare metrics before promote. - Versioned APIs: include model_version header; allow consumers to request forecasts by model_version or “latest-stable”. - Migration strategy: keep old model serving around; support multi-model routing and data adapters to translate older outputs.- Reliability, testing & CI/CD: - Automated tests: data schema, feature unit tests, model quality gates, integration tests for serving. - Model CI pipeline: on commit or on successful backtest -> pack container image, push to registry, run canary tests, then promote.- Cost & operational trade-offs: - Precompute vs online: precompute reduces load but uses storage; choose hybrid. - Full retrain vs incremental: use incremental/online updates for frequent small changes, full retrain nightly.- Security & governance: - RBAC for model registry and metadata, encryption at rest/in transit, audit logs, data access controls.Why this design:- Separates concerns (ingest, features, training, serving) enabling independent scaling.- Versioned artifacts and immutable snapshots provide auditability and reproducibility.- Sharding, precomputation, caching and autoscaling meet scale for 100k series.- Drift detection + automated retrain closes the loop for model freshness while canarying and model contracts ensure backward compatibility and safe rollouts.
MediumTechnical
79 practiced
Case study: You are given SKU-level daily sales for 3 years with columns {date, sku_id, price, promo_flag, store_id, units_sold}. Holidays calendar and weather history are also available. Stakeholders want a 90-day probabilistic forecast per SKU for inventory planning. Describe an end-to-end pipeline: data preprocessing, feature engineering (lags, rolling stats, holiday treatment), model candidates (statistical vs ML vs global), validation scheme including multi-horizon backtesting, choice of metrics, uncertainty estimation, deployment schedule, monitoring and how you would present results to stakeholders. Be specific about trade-offs for forecasting many SKUs.
Sample Answer
Overview: build a scalable pipeline that produces SKU-level 90-day probabilistic forecasts (per-day or per-week) for inventory. Focus on automation, model performance, and operational constraints (many SKUs, intermittent demand).Data preprocessing- Ingest and join sales, price, promo, store, holiday calendar, weather; enforce schema, timezone, and a single daily granularity per SKU-store.- Clean: remove duplicates, fix anomalies (negative units), impute missing covariates (forward-fill price, weather by nearest station).- Demand intermittency handling: flag zero-heavy SKUs; aggregate to weekly for very sparse SKUs.Feature engineering- Date features: day-of-week, month, week-of-year, is_month_end, seasonality flags.- Holiday treatment: binary holiday indicators, proximity flags (days to/from holiday), holiday type; treat holidays as special events (separate seasonal component).- Lags & rolling stats: lags at 1,7,14,28,90; rolling sums/means/std over 7,28,90 days; rolling promo counts; exp-weighted means.- Price/promo features: price elasticity proxies (price relative to median), promo intensity, interaction terms (promo x holiday).- Weather: recent temp/precip lags and anomalies vs. historical average.- Cross-SKU/store features: category-level demand, parent SKU rolling demand to borrow strength.- Static features: SKU size, shelf-life, lead-time buckets.Model candidates & trade-offs- Baseline statistical: Prophet / SARIMAX for explainability and fast per-SKU models — good for stable SKUs.- Count/time-series models: Negative Binomial / Poisson state-space models with covariates for intermittent/count demand.- Machine learning (local): Gradient Boosting (LightGBM/CatBoost) per-SKU using engineered features — flexible but heavy to train per SKU.- Global ML models: DeepAR/DeepState or Transformer-based global probabilistic models trained across all SKUs — share patterns, handle sparse SKUs, produce quantiles natively. Trade-off: more complex infra, needs careful hyperparameter tuning and compute.Recommendation: Hybrid: global probabilistic model (e.g., DeepAR or Temporal Fusion Transformer) as primary, fall back to lightweight per-SKU statistical models for top-volume SKUs needing interpretability and rare SKUs aggregated weekly.Validation & backtesting- Use rolling-origin multi-horizon backtesting: multiple windows (e.g., forecast origins every month for last 12 months), produce 90-day horizons, evaluate at horizons 1,7,30,90.- Use hierarchical validations: SKU-store, SKU, category aggregated metrics.- Time-series cross-validation preserving temporal order; avoid shuffling.Metrics- Point: MAE, RMSE; scale-invariant: MASE (handles different volumes).- Probabilistic: CRPS, Pinball loss at relevant quantiles (e.g., 5, 10, 50, 90, 95), PICP and MPIW for prediction intervals.- Business metrics: stockouts avoided, fill rate improvement, overstock costs (use service-level cost function).Uncertainty estimation- Use models that output quantiles (TFT, DeepAR) or sample paths (Monte Carlo dropout for NN, bootstrap for tree models).- Decompose uncertainty: aleatoric (count variability) vs epistemic (model uncertainty); capture both via ensembles or Bayesian approximations.- Provide full predictive distribution and pre-computed reorder quantities at target service levels.Deployment schedule & infra- Batch daily scoring pipeline (e.g., Airflow): feature materialization, model scoring, aggregation, saving forecasts to database.- Retrain cadence: full global model monthly, incremental weekly for fast-moving SKUs; quick nightly refresh of features and scoring.- Use containerized models (Docker), GPU for training, CPU for inference; store artifacts in model registry (MLflow).Monitoring & alerting- Data quality checks (volume shifts, missing covariates).- Model performance drift: track metrics (MAE, CRPS) per SKU/category; alert when degrade beyond threshold.- Prediction serving checks: coverage (PICP) and calibration; explainability heatmaps for top SKUs.- Business feedback loop: compare forecast vs actual sales, compute inventory KPIs (stockouts, days-of-inventory) and close loop.Presentation to stakeholders- Deliver dashboard with: aggregate and SKU-level forecasts, prediction intervals, top drivers (price/promo/holiday), expected stockouts and recommended reorder quantities per service level.- Scenario analysis: “what-if” promo/price/holiday simulations with resulting quantiles.- Executive summary: expected demand, confidence, inventory impact and recommended actions; highlight model limitations and plan (retrain cadence, margin of error).- Start with pilot on top 200 SKUs to validate business impact, then scale using global models ensuring monitoring and governance.Key trade-offs- Per-SKU models are interpretable but not scalable; global models scale and improve sparse SKU performance but require more infra and careful validation.- Forecast horizon vs granularity: weekly aggregation reduces noise for intermittent items but loses daily nuance for promotions.- Retrain frequency balances freshness vs stability; prioritize high-volume SKUs for frequent retrain.This pipeline balances scalability, probabilistic rigor, and business usability for inventory planning.
EasyTechnical
103 practiced
Explain Holt-Winters (triple exponential smoothing): describe how the level, trend, and seasonal components are updated, how smoothing parameters (alpha, beta, gamma) affect responsiveness, and when to choose additive versus multiplicative seasonality. Also discuss practical approaches to initialize and tune the smoothing parameters for production forecasting.
Sample Answer
Holt–Winters (triple exponential smoothing) decomposes a time series into level (ℓ), trend (b), and seasonal (s) components and updates them recursively to produce forecasts that capture trend and seasonality.Core update equations (additive seasonality, season length m):- Level: ℓt = α*(yt − s_{t−m}) + (1−α)*(ℓ_{t−1} + b_{t−1})- Trend: b_t = β*(ℓ_t − ℓ_{t−1}) + (1−β)*b_{t−1}- Season: s_t = γ*(yt − ℓ_t) + (1−γ)*s_{t−m}Forecast h steps ahead: ŷ_{t+h} = ℓ_t + h*b_t + s_{t+h−m·ceil(h/m)}For multiplicative seasonality replace (yt − s) with (yt / s) and add/subtract with multiply/divide accordingly; forecast multiplies seasonal factor.Effect of smoothing params:- α (level): larger → model tracks recent level changes faster; smaller → smoother, less noise-sensitive.- β (trend): larger → trend adapts quickly to change; smaller → stable trend estimate.- γ (seasonal): larger → seasonality adapts rapidly (useful if seasonal pattern changes); smaller → assumes stable seasonal shape.When additive vs multiplicative:- Additive if seasonal swings are roughly constant in magnitude (seasonal component independent of level).- Multiplicative if seasonal amplitude scales with level (e.g., peaks grow when baseline grows). Check by plotting seasonal subseries or testing variance proportionality.Initialization & tuning for production:- Initialize ℓ0 as average of first season, b0 as average difference between first two seasons divided by m, seasonal indices from first k seasons (subtract or divide by mean).- Tune α,β,γ by optimizing an objective (MLE assuming Gaussian errors, or minimizing sMAPE/MAPE/RMSE) via grid search, randomized search, or gradient-based methods. Use time-series cross-validation (rolling origin) to avoid look-ahead bias and evaluate stability across multiple folds.- Practical tips: constrain β,γ to be smaller if data noisy; regularize or tie parameters across related series; re-fit periodically (or use online updating) if regime/seasonality changes; monitor residuals and forecast intervals; use automatic libraries (statsmodels, Prophet-like alternatives) but validate assumptions and backtest.
MediumTechnical
88 practiced
You have 5 years of daily transactions and need to forecast the next quarter for the top 100 SKUs. Promotions are flagged in the data. Describe how you would estimate promotion uplift, ensure your forecasting model properly learns promotion effects without leakage, and how to incorporate uplift-adjusted forecasts into inventory decisions.
Sample Answer
High-level approach1) Define goals: quantify average and SKU-level promotion uplift (relative increase vs baseline), build a forecasting model that learns promotion effects without leaking future info, and produce uplift-adjusted demand scenarios for inventory decisions.Estimating promotion uplift- Exploratory: aggregate daily sales by SKU, promo flag, price, and features (holiday, day-of-week, seasonality). Compute simple lift = mean(sales|promo)/mean(sales|no_promo) as baseline.- Causal models: - Difference-in-differences (DiD) for SKUs with intermittent promos: model log(sales) ~ SKU FE + time FE + promo + covariates; promo coefficient → average uplift. - SKU-level uplift: use causal forests / generalized random forests to estimate heterogeneous treatment effects (HTE) using covariates (price discount percent, channel, holiday, baseline velocity). - Supplement with synthetic controls for long promotions or missing controls.- Validate with holdout promo periods and backtesting: compare predicted uplift to observed incremental units and profit margins.Preventing leakage / model training- Time-aware splitting: use rolling-origin CV (walk-forward) so training data only contains past promos relative to validation.- Feature engineering: represent promotions as lagged features (promo_t, promo_t-1..t-n), discount magnitude, and interaction terms with seasonality. Never include future promo flags or post-promo outcomes in features.- Use causal identification in model (control for time & SKU fixed effects) or train models to predict baseline (no-promo) demand and separately learn uplift so the model doesn’t implicitly “peek” ahead.- Regularization and careful covariate selection to avoid proxies that leak future events (e.g., shipment receipts that correlate with upcoming promos).Incorporating uplift-adjusted forecasts into inventory- Produce two streams per SKU: baseline forecast (expected demand without promo) and estimated incremental demand (uplift) per day for the quarter. Combine for scenario forecasts: expected, optimistic (upper CI), conservative (lower CI).- Inventory decisions: - Safety stock: inflate by variance from uplift uncertainty (use Monte Carlo simulations of uplift draws to compute service-level-based safety stock). - Reorder point & order quantity: include expected promo spikes in demand during lead time; for short lead times consider temporary surge inventory or vendor agreements. - Cost trade-offs: compute expected lost-sales cost vs holding cost to decide pre-buying for promos.- Monitoring & feedback: track realized uplift vs estimates, retrain causal/uplift models monthly, and update reorder policies.Metrics & validation- Use MAPE/MASE for baseline forecasts and incremental MAPE for uplift. Use cumulative incremental sales and ROI (promo incremental margin) to validate business impact.This pipeline provides causal uplift estimates, avoids leakage via time-aware training and separate baseline/uplift modeling, and translates uncertainty into inventory actions using scenario planning and simulation.
Unlock Full Question Bank
Get access to hundreds of Machine Learning and Forecasting Algorithms interview questions and detailed answers.
