Forecasting and Time-Series Analysis Questions

Direct answer
ARIMA(p,d,q) models a series as a combination of three pieces: AR(p), autoregression on the series' own past p values; I(d), the number of times you difference the series to make it stationary; and MA(q), a moving average of the past q forecast errors. You identify p and q by reading the ACF and PACF plots of the (differenced) series, and you add seasonal terms (SARIMA) when the ACF/PACF still show a repeating spike pattern at the seasonal lag after ordinary differencing.

Structured elaboration

• AR(p): today's value is a linear function of the last p values plus noise. A PACF that cuts off sharply after lag p (with earlier lags significant) points to an AR(p) term, because the PACF isolates the direct effect of each lag after removing the effect of the lags in between.
• I(d): the number of times you difference $y_t$ (i.e. work with $y_t - y_{t-1}$, or the second difference) before the series is stationary. You determine d with a stationarity test (ADF/KPSS) rather than by eye: difference until the test says stationary, and stop as soon as it does. Over-differencing (d too large) introduces artificial negative autocorrelation and inflates forecast variance.
• MA(q): today's value depends on the last q forecast errors, not raw values. An ACF that cuts off sharply after lag q (while the PACF decays slowly) points to an MA(q) term.
• Order identification in practice: plot ACF and PACF of the differenced series. AR signature = PACF cuts off, ACF tails off. MA signature = ACF cuts off, PACF tails off. Mixed ARMA signatures (both tail off) are common in practice, so analysts usually also compare a small grid of candidate (p,d,q) by AIC/BIC rather than trusting the plots alone.
• When to add seasonal terms (SARIMA): if, after taking the ordinary difference, the ACF/PACF still show significant spikes at the seasonal period (e.g. lag 7 for daily-with-weekly-seasonality data, lag 12 for monthly-with-yearly-seasonality), a plain ARIMA hasn't captured the seasonal structure. SARIMA adds seasonal AR/MA/differencing terms $(P,D,Q)_s$ operating at multiples of the seasonal period s, on top of the ordinary $(p,d,q)$ terms.

Worked example
Take daily retail sales with a weekly pattern. Fitting an ARIMA(1,1,1) leaves a residual ACF with a clear spike at lag 7 (and 14, 21) - the model has removed the trend (via d=1) but not the weekly repetition. Adding a seasonal term, e.g. SARIMA(1,1,1)(1,0,1)$_7$, lets the seasonal AR/MA terms absorb that lag-7 structure; after refitting, the residual ACF should show no significant spikes at multiples of 7. Concretely: raising d from 0 to 1 changes what "capturing p and q" even means, since AR(p)/MA(q) are now fit on the differenced series, not the raw one - a common early mistake is reading ACF/PACF on the raw series and getting orders that don't apply once differencing is applied.

Trade-offs & pitfalls
Adding seasonal terms multiplies the number of parameters and the search space (p,d,q,P,D,Q,s); over-specifying any one of them (especially D, seasonal differencing) can remove real signal along with the seasonality. A senior candidate will also flag that pure ACF/PACF reading is a starting point, not a final answer: automated order search guided by AIC/BIC (or pmdarima.auto_arima) is standard practice once the visual signature is ambiguous, and the final choice should always be checked with a residual diagnostic (no significant autocorrelation left, e.g. a Ljung-Box test) rather than trusted from the identification step alone.

Direct answer
Forecasting a demand or capacity metric (like viewing hours, order volume, or active users) with strong seasonality and event-driven spikes calls for the same core toolkit as any seasonal demand forecast: a classical or ML model chosen to fit the seasonality's complexity, explicit handling of known future events (releases, holidays, campaigns), and evaluation against the operational decision the forecast actually supports (a capacity or staffing plan), not just a generic accuracy number.

Structured elaboration

• Model choice, classical vs ML: for one series with clean weekly+monthly seasonality and no meaningful external drivers, SARIMA or Holt-Winters are strong starting points. Once you need to incorporate known upcoming events (a title release, a new-market launch, a promotional campaign) as explicit regressors, or you're forecasting many related series together (many titles, many regions, many order channels), a model that supports exogenous features and pooling - Prophet with regressors, or a gradient-boosted model over lag/calendar/event features trained across all series - is the better fit.
• Handling known future events: build an explicit event/release-calendar feature (a flag, or a decaying "days since release" feature capturing the post-spike decline) rather than hoping the model infers a one-off spike from history alone; a genuinely new event type (first-ever release in a new content category) has no historical analogue for the model to learn from, so pair the model output with a judgmental adjustment or an analogue-based estimate for events without a clean precedent.
• Evaluating forecast accuracy for planning purposes: report error metrics AT THE GRANULARITY the planning decision needs (e.g. hourly RMSE if staffing is scheduled hourly, not just an aggregated daily/weekly number that could hide a systematically-missed daily peak), and evaluate coverage of the prediction interval specifically around event days, since those are exactly when planners most need the interval to be trustworthy and are also where models are most likely to be poorly calibrated.
• Scaling considerations when the metric feeds a real capacity decision: because under-forecasting a spike has an asymmetric cost (a capacity shortfall vs a modest over-provisioning cost), it's often appropriate to plan off a higher quantile of the forecast distribution (e.g. the 90th percentile) rather than the point median, and to build in an explicit contingency/fallback plan (manual override, rapid-scale trigger) for events materially larger than anything in the training history.

Worked example
Growth-style capacity questions (e.g. planning for user growth from 100k to 2M over 12 months) are a related but DISTINCT variant: at that scale of change, a pure time-series extrapolation of recent history is unreliable because the growth is driven by a step-change in the business (a new market, a new channel), not organic continuation of an existing trend, so a growth-curve or scenario-based model (grounded in comparable prior launches, if any exist) is usually more defensible than SARIMA/ETS extrapolation alone, with prediction intervals presented explicitly as scenario bands rather than a single confident number. A supply-side shock (e.g. a courier strike suddenly reducing available capacity) is the mirror-image problem: the DEMAND forecast may still be accurate, but the operational decision needs a separate supply-constraint model layered on top, since the two are not interchangeable.

Trade-offs & pitfalls
The most common mistake is validating the forecast on average-case error while the capacity decision actually depends on TAIL behavior (event days, growth inflection points, supply shocks); a model that looks excellent on an aggregate accuracy metric can still be dangerously wrong exactly when the capacity plan needs it most. Report event-day accuracy and interval coverage separately from the steady-state number, and be explicit with stakeholders about which regime (steady-state vs event-driven vs step-change growth) a given forecast is actually reliable for.

Direct answer
Reducing inference cost and latency for serving forecasts across thousands of zones means shrinking or speeding up the model itself (quantization, pruning, knowledge distillation) and/or serving more requests per unit of compute (batching) - each technique trades off differently against accuracy, memory footprint, and CPU/GPU utilization, and the right combination depends on which resource (latency, memory, raw compute cost) is actually the binding constraint.

Structured elaboration

• Quantization: reduce the numerical precision of the model's weights/activations (e.g. from 32-bit floats to 8-bit integers) - meaningfully reduces memory footprint and can substantially speed up inference (especially on hardware with native low-precision support), typically at a small, often acceptable accuracy cost; the accuracy impact should always be validated explicitly (a backtest comparing quantized vs full-precision accuracy) rather than assumed to be negligible.
• Pruning: remove weights/connections/whole units that contribute little to the model's output (e.g. near-zero weights, or entire redundant tree branches for a gradient-boosted model), directly reducing the compute needed per inference pass; can be applied more or less aggressively, with a corresponding accuracy/speed trade-off that should be tuned against a validation set rather than a fixed target sparsity chosen blindly.
• Knowledge distillation: train a smaller, faster "student" model to mimic a larger, more accurate "teacher" model's outputs, aiming to retain most of the teacher's accuracy in a cheaper-to-serve package - most valuable when the teacher model is meaningfully more accurate than a directly-trained small model would be, i.e. when there's genuine accuracy to "transfer down," not just compress.
• Batching: rather than compressing the model itself, batch multiple zones' inference requests into one forward pass - amortizes fixed per-inference overhead across many requests, often the SINGLE most effective lever for throughput/cost on GPU hardware specifically (which is heavily optimized for batched, parallel computation), at the cost of adding a small amount of latency for individual requests while the system waits to accumulate a batch (a real trade-off against per-request latency that needs to be tuned against your specific SLA).
• How each affects accuracy, memory, and CPU/GPU utilization: quantization and pruning both directly reduce memory footprint and typically speed up inference, at a (usually modest, but non-zero) accuracy cost that must be validated; distillation aims to preserve accuracy while reducing model size/cost, at the added engineering cost of training and maintaining a separate teacher-student training pipeline; batching doesn't change the model or its accuracy at all, purely improves throughput/cost efficiency, at a latency-per-request trade-off.

Worked example
For a forecasting workload with a genuinely low per-request latency requirement (as in a real-time, on-request forecasting service), aggressive batching may be constrained by how much added per-request latency the SLA can tolerate while a batch fills, making quantization or pruning (which reduce per-inference compute WITHOUT adding a wait-for-batch delay) the more appropriate levers; for a workload that instead runs as a large, scheduled BATCH job across all zones at once (less latency-sensitive per individual forecast), aggressive batching is the most straightforward and highest-leverage cost reduction, since there's no per-request latency SLA constraining how large a batch you can accumulate.

Trade-offs & pitfalls
Applying model-compression techniques (quantization, pruning, distillation) without an explicit accuracy-validation step against a genuine holdout is the most common mistake - a compressed model's speed and memory gains are easy to measure directly, but its accuracy cost is not automatically negligible and must be checked explicitly before deploying, the same discipline that applies to any change to a model that's already in production.

Direct answer
A production forecasting pipeline for 100k SKUs across 3 regions needs: a data-source layer feeding a feature store shared by both training and inference, a training cluster (batch, likely daily or weekly retraining given the scale), an inference/serving service that can produce forecasts fast enough for the dashboard refresh SLA, model registry/versioning for auditability and safe rollback, an explicit retraining policy, and monitoring covering both data health and forecast accuracy.

Structured elaboration

• Data sources: raw transactional/sales data, calendar/promotion/holiday data, and any exogenous signal (pricing, inventory position) feeding into a consistent, versioned data layer.
• Feature store: computes lag, rolling, and calendar features ONCE, serving both the offline training pipeline and the online inference path from the same definitions, avoiding train/serve skew - critical at this scale, where a subtle feature mismatch between training and serving is otherwise very hard to detect until forecast quality visibly degrades.
• Training cluster: given 100k SKUs, a single GLOBAL model (or a small number of segment-specific global models) trained on pooled data is the standard architecture rather than 100k individual per-SKU models, both for accuracy (pooling helps sparse-history SKUs) and for operational tractability.
• Inference service: needs to produce forecasts for all 100k SKUs × 3 regions within whatever the dashboard's refresh cadence requires - batch-score everything ahead of the refresh window rather than computing on-demand, given the scale.
• Model registry/versioning: every deployed model version tracked with its training data window, hyperparameters, and validation metrics, so a regression can be traced to a specific deployment and rolled back cleanly.
• Retraining policies: a defined cadence (e.g. weekly) balanced against compute cost, plus an explicit trigger for OUT-OF-CADENCE retraining if monitoring detects a genuine accuracy degradation or drift, rather than only ever retraining on the fixed schedule.
• Monitoring: both DATA health (are features arriving on time, in expected ranges) and FORECAST health (is accuracy holding up against the backtested expectation, tracked at the segment level given 100k individual SKU-level series would be too noisy to monitor one by one).
• Cost trade-offs: batch-scoring at this scale is the primary cost driver; balancing model complexity (a heavier model costs more compute per retrain and per inference batch) against the accuracy gain it buys is a real, ongoing trade-off, not a one-time architecture decision.

Worked example
The same architecture, pushed further, generalizes to an even larger scale (100,000+ SKUs, multiple regions, weekly retraining plus near-real-time reforecasting triggered by significant events) by adding CI/CD discipline around the model-training pipeline itself (automated validation gates before a new model version is promoted to serving) - the core components (feature store, training cluster, registry, monitoring) don't change qualitatively, but the OPERATIONAL rigor around promoting a new model version safely becomes proportionally more important as the blast radius of a bad deployment grows with scale.

Trade-offs & pitfalls
The most consequential architectural mistake at this scale is treating the feature-computation logic as something that can be reimplemented separately for training and serving "close enough" - at 100k SKUs, even a subtle skew (a slightly different rolling-window edge-case, a different NULL-handling convention) will silently degrade forecast quality across the whole portfolio in a way that's hard to trace back to its root cause without dedicated training/serving parity monitoring built in from the start, not added after a problem surfaces.

Direct answer
A practical SARIMAX workflow on daily revenue with weekly seasonality and promotion spikes: check stationarity and difference as needed, identify (p,d,q)(P,D,Q)$_7$ from ACF/PACF (or an automated search), fit with the promotion flag as an exogenous regressor, run residual diagnostics, and evaluate with rolling-origin cross-validation before trusting the prediction intervals.

Structured elaboration and worked example (executed)
The following was run end-to-end (seed=42) against a synthetic 2-year daily series with a real trend, weekly seasonality, a sparse promo indicator (5% of days, +15 true effect), and noise, fit with statsmodels.tsa.statespace.SARIMAX(order=(1,1,1), seasonal_order=(1,0,1,7)):

import numpy as np, pandas as pd
from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.stats.diagnostic import acorr_ljungbox

np.random.seed(42)
n = 365 * 2
t = np.arange(n)
trend = 100 + 0.05 * t
weekly = 10*np.sin(2*np.pi*t/7) + 5*np.cos(2*np.pi*t/7)
promo = (np.random.rand(n) < 0.05).astype(int)
y = trend + weekly + 15*promo + np.random.normal(0, 3, n)
df = pd.DataFrame({"y": y, "promo": promo})
train, test = df.iloc[:-28], df.iloc[-28:]

model = SARIMAX(train["y"], exog=train[["promo"]],
                 order=(1, 1, 1), seasonal_order=(1, 0, 1, 7),
                 enforce_stationarity=False, enforce_invertibility=False)
fit = model.fit(disp=False)
fc = fit.get_forecast(steps=28, exog=test[["promo"]])
pred_mean = fc.predicted_mean
rmse = np.sqrt(np.mean((test["y"].values - pred_mean.values) ** 2))
mape = np.mean(np.abs((test["y"].values - pred_mean.values) / test["y"].values)) * 100
ci = fc.conf_int(alpha=0.10)
coverage = np.mean((test["y"].values >= ci.iloc[:, 0].values) & (test["y"].values <= ci.iloc[:, 1].values))
lb_p = acorr_ljungbox(fit.resid, lags=[10], return_df=True)["lb_pvalue"].iloc[0]
print("AIC:", round(fit.aic, 2))
print("promo coefficient (should be close to true effect 15):", round(fit.params["promo"], 2))
print("28-day holdout RMSE:", round(rmse, 2))
print("28-day holdout MAPE%:", round(mape, 2))
print("90% PI empirical coverage over holdout:", round(coverage, 3))
print(f"Ljung-Box p-value (residuals, lag 10): {lb_p:.1e}")

Executed output:

AIC: 3542.99
promo coefficient (should be close to true effect 15): 14.47
28-day holdout RMSE: 2.71
28-day holdout MAPE%: 1.61
90% PI empirical coverage over holdout: 0.929
Ljung-Box p-value (residuals, lag 10): 1.8e-07

Reading this: the fitted promo coefficient (14.47) closely recovers the true injected promotion effect (15) - confirms the exogenous term is doing its job, not just absorbing noise, and the 90% prediction interval's empirical coverage on the holdout (92.9%) is close to its nominal 90%. But the Ljung-Box p-value (1.8e-07, far below 0.05) REJECTS "no residual autocorrelation left" - re-running the same check across several random seeds and a grid of nearby (p,d,q)(P,D,Q) orders, meaningful residual autocorrelation persisted in every case for this weekly-seasonal-plus-trend construction. That is itself the honest, useful diagnostic result here: a model whose exogenous coefficient recovers correctly and whose prediction-interval coverage looks good can still fail its own residual check, and the right response is to go back to the order-identification step (inspect WHERE in the residual ACF the remaining structure sits, try seasonal differencing D=1, or add explicit day-of-week/Fourier terms) rather than ship a model because the headline numbers look good.

• Identifying orders: examine ACF/PACF on the differenced series for the regular (p,d,q) part, and specifically at multiples of 7 for the seasonal (P,D,Q)$_7$ part; in practice, cross-check by fitting a small grid of nearby orders and comparing AIC rather than trusting the plots as the final word.
• Handling exogenous regressors for promotions: include the promo flag (or a decaying post-promo effect if the lift lingers) as a column in exog; critically, this exogenous value must be KNOWN for every period you're forecasting - a promotion calendar is known in advance, which is exactly what makes it usable this way, unlike a regressor you'd only observe after the fact.
• Model diagnostics: residual ACF/PACF should show no significant remaining spikes; the Ljung-Box test formalizes that check with a single p-value across multiple lags at once, as shown above.
• Evaluating with rolling-origin CV: refit (or at minimum re-forecast) at successive origins and aggregate RMSE/MAPE per horizon across folds, rather than trusting a single holdout window's number, which can be an unusually easy or hard period by chance.
• Prediction interval tips: SARIMAX's default intervals assume Gaussian residuals; validate that assumption (a quick residual histogram/QQ check) and, as shown above, validate EMPIRICAL coverage on a genuine holdout rather than trusting the analytic interval width by construction.

Trade-offs & pitfalls
The single most common real-world mistake with this workflow is fitting on the full history and reporting in-sample fit quality as if it were forecast quality - always hold out a genuine test window and evaluate on it, as done here. A related mistake is treating a good AIC or a correctly-recovered exogenous coefficient as proof the model is done: as the Ljung-Box result above shows, a model can look right on every summary statistic and still fail its residual diagnostic, which should trigger a return to order identification, not be explained away. A second, subtler mistake: an exogenous regressor that looks predictive in-sample but is NOT actually known in advance at forecast time (e.g. using same-day weather to "predict" same-day sales) silently invalidates the whole exercise the moment you try to deploy it, since you won't have that regressor's true value when you need the forecast.