Trend Analysis and Anomaly Detection Questions

Covers methods for detecting and interpreting deviations in metric behavior over time and determining whether changes reflect real product or user behavior versus noise. Topics include baseline establishment, seasonality and holiday effects, time series decomposition, smoothing and aggregation choices, statistical detection techniques such as control charts, z scores, EWMA and CUSUM, thresholding strategies, and modern algorithmic approaches like isolation forest or LSTM-based detectors. Also covers visualization and dashboarding practices for communicating trends, setting sensible alerting rules, triage workflows for investigating anomalies, and assessing business impact to prioritize fixes or rollbacks.

HardTechnical

52 practiced

You're building an LSTM-based anomaly detector for multiple time-series metric types. Describe data preprocessing (normalization, windowing, handling missing values), model architecture choices (sequence length, layers, bidirectional?), loss functions (reconstruction vs next-step prediction), training regime, and how to serve the model for low-latency online inference. Also address model monitoring and fallback strategies.

Sample Answer

Situation: Building an LSTM-based anomaly detector for multiple metric types requires careful preprocessing, model choices, serving design, and operational plans so the system is robust and low-latency in production.

Data preprocessing- Normalization: per-metric z-score or min-max computed on rolling baseline windows (e.g., last 30 days) stored as metadata. For heterogeneous scales use per-metric scaling factors; for categorical/time-of-day effects include sin/cos cyclical time features.- Windowing: create fixed-length sliding windows (overlap e.g., stride=1 or tuned) of sequence length T determined by metric dynamics (see architecture). For multivariate metrics group correlated metrics into feature vectors; preserve ordering and timestamps.- Missing values: impute short gaps with linear interpolation or forward-fill; flag imputed points with a mask feature. For long gaps, drop windows or inject sentinel values and mask so model learns to ignore.- Data quality pipeline: validate timestamps, deduplicate, and compute feature drift stats in ETL (Spark/Beam), emitting alerts when baseline shifts.

Model architecture choices- Sequence length T: choose based on domain — e.g., T covering at least one seasonality period or autocorrelation decay. Use ACF/PACF to guide.- Layers: 1–2 LSTM layers with 64–256 units, dropout and layer-norm. Consider stacked LSTM if long-term dependencies exist.- Bidirectional? Use only for offline reconstruction tasks; for real-time next-step prediction use unidirectional LSTM to avoid future leakage.- Multivariate vs per-metric: train a global model with metric ID embedding + feature vectors to share learning and handle cold-starts, or per-metric models when behaviors are highly heterogeneous.- Output head: reconstruction (sequence-to-sequence) or next-step forecasting (sequence-to-one or -many).

Loss functions & training regime- Reconstruction loss: MSE or MAE over reconstructed window; good for unsupervised anomaly scoring by reconstruction error distribution.- Next-step prediction: MSE/MAE for prediction error; often more sensitive to sudden changes.- Hybrid: combine reconstruction + prediction + regularization (e.g., KL if using VAE).- Training: class-imbalance handled by using mostly normal windows; augment with synthetic anomalies for calibration. Use robust optimizers (AdamW), learning-rate schedules, early stopping on validation normal data. Store model artifacts and training metrics in registry.

Serving for low-latency online inference- Feature pipeline: light-weight real-time preprocessing (scaling using cached baselines, interpolation, window assembly) implemented in stream processors (Flink/Kafka Streams) or a microservice.- Model serving: export to ONNX/TorchScript and serve via fast runtime (Triton, TorchServe) with batching disabled or tiny batches to meet latency SLOs. Cold-start: keep warm replicas.- Memory/state: maintain rolling window state per time series in fast in-memory store (Redis) to avoid re-assembling sequences from raw store each request.- Throughput/latency: autoscale inference pods; use async calls and rate-limit upstream.

Model monitoring and fallback- Monitoring: track input feature drift, reconstruction/prediction error distributions, latency, QPS, and false-positive/negative rates using labeled incidents. Produce alerts when drift or KPI degradations occur.- Retraining triggers: automated pipeline that retrains when drift exceeds threshold or performance drops; use canary deployments and shadow testing.- Fallback strategies: if model unavailable or degraded, fall back to simple statistical rules (EWMA, z-score thresholds) served as a lightweight policy. Also expose safety gates: suppress alerts during known maintenance windows or when data quality is poor.- Explainability: log top contributing features and error timestamps to aid triage.

This approach balances model accuracy with production requirements: real-time lightweight inference, robust ETL and monitoring, and safe fallbacks.

EasyTechnical

40 practiced

Define Exponentially Weighted Moving Average (EWMA) and provide the recursive update formula. Explain how the smoothing factor alpha controls responsiveness versus noise reduction and give an example heuristic for choosing alpha when you want to detect a sudden 10% drop in conversion rate.

EasyTechnical

70 practiced

Explain the difference between a one-off anomaly (outlier), a structural change (change-point), and a sustained trend shift in time series metrics such as daily active users. For each type, give (1) a product-telemetry example, (2) a simple detection approach a data engineer would implement, and (3) which downstream stakeholders should be notified and why.

Sample Answer

One-off anomaly (outlier)- Definition: A short-lived, isolated deviation from normal (single-day spike/drop) that reverts quickly.- Example: A bot-driven spike in DAU after a scraped promo URL or a tracking duplication that inflates counts for one day.- Detection (simple): Z-score or IQR on daily residuals from a 7–14 day rolling median; flag points beyond ±3σ or outside 1.5×IQR. Optionally require volume threshold to avoid noise.- Who to notify: Data Quality/Observability team (to investigate instrumentation or bot traffic) and Product Ops (if user-facing). Notify Analysts if they use daily dashboards so they ignore that day in reports.

Structural change (change-point)- Definition: A discrete shift in the baseline level or variance that persists from some point onward (step change).- Example: A release changes default onboarding so DAU increases by +20% from that date onward.- Detection (simple): Cumulative Sum (CUSUM) or rolling-window mean comparison with statistical test (two-sample t-test between pre/post windows) or a simple breakpoint detection on a 14/14 day split with significance threshold.- Who to notify: Product Manager and Release Owner (to correlate with deployments), Analytics/Data Science (to update models/segmentations), and Data Engineering (if instrumentation change suspected).

Sustained trend shift- Definition: A gradual, continuous change in slope (up/down) over time rather than an abrupt jump.- Example: Organic adoption grows steadily after viral feature, causing a multi-week upward trend in DAU.- Detection (simple): Fit rolling linear regression on windowed time (e.g., 28-day) and monitor slope; flag when slope magnitude exceeds historical variability for several consecutive windows (e.g., 2–4).- Who to notify: Product Leadership and Growth/Marketing (business implications), Analytics/Modeling teams (retrain forecasts, cohort analyses), and Finance if revenue/metrics tied to DAU.

Notes on implementation and priorities:- Add simple pre-filters (minimum volume, working days) and run detections daily in the pipeline with severity levels (outlier vs persistent) to avoid alert fatigue.- Always include context (recent deploys, campaigns, tracking changes) in alerts to speed triage.

HardTechnical

43 practiced

A daily PySpark job computes anomaly scores for 200M data points and currently takes 6 hours. Propose concrete optimizations (repartitioning, avoiding wide shuffles, vectorized UDF alternatives, caching, adaptive query execution, tuning memory/executor configs) and show example Spark config changes or code snippets that could reduce runtime toward under 1 hour.

Sample Answer

Approach summary: reduce shuffle volume, use vectorized operations (Pandas UDFs / built-in), minimize serialization, tune cluster/GC and enable AQE + shuffle/serde optimizations. Target: cut I/O and shuffle, increase parallelism and memory efficiency.

Code/config examples:

1) Spark submit / defaults tuned

--conf spark.sql.adaptive.enabled=true
--conf spark.sql.adaptive.coalescePartitions.enabled=true
--conf spark.sql.adaptive.skewJoin.enabled=true
--conf spark.sql.shuffle.partitions=2000
--conf spark.sql.files.maxPartitionBytes=134217728   # 128MB
--conf spark.sql.execution.arrow.pyspark.enabled=true
--conf spark.executor.memory=32g
--conf spark.executor.cores=4
--conf spark.executor.instances=50
--conf spark.memory.fraction=0.6
--conf spark.memory.storageFraction=0.2
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
--conf spark.kryoserializer.buffer.max=512m

2) Repartition & avoid wide shuffles- Use range repartitioning on join keys to make joins local and balanced:

python

from pyspark.sql import functions as F
df = spark.read.parquet("s3://.../data")
df = df.repartitionByRange(2000, F.col("device_id"))

- Broadcast small reference tables:

python

from pyspark.sql.functions import broadcast
large.join(broadcast(small), "key")

3) Replace Python UDFs with vectorized Pandas UDFs or built-ins

python

from pyspark.sql.functions import pandas_udf, PandasUDFType

@pandas_udf("double", PandasUDFType.SCALAR)
def anomaly_score(vals: pd.Series) -> pd.Series:
    # vectorized numpy/pandas operations
    z = (vals - vals.mean()) / vals.std()
    return np.abs(z)

df = df.withColumn("score", anomaly_score(F.col("metric")))

Or use SQL/built-ins whenever possible (avoid row-at-a-time Python UDF).

4) Cache carefully and persist hot intermediate results

python

df = heavy_transformations(df)
df.persist(StorageLevel.MEMORY_AND_DISK)
# later reuse df for joins/aggregations

5) Reduce shuffle data size- Project only required columns early- Filter (predicate pushdown) before joins/aggregations- Use map-side combines (aggregate before shuffle):

python

from pyspark.sql import Window
# if computing per-key aggregates, use groupBy which does map-side combine
agg = df.groupBy("device_id").agg(F.count("*"), F.avg("metric"))

6) Skew handling- Detect skew via AQE skew detection or sample key counts; salt heavy keys:

python

# simple salting pattern
salted = df.withColumn("salt", (F.rand()*10).cast("int"))
df_salted = salted.groupBy("key","salt").agg(F.sum("value"))
df_combined = df_salted.groupBy("key").agg(F.sum("sum(value)"))

7) IO format & compression- Use columnar formats with snappy/none for CPU balance:write as parquet partitioned by date/device and enable vectorized parquet reader

spark.conf.set("spark.sql.parquet.filterPushdown", "true")
spark.conf.set("spark.sql.parquet.mergeSchema", "false")

Measurement & iterative steps:- Add metrics: spark event logs, shuffle read/write sizes, GC times.- Start with small change set (enable Arrow + pandas_udf + AQE) and benchmark; then tune partitions and executor sizing.Expected impact: vectorized UDFs + Arrow reduces Python overhead by 5-10x; reducing shuffle partitions and using broadcast/range partitioning can cut shuffle IO and network by 5-20x; together plus proper cluster sizing, runtime can move from 6h toward <1h depending on IO bounds. Edge cases: watch memory pressure (OOM), validate AQE behavior, ensure deterministic partitioning for downstream consumers.

MediumTechnical

54 practiced

You're using Isolation Forest to detect anomalies across thousands of time series. Describe how you would turn raw time-series into feature vectors for the model: windowing approach, summary statistics, trend/seasonality features, and pitfalls such as leakage or seasonal bias.

Sample Answer

Start by converting each time series into fixed-length feature vectors that capture local behavior and longer-term context. I’d pipeline this in stages (suitable for Spark or batch streaming):

1) Windowing- Use rolling windows (e.g., last 1h, 6h, 24h) and/or sliding windows with overlap. Choose window length based on domain periodicity (minute, hourly, daily).- For each timestamp t produce a vector representing window [t - W, t). Optionally keep multiple windows per timestamp for multi-scale signals.

2) Summary statistics per window- Basic: mean, median, std, min, max, skewness, kurtosis- Robust: trimmed mean, IQR, median absolute deviation- Counts: number of zeros/missing, outlier counts (z-score threshold)

3) Trend / seasonality / shape features- Slope (linear fit) and R² on the window- Seasonal decomposition residual energy (e.g., STL residual variance) or Fourier coefficients for dominant frequencies- Autocorrelation at relevant lags, partial autocorrelations- Recent change metrics: delta_mean, percent_change, rolling_pctiles- Shape: entropy, number of peaks, duration above threshold

4) Contextual / cross-series features- Relative rank across similar series, ratio to group median, aggregate deviation from cohort- Time-of-day / day-of-week categorical embeddings

5) Preprocessing & scaling- Impute or flag missing values (don’t silently fill with zeros)- Normalize per-series (z-score or robust scaling) to prevent high-volume series dominating- Optionally use PCA/UMAP to reduce dimensionality before Isolation Forest

Pitfalls- Leakage: don’t use future data in the window; when evaluating, ensure training windows exclude known anomalies and test is strictly forward-looking.- Seasonal bias: if training set is dominated by one season, model will treat other-season behavior as anomalous—sample uniformly across seasons or include seasonal features/time-of-day.- Label contamination: unsupervised training should avoid seeding with known anomalies that skew the model.- Scale and memory: fit Isolation Forest on sampled/feature-reduced data or use approximate/online implementations for thousands of series.- Drift: monitor feature distributions and retrain periodically.

This produces explainable vectors (statistical + time features) that Isolation Forest can use to detect atypical windows while avoiding common time-series traps.

Unlock Full Question Bank

Get access to hundreds of Trend Analysis and Anomaly Detection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.