Trend Analysis and Anomaly Detection Questions

Covers methods for detecting and interpreting deviations in metric behavior over time and determining whether changes reflect real product or user behavior versus noise. Topics include baseline establishment, seasonality and holiday effects, time series decomposition, smoothing and aggregation choices, statistical detection techniques such as control charts, z scores, EWMA and CUSUM, thresholding strategies, and modern algorithmic approaches like isolation forest or LSTM-based detectors. Also covers visualization and dashboarding practices for communicating trends, setting sensible alerting rules, triage workflows for investigating anomalies, and assessing business impact to prioritize fixes or rollbacks.

HardTechnical

40 practiced

A metric for Region X drops to zero at 03:00. Describe detailed diagnostics to determine whether this is an instrumentation/pipeline problem or a real user behavior change. Include which tables/logs to check, how to compare raw event ingestion vs aggregated metrics, schema changes, deployment timelines, and how to rule out experiment or config changes.

Sample Answer

Situation: At 03:00 a metric for Region X drops to zero. My goal is to determine whether this is instrumentation/pipeline failure or real user behavior change.

Immediate checklist (first 10–30 minutes)- Confirm alert, note exact timestamp, metric definition, and owners (data infra, product).- Snapshot dashboard and raw query results for rollback/forensics.

Diagnostics steps (ordered, with artifacts to check)

1) Check raw event ingestion- Query raw event table (events_raw / kafka_topic / s3 logs) for Region X around 02:30–03:30.- Example SQL:

sql

SELECT event_type, COUNT(*) FROM events_raw
WHERE region='X' AND ts >= '2025-12-06 02:00' AND ts < '2025-12-06 04:00'
GROUP BY event_type;

- If raw events = 0 → upstream collection/SDK/edge issue.

2) Compare staged vs aggregated- Check staging/validated table (events_validated) and the aggregation table (daily_metrics_region).- Run rollup query to reproduce aggregated metric from raw data:

sql

SELECT region, date_trunc('hour', ts) as hour, COUNT(*) as cnt
FROM events_validated
WHERE region='X' AND ts BETWEEN ...
GROUP BY 1,2;

- If raw shows events but validated/aggregated = 0 → ETL validation or transformation bug.

3) Inspect pipeline logs & monitoring- Check Kafka consumer/group offsets, connector logs, Airflow/DBT run history, job durations, error traces between 02:45–03:15.- Look for schema validation errors, null-pointer, commit failures.- Files/logs: Kafka metrics, S3 write logs, Airflow task logs, dbt run logs.

4) Schema and deployment timeline- Review schema-change logs (schema_registry, git commits, db migration PRs) and deployment windows. Did any migrations, PRs, or config changes target 02:00–03:00?- Check CI/CD deploy history and team release calendar.

5) Configs and experiments- Query feature flag / experiment service logs for Region X toggles:

sql

SELECT flag, user_segment, changed_by, changed_at FROM feature_flags_audit
WHERE region='X' AND changed_at BETWEEN ...

- Check experiment assignments and targeting rules that could remove tracked events.

6) Infra/ops signals- Check CDN, mobile crash rates, app versions in Region X, and network errors (4xx/5xx). High client errors + zero events → client-side outage.- Check user sessions: session_start counts vs metric counts.

7) Reprocess and A/B sanity- Re-run aggregation on raw data for that hour locally. If re-aggregated metric >0, bug is in pipeline/aggregation; if re-aggregated =0 and raw events=0, likely real behavior or upstream outage.

Ruling out false positives- Correlate with other metrics (e.g., pageviews, logins). If all drop → outage. If only this metric drops while other events remain → instrumentation for that metric (missing event emission or a change in event schema/key).

Communicate and next steps- If instrumentation/pipeline: open incident with infra, attach queries, logs, and proposed rollback or reprocess plan.- If real behavior: prepare analysis of magnitude, affected cohorts, and notify product/stakeholders.- Always schedule a postmortem: root cause, monitoring gaps, alert threshold adjustments, and automated replay/reprocess capability.

EasyTechnical

50 practiced

Explain Exponentially Weighted Moving Average (EWMA): provide the update formula for the EWMA, compare it to a simple moving average, and describe how the smoothing factor alpha affects responsiveness to recent changes. Give an example of when EWMA is preferable to SMA in anomaly detection.

EasyTechnical

47 practiced

Explain the difference between an anomaly and an outlier in the context of time-series product metrics (for example, daily active users or revenue). Describe how trend, seasonality and noise can each be mistaken for anomalies and give concrete examples for each case. List the first 4 data-science checks you would run to determine whether a detected deviation is a real product/user change or instrumentation/noise.

HardTechnical

51 practiced

Discuss limitations and failure modes of deep-learning approaches (LSTM autoencoders, seq2seq) for anomaly detection on product metrics: data requirements, interpretability, false positives on distribution shift, compute cost, and maintenance. Propose hybrid or practical mitigations that combine statistical and ML methods for production reliability.

Sample Answer

Deep-learning sequence models (LSTM autoencoders, seq2seq) can be powerful for anomaly detection on product metrics but have important limitations and failure modes to plan for.

Key limitations / failure modes- Data requirements: Require large, representative time-series with labels (or long normal-history) to learn temporal patterns. Sparse, noisy, or non-stationary metrics lead to poor generalization.- Interpretability: Latent embeddings and reconstruction errors are opaque; hard to explain which feature or user behavior triggered an alert—bad for stakeholder trust.- Distribution shift / false positives: Models trained on historical “normal” behavior will flag legitimate regime changes (product launches, seasonality changes, instrumentation changes) as anomalies.- Compute and latency: Training and inference (especially with long sequences, multivariate inputs) are compute- and memory-intensive, increasing cost and complexity for real-time detection.- Maintenance and drift: Retraining, monitoring, data pipeline changes, and hyperparameter tuning are required; models silently degrade without robust monitoring.

Practical hybrid mitigations- Baseline statistical detectors: Run lightweight statistical rules (EWMA, STL decomposition + thresholding, z-scores on detrended series) in parallel as a sanity-check; use these for low-latency, high-precision alerts.- Ensemble gating: Use statistical methods to filter obvious cases and only pass ambiguous or high-signal windows to DL models. Conversely, require agreement between ML and statistic detector to trigger high-confidence alerts.- Context-aware thresholds: Combine reconstruction error with contextual signals (release flags, holiday calendar, traffic volume) to reduce false positives on known shifts.- Explainability layer: Augment DL outputs with feature-attribution (integrated gradients, attention weights) and per-metric residuals so engineers see which metrics/timepoints contributed most.- Continuous evaluation: Deploy shadow mode, label a sample of alerts, monitor precision/recall, and automate drift detection using embedding distance or change-point tests; schedule retrain or prompt human review when drift exceeds threshold.- Resource-aware architecture: Use temporal downsampling, sliding-window inference, or distilled/lightweight seq models for real-time; keep heavier models for offline scoring and root-cause analysis.- Operational playbook: Maintain metadata (deployments, experiments), runbook for triage, and feedback loop to push validated anomalies back into training data.

By combining robust statistical baselines, contextual business signals, explainability, and continuous evaluation, you can get the sensitivity of deep models while controlling false positives, cost, and maintenance burden—making anomaly detection reliable for production product metrics.

MediumTechnical

46 practiced

Given raw event logs with fields (user_id, event_type, timestamp, properties), outline a feature-engineering pipeline to craft features for anomaly detection of user behavior at daily and session granularity. Discuss aggregations (counts, rates), temporal features (time-between-events), sessionization, embeddings for categorical properties, and normalization strategies.

Sample Answer

Start by clarifying scope (daily per-user and per-session labels), then build a reproducible pipeline with these stages:

1) Ingestion & cleanup- Parse timestamps, canonicalize event_type and properties, drop duplicates, filter bots.- Ensure timezone normalization.

2) Sessionization- Define session boundary (e.g., 30 min inactivity) and assign session_id per user by sorting timestamps and computing gaps.- Emit session start/end, duration, number of events.

3) Aggregations (daily & per-session)- Counts: total_events, unique_event_types, events_per_type (sparse).- Rates: events_per_minute, events_per_session, fraction_of_type = count(type)/total_events.- Distributional stats: mean/median/std of inter-event times, event value quantiles, session duration quantiles.- Ratios: pct_events_in_top_session, events_last_1h / events_last_24h (recency features).

4) Temporal features- Time-between-events: mean, std, max inter-event interval; time since last event; hour-of-day, day-of-week histograms.- Burstiness: coefficient of variation of inter-event intervals.- Seasonality indicators: is_weekend, is_holiday.

5) Categorical property embeddings- For high-cardinality properties (page_id, item_id, referrer): options - Frequency/target encoding (with smoothing and leakage guards). - Learned embeddings via shallow neural net or embedding tables trained within downstream model (or pretrain with skip-gram on event sequences). - Aggregate embeddings per session/day by mean, max, or attention-weighted pooling.

6) Normalization & baselining- Per-user baselines: compute historical mean/std (rolling 7/28 days) and express features as z-score or relative change (current - baseline)/baseline to detect deviations.- Global scaling: log1p for heavy tails, robust scaling (median/IQR) for outliers, then standard scaling for modeling.- Categorical vectors L2-normalize after pooling.

7) Practicals / engineering- Avoid label leakage: only use past events when predicting anomaly at timestamp.- Handle sparsity: impute zeros for missing aggregates; use flags for insufficient history.- Feature selection & drift: monitor feature distributions, use incremental recomputation for rolling features.- Batch vs streaming: compute per-session in streaming, daily aggregates in batch; use incremental store (Redis/Materialized Views) for user baselines.

8) Validation & monitoring- Create synthetic anomalies and backtest detection sensitivity.- Track feature importance, covariate shift, and downstream model performance.

This pipeline balances interpretable aggregations (counts, rates, temporal stats) with expressive representations (embeddings) and robust normalization (per-user baselines + log/robust scaling) suitable for anomaly detection at daily and session granularity.

Unlock Full Question Bank

Get access to hundreds of Trend Analysis and Anomaly Detection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.