Exploratory Data Analysis Questions

Exploratory Data Analysis is the systematic process of investigating and validating a dataset to understand its structure, content, and quality before modelling or reporting. Core activities include examining schema and data types, computing descriptive statistics such as counts, means, medians, standard deviations and quartiles, and measuring class balance and unique value counts. It covers distribution analysis, outlier detection, correlation and relationship exploration, and trend or seasonality checks for time series. Data validation and quality checks include identifying missing values, anomalies, inconsistent encodings, duplicates, and other data integrity issues. Practical techniques span SQL profiling and aggregation queries using GROUP BY, COUNT and DISTINCT; interactive data exploration with pandas and similar libraries; and visualization with histograms, box plots, scatter plots, heatmaps and time series charts to reveal patterns and issues. The process also includes feature summary and basic metric computation, sampling strategies, forming and documenting hypotheses, and recommending cleaning or transformation steps. Good Exploratory Data Analysis produces a clear record of findings, assumptions to validate, and next steps for cleaning, feature engineering, or modelling.

HardTechnical

73 practiced

During EDA you ran 200 hypotheses across many segments and obtained p-values. Describe how to control false discoveries: explain Bonferroni, Holm, and Benjamini-Hochberg procedures, and demonstrate (in pseudo-code or Python/pandas) how to compute adjusted p-values and select significant signals under an FDR target.

Sample Answer

When you run 200 hypothesis tests you must correct for multiple comparisons to avoid excess false positives. Three common procedures:

- Bonferroni: very conservative. Multiply each p-value by m (or compare p <= α/m). Controls family-wise error rate (FWER) — probability of any false positive ≤ α.- Holm (step-down Bonferroni): uniformly more powerful than Bonferroni while still controlling FWER. Sort p-values ascending; for the i-th smallest p_(i) compare to α/(m-i+1). Reject sequentially until a p fails; all earlier are rejected.- Benjamini–Hochberg (BH): controls false discovery rate (FDR) — expected proportion of false discoveries among rejections. Sort p-values ascending and find largest i such that p_(i) ≤ (i/m) * q. Reject all p_(1..i). Less conservative, appropriate when some false positives are tolerable.

Example Python (pandas) to compute adjusted p-values and select discoveries under FDR target q:

python

import numpy as np
import pandas as pd

def adjust_bonferroni(p):
    m = len(p)
    return np.minimum(p * m, 1.0)

def adjust_holm(p):
    m = len(p)
    order = np.argsort(p)
    sorted_p = p[order]
    adj = np.empty(m)
    for i in range(m):
        adj[i] = min((m - i) * sorted_p[i], 1.0)
    # make monotone non-decreasing when mapped back
    adj_sorted = np.maximum.accumulate(adj[::-1])[::-1]
    adj_back = np.empty(m)
    adj_back[order] = adj_sorted
    return adj_back

def adjust_bh(p):
    m = len(p)
    order = np.argsort(p)
    sorted_p = p[order]
    adj = np.empty(m)
    cummin = 1.0
    # compute BH adjusted p-values (Benjamini-Hochberg, adjusted)
    for i in range(m-1, -1, -1):
        rank = i + 1
        val = sorted_p[i] * m / rank
        cummin = min(cummin, val)
        adj[i] = cummin
    adj_back = np.empty(m)
    adj_back[order] = np.minimum(adj, 1.0)
    return adj_back

# Example usage
np.random.seed(0)
p = np.random.rand(200)  # replace with your p-values array
p_series = pd.Series(p, name='p')

df = pd.DataFrame({
    'p': p_series,
})
df['bonferroni_adj'] = adjust_bonferroni(df['p'].values)
df['holm_adj'] = adjust_holm(df['p'].values)
df['bh_adj'] = adjust_bh(df['p'].values)

# Select discoveries under FDR target q (BH)
q = 0.05
df['reject_bh'] = df['bh_adj'] <= q
# Or using threshold procedure to find maximum i:
# sorted = df.sort_values('p')
# find max i with p_i <= (i/m)*q, reject all up to i

print(df[['p','bonferroni_adj','holm_adj','bh_adj','reject_bh']].head())

Key considerations:- Use Bonferroni/Holm when you need strong control of any false positive (FWER). Use BH when you accept some false discoveries but want control of their rate.- Dependence among tests: BH is valid under certain positive dependence; use Benjamini–Yekutieli for arbitrary dependence (more conservative).- Report both raw and adjusted p-values, method used, and number of discoveries.

EasyTechnical

82 practiced

Describe practical strategies to detect and handle duplicate records in a dataset. Include examples of exact duplicates, near-duplicates (fuzzy), and strategies for deduplication such as fingerprinting, hashing, blocking, and manual review. Mention how you would log and validate deduplication decisions during EDA.

Sample Answer

Practical strategy to detect and handle duplicates:

1) Define types & examples- Exact duplicates: identical rows (same values across all fields). Example: two identical user records from re-ingest.- Near-duplicates (fuzzy): small differences (typos, punctuation, different order). Example: "John A. Smith" vs "Smith, John".

2) Detection techniques- Exact match: use dataframe .duplicated() or group-by hashes on key columns.- Fingerprinting/hashing: normalize (lowercase, strip, canonicalize whitespace, remove punctuation), then compute hash (MD5/SHA1) of concatenated key fields to find probable duplicates.- Blocking/indexing: partition data by blocking key (e.g., first letter of last name, zip code) to only compare within blocks — reduces O(n^2).- Fuzzy matching: compute similarity metrics (Levenshtein, Jaro-Winkler) or use token-based measures (TF-IDF cosine, MinHash + LSH) for near-duplicate detection.

3) Deduplication strategies- Rule-based merge: prioritize authoritative source, most recent, or most complete record when merging fields.- Probabilistic record linkage: use Fellegi–Sunter or supervised matcher (train model on labeled pairs).- Fingerprint + cluster: group by fingerprint, then within groups apply fuzzy thresholds to merge clusters.- Manual review queue: present ambiguous pairs (scores in gray zone) to human annotators.

4) Logging & validation during EDA- Log every dedup action: source IDs, matched IDs, matching method, similarity score, timestamp, rule used.- Create audit table and store original records and merged output for reproducibility.- Validate: sample precision/recall by labeling a random sample of matched/non-matched pairs; plot similarity score distributions, precision-recall curves, and monitor downstream impact (e.g., label distribution shift, feature importance changes).- Maintain metrics: number of merges, percent reduction, false-positive rate; add roll-back capability.

This workflow balances automation (hashing, blocking, LSH) with oversight (threshold tuning, manual review, audit logs) suited for ML pipelines where data quality directly affects model performance.

HardTechnical

75 practiced

You receive a multimodal dataset for predictive maintenance: image file paths, multi-sensor time-series, and categorical metadata. Create a comprehensive EDA plan to characterize each modality, detect cross-modal anomalies (e.g., missing image with sensor anomaly), suggest candidate features, and list pitfalls such as timestamp misalignment and missing modalities. Include queries/visuals you would produce.

Sample Answer

Objective: characterize data quality and signal, identify anomalies (per-modality and cross-modal), propose candidate features for predictive models, and list pitfalls & mitigations.

1) Clarify scope & labels- Define failure/upcoming-maintenance window, sampling rates, timezone, ID keys linking modalities (asset_id, timestamp).

2) Per-modality EDA

Images- Inventory: count images per asset/time, resolution distribution, file-size histogram.- Visual checks: random sample grid, edge cases (blurry, dark).- Metadata: EXIF timestamps, camera id.- Code (pandas + PIL) to list missing files and sample images:

python

import pandas as pd, PIL.Image as Image
df_images = pd.read_csv('images.csv') # cols: asset_id, ts, path
df_images['exists'] = df_images['path'].apply(lambda p: os.path.exists(p))
df_images['filesize_kb'] = df_images['path'].apply(lambda p: os.path.getsize(p)/1024 if os.path.exists(p) else None)

- Visuals: histogram(filesize), heatmap of missing images over time per asset, montage of failure vs normal.

Time-series sensors- Aggregate counts, sampling intervals, missing-rate per sensor, PSD for periodicities, autocorrelation, cross-correlation across sensors.- Plots: timeline of sensor streams with flagged gaps, spectrograms.- Query (pandas):

python

df_s = pd.read_parquet('sensors.parquet') # cols: asset_id, ts, sensor, value
df_s.groupby(['asset_id','sensor']).agg({'ts':['min','max','count']})

- Anomaly detection: rolling z-score, STL decomposition residuals, isolation forest on sliding windows.

Categorical metadata- Cardinality, mode, missingness, rare categories, correlation with failures.- Visuals: bar charts, mosaic plots, embedding (target-mean encoding distribution).

3) Cross-modal checks & cross-modal anomaly detection- Join on asset_id+aligned ts (define tolerance window). Build a table with flags: image_exists, sensor_gap_flag, large_residual_flag, metadata_missing.- Queries/visuals:

sql

SELECT a.asset_id, a.ts, i.exists as image, s.gap_flag, m.meta_missing
FROM sensor_agg a
LEFT JOIN images i USING(asset_id, time_bucket)
LEFT JOIN metadata m USING(asset_id);

- Heatmap: co-occurrence matrix of image_missing vs sensor_anomaly; timeline examples where sensor spikes coincide with missing images.- Case examples: extract segments where sensors show fault signature but images missing -> possible capture failure.

4) Candidate features- Image-derived: defect heatmaps, CNN embeddings (ResNet), color histograms, blur score, object counts.- Sensor-derived: statistical window features (mean, std, skew, kurtosis), spectral features (FFT band energies), trend/derivative, time-to-failure features, change-point flags, multivariate PCA/TS embeddings.- Metadata: age, model_type_onehot/target-encoded, maintenance_history_count, operating_conditions (shift).- Cross-modal: time delta between sensor anomaly and closest image, image-sensor embedding similarity, missing_modality_indicator.

5) Pitfalls & mitigations- Timestamp misalignment: ensure timezone normalization, clock drift correction, use sync-events; resample to common timeline with careful interpolation.- Missing modalities: non-random missingness (e.g., camera fails at high vibration) — treat as signal; include missingness indicators.- Variable sampling rates: align via time-buckets, preserve high-frequency info via engineered features.- Label leakage: avoid using post-failure images/sensors within label horizon.- Class imbalance: use stratified sampling, anomaly-aware metrics (precision@k), calibration.- Data drift: monitor feature distributions per asset and retrain triggers.- Image preprocessing pitfalls: compression artifacts, viewpoint variation — use augmentation and domain-specific normalization.

6) Deliverables & visuals- Dashboard with: missingness heatmap, sensor gap timeline, PSD plots, sample image galleries, co-occurrence matrix, top correlated features with failure (SHAP).- Shortlist of top 30 features with provenance and stability score.

This plan emphasizes reproducible checks (code/notebooks), curated visual examples for root-cause, and features tailored for multimodal predictive maintenance while noting practical pitfalls to avoid in modeling and deployment.

HardSystem Design

69 practiced

Design a monitoring system to continuously track EDA-like metrics in production: schema changes, missing rate spikes, cardinality growth, new categories, distribution shifts, and sample drift. Describe where to store metrics, how to compute them (full-scan vs sampling), alerting logic, dashboards, retention, and integration points with CI/CD and incident response.

Sample Answer

Requirements (clarify): track per-feature and dataset EDA metrics continuously: schema changes, missing-rate spikes, cardinality growth, new categories, distribution shifts, sample (covariate) drift. Low-latency alerts for production-impacting changes; historical retention for root-cause.

High-level architecture:- Ingestion: instrument model-serving and ETL pipelines to emit example-level metadata (feature name, value, timestamp, record-id, source). Use Kafka/Kinesis for streaming; also support periodic batch snapshots.- Compute: stream processors (Flink/Spark Structured Streaming) compute rolling aggregates and sketches; batch jobs for full re-compute and baseline snapshots.- Storage: - Timeseries DB (Prometheus, InfluxDB, or TimescaleDB) for scalar metrics (missing rate, cardinality counts, drift scores) at configurable resolution. - Feature-store / Parquet object store (S3) for sampled raw examples and baseline snapshots. - Compact sketches (HyperLogLog for cardinality, Count-Min for frequency, KS/AD sketches) stored in object store or Redis for quick reads.Metric computation strategy:- Streaming aggregated metrics (per-minute/hour) via incremental updates using sketches (low memory) — suitable for high throughput.- Full-scan periodic jobs (daily/weekly) to compute precise baselines and to detect subtle distributional changes; used for model re-training triggers.- Sampling: keep reservoir/sample of raw records per dataset (e.g., 10k/day) for detailed distribution tests and human inspection.Alerting logic:- Multi-tiered alerts: - Immediate (P0): schema breaks (missing required feature, type mismatch) - trigger automatic rollback/stop-serving. - High (P1): sudden missing-rate spikes (>X sigma vs rolling window or >relative threshold), new unseen categories exceeding rate, cardinality explosion. - Medium (P2): distribution shift detected by statistical tests (KS/AD for continuous, PSI for binned; require repeatable signal over 3 windows), gradual cardinality growth.- Use adaptive thresholds: combine absolute rules + anomaly detection (exponential weighted moving average with alert on divergence) + significance tests with multiple-window confirmation to avoid flapping.- Alert routing: integrate with PagerDuty for P0/P1, Slack for P2, and ticket creation (Jira) for investigations.Dashboards & UX:- Grafana dashboards: overview (service-level), per-model, per-feature drill-down. Panels: missing-rate heatmap, cardinality trend, top new categories, drift score timeline, sample viewer (link to stored sample), schema diff viewer.- Provide “explain” panel showing which features contributed most to drift (feature importance-weighted drift).Retention & storage sizing:- Timeseries: high-resolution (1m) for 7 days, 5m for 30 days, hourly for 1 year. Samples and full baseline snapshots: keep 90 days of detailed samples, long-term monthly baselines archived to S3 for 3+ years.Integration with CI/CD & incident response:- CI: include data-contract tests and synthetic-data checks in pre-deploy pipeline; run canary data through new model with monitoring hooks; block deploy if schema mismatch or synthetic drift detected.- CD: canary rollout with traffic shadowing and metric gating (auto-promote if no alerts for X hours).- Incident playbooks: automated runbook links in alerts, automated collection of recent samples/feature distributions, one-click rollback to previous model version, and postmortem templates. Provide triage API for on-call to fetch top anomalous features and related traces.Trade-offs:- Sketches/sampling reduce cost and latency but miss rare edge-cases — mitigate with periodic full-scans.- Statistical tests can false-positive on small volumes — require multi-window confirmation and sample-size-aware thresholds.Why this design:- Combines low-latency detection (streaming + sketches) with high-fidelity periodic analysis (full-scan), provides actionable alerts, integrates with CI/CD to prevent bad deploys, and equips responders with data and playbooks to remediate quickly.

EasyTechnical

79 practiced

You have a numeric feature 'price'. As part of EDA list and justify at least four different visualizations or transformations you would use to understand its distribution and outliers (e.g., histogram, boxplot, log-transform). For each visualization explain what it reveals, what parameters you'd choose (bins, log scale), and when to use that approach.

Sample Answer

1) Histogram (with density overlay)- What it reveals: shape (skewness), modality, mass around ranges.- Parameters: 30–50 bins for large datasets; use Freedman–Diaconis rule or Scott’s rule to choose bin width automatically; overlay KDE with bandwidth chosen by Silverman.- When to use: first-pass view of distribution; compare raw vs transformed price; good for detecting multimodality or coarse outliers.

2) Boxplot (and violin plot)- What it reveals: central tendency, IQR, whiskers, and explicit outliers; violin adds full density shape.- Parameters: use default 1.5*IQR for whiskers; split by categorical (e.g., product category) to compare groups.- When to use: quick summary and robust outlier detection; useful before deciding trimming or capping per-group.

3) Log-transform + histogram / log-scale axis- What it reveals: reduces right skew, makes multiplicative structure linear, reveals relative differences for low vs high prices.- Parameters: apply log1p if zeros exist; plot histogram on linear axis of log(price) or use log-scale x-axis if leaving raw values.- When to use: when price spans orders of magnitude and you plan models that assume approximate normality or additive errors (e.g., linear regression).

4) ECDF (empirical CDF) or percentile plot- What it reveals: exact quantiles, tail behavior, proportion of values above thresholds (e.g., what % > $1000).- Parameters: plot percentiles (0–100) or mark key quantiles (5th, 25th, 50th, 75th, 95th).- When to use: to decide clipping/capping thresholds or to communicate business-relevant cutoffs.

5) QQ-plot (against normal) or log-normal QQ- What it reveals: whether distribution (or log-transformed distribution) follows a specified theoretical distribution; highlights tail deviations.- Parameters: plot raw and log(price) QQs.- When to use: before using parametric tests/model assumptions; to justify transformations.

Combine these: start with histogram + KDE, confirm with boxplot/violin, use log-transform if skew is severe, then ECDF/QQ to set thresholds and validate modeling assumptions.

Unlock Full Question Bank

Get access to hundreds of Exploratory Data Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.