Statistical Fundamentals and Exploratory Analysis Questions

Core descriptive and exploratory statistical techniques used to summarize data, detect patterns, and generate testable hypotheses. Covers measures of central tendency and dispersion such as mean median and standard deviation, distributional assumptions, frequency and cross tabulation, visualization for exploration, cohorting and segmentation, identifying biases and data quality issues, and designing exploratory analyses to suggest causal hypotheses. Understand when to apply EDA to prepare data for formal tests and how to translate exploratory findings into confirmatory analyses. Candidates should demonstrate ability to summarize quantitative data, detect anomalies, and propose appropriate follow up hypothesis tests or experiments.

HardTechnical

58 practiced

Explain heteroscedasticity in regression residuals. Describe formal and graphical tests to detect heteroscedasticity (e.g., Breusch-Pagan, White test, residual vs fitted plot) and discuss remedies including transformations, weighted least squares, and robust standard errors. Provide code sketch for performing White's test in Python/statsmodels.

Sample Answer

Heteroscedasticity: residual variance depends on predictor values (Var(ε|X)=σ^2(X)), violating OLS assumption of constant variance. Consequence: OLS coefficients remain unbiased but standard errors (and thus t/Z-tests, CIs) are invalid — leading to misleading inference.

Detection — graphical:- Residuals vs fitted plot: look for “funnel” or pattern (increasing/decreasing spread). Also plot sqrt(|residuals|) vs fitted or vs key predictors.- Scale-Location plot (spread ~ fitted) and residuals vs individual predictors help locate which X relates to variance.

Formal tests:- Breusch–Pagan: regress squared residuals on original X (or subset). Null: homoscedasticity. Sensitive to linear variance patterns.- White test: regress squared residuals on X, X^2 and cross-products (flexible for non-linear heteroscedasticity). Null: homoscedasticity. More general but can lose power with many regressors.- Goldfeld–Quandt: splits data by suspected variance-driving variable.Report p-values and complementary visual checks.

Remedies:- Transform response (log, Box–Cox): can stabilize variance and sometimes linearize relationships. Use when transformation is interpretable.- Weighted Least Squares (WLS): weight by 1/Var(ε|X) estimate (or proxy), gives efficient, unbiased estimates with correct SEs if weights correct. Iteratively reweighted least squares if variance function unknown.- Robust (heteroscedasticity-consistent) standard errors: e.g., HC0–HC4 (sandwich estimators). Easy to apply (preserves OLS point estimates; fixes inference). Use when you trust model form but not constant variance.- Model variance explicitly (e.g., GLS, mixed models) or use bootstrap for SEs.

Python sketch — White test (statsmodels):

python

import statsmodels.api as sm
import statsmodels.formula.api as smf
from statsmodels.stats.diagnostic import het_white, het_breuschpagan

# fit OLS
model = smf.ols('y ~ x1 + x2', data=df).fit()

# Breusch-Pagan
bp_test = het_breuschpagan(model.resid, model.model.exog)
# returns (lm_stat, lm_pvalue, f_stat, f_pvalue)

# White test
white_test = het_white(model.resid, model.model.exog)
# returns (lm_stat, lm_pvalue, f_stat, f_pvalue)

print('BP p:', bp_test[1], 'White p:', white_test[1])
# Robust SEs example
robust_model = model.get_robustcov_results(cov_type='HC3')
print(robust_model.summary())

Practical advice: always combine plots and tests; if inference is primary and model form is plausible, prefer WLS/GLS or robust SEs; if prediction matters, transformations or models that directly model variance often help.

MediumTechnical

69 practiced

A categorical feature has very high cardinality (e.g., 1M unique product IDs). For exploratory analysis and feature engineering, describe strategies you would apply to summarize and transform this variable so it is useful for modeling and visualization. Mention trade-offs for each approach.

Sample Answer

Start by clarifying the modelling goal (prediction vs. interpretability) and available side information (product taxonomy, prices, categories). Then apply one or more of these strategies:

1) Top-K + "other"- Replace rare product_ids by an "other" bucket after keeping top-K by frequency or revenue.- Pros: simple, reduces cardinality, interpretable.- Cons: loses per-product signal for long tail; choice of K is heuristic.

2) Frequency / Count encoding- Encode each id as its count or log-count.- Pros: captures popularity signal, single numeric feature.- Cons: conflates popularity with causal effect; sensitive to time shifts.

3) Target (mean) encoding with smoothing and CV- Replace id with smoothed mean(target) using global prior and K-fold out-of-fold estimates to avoid leakage.- Pros: strong predictive power.- Cons: leakage/overfitting risk if not properly regularized; reduces interpretability.

Example (python):

python

# out-of-fold target encoding sketch
import numpy as np
def oof_target_encode(train, col, target, folds):
    encoded = np.zeros(len(train))
    for tr_idx, val_idx in folds:
        stats = train.iloc[tr_idx].groupby(col)[target].agg(['sum','count'])
        prior = train[target].mean()
        smoothed = (stats['sum'] + prior*10) / (stats['count'] + 10)
        encoded[val_idx] = train.iloc[val_idx][col].map(smoothed).fillna(prior)
    return encoded

4) Hashing / Feature hashing- Hash IDs into B buckets (e.g., 2^10–2^16).- Pros: fixed memory, fast, no lookup table.- Cons: collisions introduce noise; less interpretable.

5) Learned embeddings- Train embeddings (neural nets or entity embeddings) for downstream models.- Pros: captures latent similarities; good for deep models.- Cons: needs large data and training; less interpretable.

6) Aggregate features & metadata- Join product attributes (category, price stats, supplier) and create aggregates: avg price per product, conversion rate, recency.- Pros: robust, interpretable, reduces reliance on raw id.- Cons: needs external data and engineering.

EDA & visualization tips:- Plot top-N bar, Pareto (cumulative coverage), distribution of counts, heatmaps of product vs. target metric.- Inspect long-tail behavior and time-sliced counts to detect drift.

Trade-offs summary:- Simpler encodings (counts, top-K) are interpretable but may underfit; target encoding and embeddings give power but risk leakage/overfit and reduce interpretability; hashing trades collisions for scalability. Use cross-validation, smoothing, and monitor for concept drift. Combine approaches (e.g., top-K + target-encode top items + count-encode rest) to balance signal and stability.

MediumTechnical

74 practiced

Design a systematic approach to detect and quantify data quality issues in a new production table (50M rows): duplicates, impossible values, schema drift, null patterns, and inconsistent formats. What metrics would you compute nightly to measure data quality and how would you report them to stakeholders?

Sample Answer

Approach — high level framework1) Discover & baseline: run an initial profiling job over the whole table to capture schema, data types, value distributions, unique key candidates, null rates, regex patterns, min/max, and representative samples. Store this baseline as the "golden profile".2) Implement nightly automated checks that compare the new snapshot against the baseline and business rules. Use a mix of full scans (where affordable) and lightweight incremental checks (hashes, partitions, sampling) because the table is 50M rows.

Systematic checks to run- Duplicates - Candidate PK uniqueness: count distinct(pk_col) vs total rows. Also compute MD5 of concatenated business-key columns and count duplicate hashes. - Partitioned duplicate check: run dedupe per partition (e.g., date/day) to scale.- Impossible values (validity) - Range checks for numeric columns (min/max vs allowed ranges). - Categorical domain checks (set membership). - Cross-field business rules (e.g., end_date >= start_date).- Schema drift - Track column set, types, nullable flags. Flag any added/removed columns or type changes. - Record sample row-level schema (JSON schema) and field-level type mismatch rates.- Null patterns (completeness) - Per-column null% and per-row null-count distribution. - Frequent null-combinations (which groups of columns are null together). - Time-series of null% by partition (detect sudden spikes).- Inconsistent formats - Regex validation rates for formatted strings (emails, phone, ISO dates). - Normalization failures (e.g., non-UTF8, whitespace, inconsistent casing).- Distribution drift (statistical) - Compare histograms for key numeric and categorical columns using Jensen-Shannon distance or KL divergence, and KS test for numerics.

Scaling: sampling & incremental- Full nightly scan if compute allows (50M is feasible on modern clusters). Otherwise: - Maintain per-partition aggregates (counts, mins/maxs, hash-signatures). - Compute change-detection by comparing partition-level row counts and sketches (HyperLogLog) and min/max. - Use stratified sampling for heavy columns to compute regex/format checks.

Nightly metrics to compute (key metrics, with suggested thresholds)- Row count delta (abs and %): unexpected big drops/joins → alert.- Completeness: - per-column null% (alert if increase > X% or > threshold, e.g., >5% for important features). - fraction of required rows missing critical fields.- Uniqueness: - duplicate_rate = 1 - distinct(pk)/total_rows (alert if > 0). - top duplicate groups (sample).- Validity: - invalid_rate per rule (e.g., out-of-range values) with top offending values.- Schema: - schema_change_flag (boolean) + list of column/type changes.- Format consistency: - regex_failure_rate per column.- Distribution drift: - JS distance per column and a flag if > threshold (e.g., 0.1).- Freshness: - max(data_timestamp) lag behind expected ingestion time.- Data quality score (composite): weighted sum of normalized metrics (completeness, uniqueness, validity, drift) to provide a single health indicator.

Reporting and stakeholder communication- Dashboards: build a daily dashboard (Tableau/Power BI/Looker) showing: - Top-level daily DQ score and trend - Row count delta, freshness - Top 10 failing checks and per-column metrics (null%, invalid%, duplicate_rate) - Distribution drift heatmap - Drill-down links to sample offending rows and the raw query used- Alerts: - Critical alerts (email/Slack) for severe issues: schema_change_flag, row_count drop > X%, duplicates > Y%, or freshness lag > SLA. - Informational alerts for metric changes (e.g., null% up by 2x) sent to data owners.- Nightly digest: - Automated email/Slack message summarizing the DQ score, top 3 issues, and links to drill-downs and a replay SQL or notebook to reproduce the failing rows.- Playbooks & ownership: - For each alert include owner (data producer / ETL team), severity, reproducible query, sample rows, and suggested remediation steps. - Maintain runbook for triage steps: rollback, source-check, schema-mapping fix.

Why this works- Baseline + nightly comparisons catch regressions early.- Mix of full scans and incremental diagnostics balances cost and completeness for 50M rows.- Combining statistical drift with rule-based checks reduces false positives.- Clear metrics, thresholds, owners and automated drill-downs enable fast remediation and trust for downstream modeling.

Example actionable metric set to display nightly- DQ Score: 0.0–1.0- Rows ingested: 50,123,450 (Δ +0.4%)- Freshness lag: 2 hours- Duplicate rate: 0.03% (top dup key sample link)- Columns with null% > threshold: 3 (list)- Schema change: false- Avg JS distance (monitored columns): 0.04- Top 5 format failures (column, failure_rate, sample)

Implement quickly using an orchestration job (Airflow), profiling library (Great Expectations / Deequ) and a reporting layer (Looker/Tableau + Slack webhook).

HardTechnical

77 practiced

Explain multiple hypothesis testing and false discovery control. Implement the Benjamini-Hochberg FDR correction in Python: given an array of p-values, return adjusted p-values and which hypotheses are significant at a given FDR level. Compare with Bonferroni correction and discuss trade-offs.

Sample Answer

To control error when testing many hypotheses, we distinguish family-wise error rate (FWER — probability of any false positive) and false discovery rate (FDR — expected proportion of false positives among rejected). Bonferroni controls FWER by dividing alpha by m (very conservative, reduces power). Benjamini–Hochberg (BH) controls FDR: rank p-values, find largest k with p_(k) <= (k/m) * q, reject all p <= p_(k). BH yields higher power while allowing some controlled proportion of false discoveries.

Approach: implement BH to return adjusted p-values (q-values-like) and a boolean array of significance at level q. Use step-up and monotone adjustment via cumulative minima.

python

import numpy as np

def benjamini_hochberg(pvals, q=0.05):
    """
    pvals: 1D array-like of p-values
    q: target FDR level (e.g., 0.05)
    Returns: adj_pvals (same order), reject (boolean array)
    """
    p = np.asarray(pvals, dtype=float)
    if p.ndim != 1:
        raise ValueError("pvals must be 1D")
    m = len(p)
    # handle NaNs: keep positions
    nan_mask = np.isnan(p)
    order = np.argsort(p, kind='mergesort')  # stable
    ranked = p[order]
    # compute BH critical values
    ranks = np.arange(1, m+1)
    # avoid division by zero if all NaN
    crit = (ranks / m) * q
    # find largest k where p_(k) <= crit
    valid = ~np.isnan(ranked)
    k = 0
    if valid.any():
        le = np.where(ranked[valid] <= crit[valid])[0]
        if le.size:
            k = le.max() + 1  # ranks are 1-based
    # rejections
    reject = np.zeros(m, dtype=bool)
    if k > 0:
        reject_indices = order[:k]
        reject[reject_indices] = True
    # adjusted p-values (Benjamini–Hochberg adjusted, monotone)
    # adj_p_(i) = min_{j>=i} { m/j * p_(j) }
    adj = np.full(m, np.nan)
    if valid.any():
        adj_ranked = np.empty(m)
        with np.errstate(divide='ignore', invalid='ignore'):
            adj_ranked = (m / ranks) * ranked
        # enforce monotonicity from largest to smallest
        adj_ranked = np.minimum.accumulate(adj_ranked[::-1])[::-1]
        adj_ranked = np.clip(adj_ranked, 0, 1)
        adj[order] = adj_ranked
    # ensure NaNs preserved
    adj[nan_mask] = np.nan
    return adj, reject

Key points:- Time complexity O(m log m) dominated by sort; space O(m).- Edge cases: NaNs preserved, p-values of 0 handled, all p equal handled.Comparison with Bonferroni:- Bonferroni: adjusted_p = min(p*m,1), controls FWER -> fewer false positives but lower power.- BH: controls FDR -> more rejections, better power in high-dimensional testing.Trade-offs:- Use Bonferroni when any false positive is costly (clinical trials). Use BH when tolerating some false discoveries for greater discovery yield (exploratory genomics, feature selection). BH assumes independent or positively dependent tests; for arbitrary dependence, use Benjamini–Yekutieli (more conservative).

HardTechnical

78 practiced

During EDA you suspect label leakage: a feature that perfectly predicts the target but is not available at prediction time. Describe methods to detect and confirm label leakage, examples of common leakage sources, and remediation strategies to prevent leakage from reaching model training.

Sample Answer

Situation: During EDA I found a feature that had near-perfect correlation with the target. Before trusting it, I treated this as possible label leakage.

Detection & confirmation (practical steps):- Temporal sanity check: ask “Would this feature be known at prediction time?” If it’s recorded after the event (e.g., discharge code, invoice paid flag), it’s likely leakage.- Holdout / forward-chaining validation: train on earlier time windows, validate on later windows. Leakage often collapses predictive performance when applied in truly prospective splits.- Permutation / feature-shuffle test: permute that feature—if model AUC drops massively, it’s influential; then retrain without it to see real-world drop. If performance only occurs with that feature, suspicious.- Target-only model / univariate check: compute mutual information, point-biserial or chi-square with target. Extremely high values vs other features is a red flag.- Leakage provenance: trace upstream ETL — join keys, timestamps, look at feature generation code. Ask data owners where/when it’s written.

Common leakage sources (examples):- Post-event flags: “was_fraud_reviewed”, “refund_issued”- Aggregates computed with future data: rolling mean including target window- ID mappings joined from future tables (e.g., churn label join)- Leaky keys: user segments derived from label or target-derived clustering

Remediation strategies:- Remove or freeze leaky features; drop any feature not available at prediction time.- Recompute features using only training-window data (causal/temporal feature engineering).- Use proper time-aware pipelines: feature-store with timestamps, online vs offline feature parity.- Adopt forward-chaining CV and backtesting as validation standard; include a “no-peek” policy in code reviews.- Instrument tests: unit tests that assert feature timestamp <= prediction_timestamp, and data-contract checks on joins.- If a useful signal exists but is unavailable at prediction time, engineer proxy features that are causal and observable earlier.

Result / principle: Treat any disproportionate feature importance as a process question — confirm with temporal validation and ETL tracing, then remediate with temporal-safe features and stricter pipelines so leakage never reaches production.

Unlock Full Question Bank

Get access to hundreds of Statistical Fundamentals and Exploratory Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.