Exploratory Data Analysis Questions

Exploratory Data Analysis is the systematic process of investigating and validating a dataset to understand its structure, content, and quality before modelling or reporting. Core activities include examining schema and data types, computing descriptive statistics such as counts, means, medians, standard deviations and quartiles, and measuring class balance and unique value counts. It covers distribution analysis, outlier detection, correlation and relationship exploration, and trend or seasonality checks for time series. Data validation and quality checks include identifying missing values, anomalies, inconsistent encodings, duplicates, and other data integrity issues. Practical techniques span SQL profiling and aggregation queries using GROUP BY, COUNT and DISTINCT; interactive data exploration with pandas and similar libraries; and visualization with histograms, box plots, scatter plots, heatmaps and time series charts to reveal patterns and issues. The process also includes feature summary and basic metric computation, sampling strategies, forming and documenting hypotheses, and recommending cleaning or transformation steps. Good Exploratory Data Analysis produces a clear record of findings, assumptions to validate, and next steps for cleaning, feature engineering, or modelling.

HardTechnical

69 practiced

A classification problem has an extremely imbalanced target and missing values that correlate strongly with the positive class. Propose an EDA-driven strategy for feature creation, sampling or weighting, validation schemes (e.g., stratified time splits), and steps to quantify the risk of optimistic bias or leakage introduced by handling the missingness.

Sample Answer

Situation & goal: We must build a classifier where positives are rare and missingness strongly correlates with positives. The priority is to use EDA to create informative features without inducing leakage, choose sampling/weighting and validation that reflect production, and quantify risk of optimistic bias from how we handle missingness.

EDA-driven feature creation- Start with univariate analyses: missingness rate per feature by target (P(Missing|Y=1) vs P(Missing|Y=0)), KS or chi-square tests and CIs to confirm association.- Visualize joint patterns: missingness heatmaps and UpSet plots; conditional distributions of other features given missingness.- Create explicit missingness indicator features (is_missing_X). These capture signal if missingness is informative.- For continuous features with missingness correlated to Y, consider imputation that preserves signal: e.g., a separate “missing” category or model-based imputation that uses only non-leaky features (avoid using future or target-proximal variables).- Engineer interactions: is_missing_X * important_feature, and counts of missing values per row.

Sampling / weighting strategy- Prefer class weighting in the loss function (inverse class frequency or focal loss) over aggressive oversampling to avoid amplifying missingness artifacts.- If oversampling positives (SMOTE), only apply it after creating missingness indicator features and inside training folds to avoid leakage. Use variant that respects sparsity and categorical patterns.- Consider calibrated probability weighting: sample negatives to match temporal or covariate distribution of positives (covariate-balanced downsampling).

Validation schemes to avoid optimistic bias- Use stratified time-aware splits if data is temporal: train on past, validate on future; ensure positive rate and missingness patterns in validation mimic production.- Within each time-split, stratify by target and (optionally) by missingness pattern so validation tests generalization across missingness regimes.- Nested CV for hyperparameter tuning where imputation, feature creation, and sampling are fully inside inner folds.- Hold out a final “blinded” test set untouched (no feature engineering leveraging test set stats).

Quantify risk of optimistic bias / leakage- Ablation experiments: train models (a) with missingness indicators, (b) with missing values imputed to neutral values without indicators, (c) with missingness indicators but masked during inference — compare AUC, precision-recall, calibration. Large drops indicate reliance on missingness.- Permutation tests: permute the missingness indicator across rows and measure performance change — if performance collapses, model heavily depends on missingness signal.- Leakage checks: confirm no feature engineering used future timestamps or aggregated stats computed across train+val. Recompute feature distributions by split; significant shifts suggest potential leakage.- Calibration and decision-curve analysis on holdout: inspect whether predicted probabilities reflect true risk across missingness strata.- Simulate production scenarios: create synthetic test sets where missingness association is weaker/stronger and measure stability of metrics.- Use SHAP or other explainability tools to ensure missingness indicators aren't just proxies for target leakage sources (e.g., label-derived timestamps).

Operational recommendations- If missingness is truly causal/legitimate signal (e.g., missing because action was taken), keep indicators but document and monitor their stability over time.- Monitor post-deployment: track P(Missing|time) and model performance by missingness subgroup; retrain when distribution shifts.- Prefer models that output calibrated probabilities and use monotonic constraints if domain knowledge supports them.

This approach balances capturing informative missingness while using strict fold separation, ablation/permutation checks, and temporal validation to quantify and limit optimistic bias or leakage.

EasyTechnical

75 practiced

Write a Python function using pandas that, given a DataFrame and a list of key columns, returns: (a) the number of duplicate groups (by keys), (b) a sample of duplicate rows, and (c) a deduplicated DataFrame keeping the first occurrence. Provide function signature, edge-case handling, and a brief usage example.

Sample Answer

To detect and handle duplicates by key columns in pandas, we can group by the keys and use duplicated() to identify repeated rows. The function below returns: (a) count of key-groups with >1 row, (b) sample of duplicate rows (all rows from duplicated groups, limited by n_samples), and (c) deduplicated DataFrame keeping the first occurrence.

python

import pandas as pd

def analyze_and_deduplicate(df: pd.DataFrame, keys: list, n_samples: int = 5):
    """
    Returns (num_dup_groups, sample_duplicates_df, deduped_df).
    - num_dup_groups: int, number of key groups with >1 row
    - sample_duplicates_df: DataFrame of up to n_samples rows from duplicated groups
    - deduped_df: DataFrame with duplicates (by keys) removed, keeping first occurrence
    """
    if not isinstance(df, pd.DataFrame):
        raise TypeError("df must be a pandas DataFrame")
    if not keys:
        raise ValueError("keys must be a non-empty list of column names")
    missing = [k for k in keys if k not in df.columns]
    if missing:
        raise KeyError(f"Keys not in DataFrame columns: {missing}")

    # Mark rows that are duplicates according to the key columns (keep=False marks all in groups)
    is_dup_group = df.duplicated(subset=keys, keep=False)

    # Number of groups with more than one row
    dup_groups = df[is_dup_group].groupby(keys).ngroups

    # Sample up to n_samples rows from duplicated groups
    sample_duplicates = df[is_dup_group].head(n_samples).copy()

    # Deduplicated DataFrame keeping the first occurrence of each key-group
    deduped = df.drop_duplicates(subset=keys, keep='first').reset_index(drop=True)

    return dup_groups, sample_duplicates, deduped

Key points:- Time: O(n) for passes over data; Space: O(n) for intermediate masks.- Edge cases handled: non-DataFrame input, empty/missing keys, keys not present, empty DataFrame.- Usage example:

python

df = pd.DataFrame({'id':[1,1,2,3,3],'value':[10,20,30,40,50]})
num, sample, deduped = analyze_and_deduplicate(df, keys=['id'], n_samples=3)

HardTechnical

115 practiced

You have hundreds of features with suspected multicollinearity. Propose a practical, scalable approach to detect and mitigate multicollinearity: include approximate VIF computation for large feature sets, correlation-based feature clustering, PCA or truncated SVD options, use of regularized models, and a plan to preserve interpretability for stakeholders.

Sample Answer

Situation: We have a high-dimensional feature set (hundreds) with suspected multicollinearity that threatens coefficient stability, model generalization, and interpretability. Below is a practical, scalable approach that detects, mitigates, and preserves stakeholder interpretability.

1) Quick detection — scalable approximate VIF- Exact VIF requires regressing each feature on others (O(p^3)). For large p, approximate VIF via random-projection / sketching: - Compute X_sketch = X · S where S is a d×m Gaussian or CountSketch (m ~ 50–200 << p). Compute approximate Gram G = X_sketch^T X_sketch. - For feature j, estimate R^2_j ≈ 1 - 1 / (1 + (X_j^T (X_{-j} X_{-j}^T)^{-1} X_j)). Use sketch to approximate (X_{-j}X_{-j}^T)^{-1} or compute leverage via randomized SVD. Approx VIF_j = 1/(1-R^2_j). - Complexity: O(n p m) for sketching + O(m^3) small inversion. Good to rank problematic features.

2) Correlation-based feature clustering (grouping)- Compute pairwise correlations using block-wise/approximate methods (approx nearest neighbors / locality-sensitive hashing for cosine).- Build graph where edges exist for |corr|>τ (τ=0.8). Find connected components or hierarchical clustering with average linkage.- For each cluster: choose representative(s): highest mutual information with target, domain-prioritized feature, or an aggregate (mean, PCA within cluster).

3) Dimensionality reduction options- Within-cluster PCA / truncated SVD: apply to large clusters, keep leading k components explaining e.g. 90% variance. Use randomized SVD for scalability (O(n p log k)).- Global PCA as last resort when interpretability less critical—use sparse PCA / varimax rotation to improve interpretability.- Keep metadata mapping original features → components.

4) Regularized models- Use elastic net (mix L1 + L2) to encourage sparsity while handling correlated groups; tune alpha/l1_ratio by CV.- Group Lasso when clusters (from step 2) define groups—encourages selection at group level.- Bayesian shrinkage (horseshoe) for uncertainty-aware selection if compute allows.

5) Preserve interpretability- Always keep cluster/aggregation metadata and representative selection rationale.- Use surrogate interpretable models: after a complex model (e.g., using PCA features or regularized model), train a shallow decision tree or rule list on model predictions for stakeholder-facing explanations.- Use feature-attribution methods (SHAP with grouping support): compute SHAP on original feature groups (sum contributions within clusters) so stakeholders see group-level importance.- For PCA/SVD components, provide component loadings and top contributing original features, and label components with domain-friendly names.

6) Practical workflow (steps)- EDA: compute approximate VIF and correlation clusters → flag clusters.- For each cluster: try representative selection + elastic net baseline. If predictive loss > tolerance, use cluster-PCA (1–3 comps).- Retrain final model (group-regularized or elastic net), validate stability via bootstrap: check coefficient variance and prediction delta.- Deliverables: model, mapping file (feature → cluster → component), SHAP grouped report, and decision log explaining why features were removed/aggregated.

Edge cases & notes:- Nonlinear collinearity: consider pairwise mutual information or kernel methods.- Time series / panel data: compute correlations within folds/time windows to avoid leakage.- Maintain reproducibility: seed randomness for sketching/SVD and store transformations.

This approach balances scalability (randomized linear algebra), predictive performance (regularization), and stakeholder interpretability (grouping, representative features, SHAP and surrogate explanations).

EasyTechnical

69 practiced

Using Python and scikit-learn or pandas, write a short function to produce a stratified sample of a DataFrame by a categorical column 'country'. The function must guarantee a minimum of 5 rows per stratum when available; for strata smaller than the minimum, include the entire stratum. Ensure reproducibility via random_state.

Sample Answer

Approach: iterate groups by 'country', compute a per-group sample size = max(ceil(group_size * frac), min_per_stratum) but not more than group_size. Use a fixed RandomState to draw an integer seed per group so sampling is reproducible but different across groups.

python

import math
import numpy as np
import pandas as pd

def stratified_sample(df, by='country', frac=0.1, min_per_stratum=5, random_state=None):
    """
    Return a stratified sample of df by column `by`.
    - frac: fraction to sample from each stratum
    - min_per_stratum: guarantee at least this many rows per stratum when available
    - strata smaller than min_per_stratum are included entirely
    - random_state: int seed for reproducibility
    """
    rng = np.random.RandomState(random_state)
    samples = []
    for name, group in df.groupby(by):
        n = len(group)
        # if group smaller than min, include whole group
        if n <= min_per_stratum:
            samples.append(group)
            continue
        # desired sample size: at least min_per_stratum, proportional otherwise
        desired = max(int(math.ceil(n * frac)), min_per_stratum)
        desired = min(desired, n)
        seed = int(rng.randint(0, 2**32 - 1))
        samples.append(group.sample(n=desired, random_state=seed))
    return pd.concat(samples).sample(frac=1, random_state=random_state).reset_index(drop=True)

Key points:- We compute per-stratum sizes, ensuring min_per_stratum when possible; small strata are fully included.- Using a single RandomState and drawing per-group seeds keeps sampling reproducible and avoids identical draws per group.- Final shuffle keeps rows mixed.

Time complexity: O(N) for grouping and sampling. Space: O(N) to build result.

Edge cases:- frac <= 0: returns only min_per_stratum or full small strata; validate inputs as needed.- If min_per_stratum > n for many groups, result size may exceed expectations.- If the DataFrame is huge, consider streaming or using chunked processing.

EasyTechnical

66 practiced

You observe a numeric column 'purchase_amount' with a heavy right skew and a long tail. During EDA, list the steps you would take to visualize and quantify skewness, identify extreme outliers, and prepare the variable for modeling. Discuss transformations (log, Box-Cox), winsorization, binning, and when to prefer each approach.

Sample Answer

Start with visualization and quantification, then detect extreme values, then decide preparation depending on model and business needs.

Visualization & quantification- Plot histogram + density (use log-x axis too), boxplot, violin, and empirical CDF. Add a QQ-plot to check tail behavior.- Compute summary stats: mean/median, IQR, skewness and kurtosis. Report robust metrics: median, MAD.- Show percentiles (99th, 95th) and count of zeros/near-zeros.

Identify extreme outliers- Rule-based: IQR rule (x < Q1 − 1.5·IQR or > Q3 + 1.5·IQR) and modified Z-score using MAD for heavy tails.- Tail-focused: flag above 95th/99th percentile for manual review.- Domain check: cross-reference suspicious high values with transaction logs (fraud, refunds).

Prepare for modeling — options & when to use them- Log transform (y' = log(x+ε)): simple, compresses right tail; requires x≥0 (add small offset if zeros). Good for linear models and when multiplicative effects expected; preserves order.- Box-Cox: finds power transform that best normalizes; requires strictly positive values. Use when normality helps modeling assumptions.- Yeo-Johnson: like Box-Cox but supports zeros/negatives.- Winsorization: cap extremes at chosen percentiles (e.g., 99th). Use when outliers are likely errors or would unduly drive parameter estimates but you want to keep rank/order.- Binning (quantile or business buckets): convert to categorical if relationship is non-linear or interpretability is key; loses fine-grained info.- Leave as-is: tree-based models (random forests, gradient boosting) tolerate skewness; consider using raw values + engineered flags for extreme high values.

Practical pipeline- Impute/clean errors first, log/Box-Cox or Yeo-Johnson inside a scikit-learn transform so it’s reproducible, optionally winsorize before or after transform depending on intent, then scale if required (standardize for linear/Ridge; not needed for trees).- Evaluate via cross-validation: compare model metrics and residuals for different handling strategies; prefer the simplest approach that improves validation performance and business interpretability.

Edge considerations: many zeros (consider zero-inflated or separate indicator), heteroscedasticity (transform helps), and interpretability for stakeholders (binning or percent-change reporting).

Unlock Full Question Bank

Get access to hundreds of Exploratory Data Analysis interview questions and detailed answers.

Join thousands of developers preparing for their dream job.