End To End Data Preprocessing & Exploration Questions

Follow a systematic, tool-agnostic data pipeline before deeper analysis: load the data, check shape and dtypes, identify missing values and duplicates, explore distributions, check for outliers, understand class or category balance where relevant, and summarize key statistics. Document findings and build visualizations that surface relationships in the data. This exploration is the foundation for whatever comes next: feature engineering and model selection for predictive/ML work, or clean aggregations and trustworthy KPIs for dashboards and reporting.

MediumTechnical

30 practiced

For fine-tuning an LLM on a domain corpus, outline an end-to-end cleaning pipeline: deduplicate near-identical documents, normalize whitespace, strip boilerplate (headers/footers), detect and remove non-linguistic artifacts (HTML, code fences), and preserve provenance metadata. Discuss heuristics and tools to scale the cleaning.

Sample Answer

Pipeline overview (end-to-end):

1. Ingest & metadata capture- Collect source, URL/filepath, crawl timestamp, extractor version, content-id, and mime-type. Store as immutable provenance fields so every downstream record links to origin.

2. Initial normalization- Unicode NFC, normalize newlines to \n, collapse repeated whitespace to single space except preserve paragraph breaks (double \n). Lower/upper-case only if domain-appropriate. Trim BOMs.

3. Boilerplate/header/footer stripping- Heuristics: detect repeated top/bottom n-grams across documents from same source; detect short lines with copyright, navigation, timestamps using regexes; template clustering (TF-IDF + k-means) to learn common boilerplate.- Tools: jusText, Boilerpipe for HTML; Readability-like extractors.

4. Remove non-linguistic artifacts- HTML: run an HTML parser (BeautifulSoup/lxml) then extract visible text; preserve alt text for images as metadata.- Code fences: regex for triple backticks / indentation; classify blocks with simple language detectors and either remove or tag as code_block with language.- Binary blobs/URLs/long hashes: strip or replace with placeholders preserving counts.- OCR noise: detect high non-word ratio and low char-vocab; flag for manual review.

5. Deduplicate near-identical documents- Token-based shingling (w-shingles), compute MinHash or SimHash signatures, cluster via LSH; within cluster keep canonical doc with best provenance and length; store dedup-score and merge provenance lists.- Also do sentence-level dedupe for repeated boilerplate.

6. Quality filters & labeling- Language detection (fastText/cld3); length thresholds; perplexity heuristics using small LM to detect gibberish; remove or flag low-quality docs.

7. Provenance preservation & lineage- When merging/deduping, keep array of source entries, original offsets, and transformation logs (actions applied, timestamp, tool versions). Ensure traceability back to raw file.

8. Scaling & tooling- Use streaming pipelines (Apache Beam/Flink) or Spark for heavy transformations.- Store intermediate artifacts in columnar format (Parquet) with provenance columns.- Use Faiss or Datasketch LSH libraries for large-scale similarity; dedupe in sharded LSH bins.- Orchestrate with Airflow/Kubernetes; use GPU nodes only for ML-based filters.- Monitoring: sample-based human audits, metrics (dupe rate, language distribution, token counts), and reproducible random seeds.

Heuristics summary:- Use conservative removal rules, tag instead of delete when uncertain.- Prioritize provenance and reproducibility: version every extractor and maintain logs.- Balance precision/speed: regex + parser for most cases, ML classifiers for ambiguous content.

MediumTechnical

28 practiced

You receive irregularly timestamped sensor readings. Describe preprocessing steps to prepare supervised ML: resampling to a fixed frequency, handling missing timestamps, interpolation strategies, generating lag and rolling features, and choosing train/validation splits to avoid leakage. Give examples where forward-fill is inappropriate.

Sample Answer

Approach: treat irregular sensor readings by converting them to a regular time grid, imputing missing values with methods appropriate to sensor dynamics, engineering temporal features (lags/rollings), and splitting data to avoid temporal leakage.

Steps:1. Resample to fixed frequency- Choose frequency based on fastest meaningful event (e.g., 1s/1min). Upsample (insert timestamps) or downsample (aggregate via mean/median/max).- When downsampling, preserve event info with aggregations (count, max, std).

2. Handle missing timestamps- Mark missing vs observed with a boolean mask — keeps model aware of gaps.- Impute using domain-appropriate methods: - Forward-fill (ffill) for slowly-varying states (e.g., thermostat setpoint). - Linear interpolation for smooth physical sensors (temperature drift). - Spline/GP/Kalman for noisy sensors requiring uncertainty-aware interpolation. - Model-based imputation (autoencoders, seq2seq) when patterns complex.

Examples where forward-fill is inappropriate:- Event sensors (motion, button presses) — ffill would create false positives.- Cumulative counters (energy meters) — ffill hides resets; interpolation breaks monotonicity.- Rapidly-changing dynamics where holding last value misrepresents trend.

3. Lag and rolling features- Create multiple lags at meaningful windows (t-1, t-5, t-60) and rolling stats (mean, std, min, max, percentiles) with window sizes chosen from domain/time-constants.- Include time-since-last-observation and missing-rate features.- Use exponentially-weighted moving averages to emphasize recent values.

4. Train/validation splits to avoid leakage- Use time-based splits (no shuffling). For forecasting, use rolling-origin (walk-forward) CV.- Ensure any lookback window used in features is fully inside train/validation segments (exclude overlap).- For sensors with periodicity, stratify validation across cycles (e.g., daily) to test generalization.- If events are rare, ensure validation contains representative events but never contains future data relative to train.

Key trade-offs: finer resampling increases imputation error and compute; coarser loses detail. Always log decisions, validate imputation by holding out observed points, and consider uncertainty-aware models when imputations are noisy.

EasyTechnical

25 practiced

Explain the different patterns of missingness: MCAR, MAR, and MNAR. Give concrete examples from AI datasets for each pattern and describe appropriate preprocessing strategies and the potential impact each pattern has on model training and evaluation, including bias considerations and diagnostic checks.

Sample Answer

MCAR (Missing Completely At Random)- Definition: Missingness is independent of observed and unobserved data — essentially random.- Example (AI): Random frames dropped from a video dataset due to logging glitches; random words missing in tokenized chat logs because of transient I/O error.- Preprocessing: Simple imputation (mean/median for features), row-wise deletion if fraction small, or model-based imputation. Methods like listwise deletion give unbiased estimates under MCAR.- Impact & bias: Minimal bias introduced; primary cost is loss of statistical power. Diagnostics: Little’s MCAR test, compare distributions of observed vs full (if possible), check missingness unrelated to covariates.

MAR (Missing At Random)- Definition: Missingness depends on observed data but not on the missing value itself.- Example (AI): In a medical imaging dataset, higher-resolution scans are more likely for older patients; image quality (missing metadata) correlates with device model recorded.- Preprocessing: Use conditional imputation (multiple imputation, MICE), include predictors of missingness in models, or use models that handle missingness (e.g., tree-based models, attention with mask). Ensure imputation models include the observed covariates driving missingness.- Impact & bias: If handled correctly, estimators can be unbiased; ignoring MAR can bias models (e.g., systematically underrepresent certain subgroups). Diagnostics: Model missingness indicator as target vs observed features — significant predictors suggest MAR.

MNAR (Missing Not At Random)- Definition: Missingness depends on the unobserved value itself.- Example (AI): Users censor toxic comments, so toxicity labels are missing precisely when text is highly toxic; in face datasets, people with disfiguring conditions opt out, so absence correlates with the feature.- Preprocessing: Harder — need explicit missingness models (selection models, pattern-mixture models), sensitivity analyses, or collect auxiliary data. Consider methods that jointly model data and missingness; avoid naive imputation.- Impact & bias: Can cause severe, systematic bias; model may learn skewed distributions and propagate harm. Diagnostics: Compare observed vs expected under plausible MNAR mechanisms, perform sensitivity analysis (vary assumptions), or attempt follow-up data collection to validate.

Practical checklist for AI engineers:- Always create missingness indicators as features.- Explore correlations between missingness and observed features.- Use multiple imputation or model-based approaches for MAR; prefer sensitivity analyses and explicit modeling for MNAR.- Report missing-data assumptions and evaluate model robustness under alternative missingness scenarios to surface possible biases.

EasyTechnical

34 practiced

You're given a dataset where numeric columns are read as strings and dates are inconsistent. Describe step-by-step how you'd detect wrong dtypes and convert them safely in Python/pandas, including robust date parsing, coercing errors, preserving original raw values for auditing, and handling locale-dependent numeric formats like '1,234.56' vs '1.234,56'. Provide example code snippets.

Sample Answer

Approach summary:1) Inspect columns to detect likely-misparsed dtypes (strings that look numeric/date).2) Save raw values for auditing.3) Convert numeric strings with locale-aware logic using pandas.to_numeric or babel for ambiguous separators.4) Parse dates robustly with dateutil.parser and pandas.to_datetime with dayfirst/infer_datetime_format and errors='coerce'.5) Log/retain rows that failed conversion for review.

Example implementation:

python

import pandas as pd
from dateutil import parser
from babel.numbers import parse_decimal, NumberFormatError

def detect_string_like(series, sample_n=100):
    s = series.dropna().astype(str).head(sample_n)
    pct_numeric = s.str.match(r'^[\d\-\+.,/\s]+$').mean()
    pct_date_like = s.apply(lambda x: bool(parser.parse(x, fuzzy=False)) if len(x)>3 else False).mean() if not s.empty else 0
    return pct_numeric, pct_date_like

def safe_to_numeric(series, locale=None):
    raw = series.copy()
    def conv(x):
        if pd.isna(x): return None
        x = str(x).strip()
        if locale:
            try:
                return float(parse_decimal(x, locale=locale))
            except NumberFormatError:
                return pd.NA
        # try smart replacements: detect if thousand sep is '.' and decimal is ','
        if x.count(',') and x.count('.')==0:
            return pd.to_numeric(x.replace('.','').replace(',','.'), errors='coerce')
        return pd.to_numeric(x.replace(',',''), errors='coerce')
    out = series.map(conv)
    return out, raw

def safe_to_datetime(series, dayfirst=False):
    raw = series.copy()
    out = pd.to_datetime(series, errors='coerce', dayfirst=dayfirst, infer_datetime_format=True)
    # fallback: try dateutil for messy strings
    mask = out.isna() & series.notna()
    def try_parse(x):
        try:
            return parser.parse(str(x), dayfirst=dayfirst)
        except Exception:
            return pd.NaT
    out.loc[mask] = series[mask].map(try_parse)
    return out, raw

# Usage
df = pd.read_csv('data.csv', dtype=str)  # read all as strings to avoid surprises
# For each column:
for col in df.columns:
    pct_num, pct_date = detect_string_like(df[col])
    if pct_num > 0.6:
        df[f'{col}_raw'] = df[col]                  # preserve raw
        df[col], df[f'{col}_raw'] = safe_to_numeric(df[col], locale='de_DE')  # try locale if known
    elif pct_date > 0.4:
        df[f'{col}_raw'] = df[col]
        df[col], _ = safe_to_datetime(df[col], dayfirst=True)
# After conversion: inspect rows with failures
failures = df[[c for c in df if df[c].isna().any()]]

Key points:- Always keep original raw column for audit.- Use locale-aware parsing when formats ambiguous.- Coerce errors to NaN/NaT and record them for manual review.- Log conversion statistics (counts of coerced values) to decide corrective steps.

EasyTechnical

51 practiced

Describe how the IQR method identifies outliers for a numeric feature and implement a Python function that flags outliers using both 1.5 and 3.0 IQR multipliers. Explain strengths and limitations of IQR for skewed or multimodal distributions and alternatives you might use.

Sample Answer

The IQR (interquartile range) method flags outliers by computing Q1 (25th percentile) and Q3 (75th percentile), IQR = Q3 - Q1, and marking points below Q1 - k*IQR or above Q3 + k*IQR as outliers (common k values: 1.5 for mild, 3.0 for extreme).

Approach:- Compute Q1, Q3, IQR- Compute lower and upper fences for both k=1.5 and k=3.0- Return boolean masks or labels for each multiplier

python

import numpy as np
import pandas as pd

def flag_iqr_outliers(arr, multipliers=(1.5, 3.0), return_bounds=False):
    """
    Flags outliers using IQR method for given multipliers.
    arr: 1D array-like (list, numpy, pd.Series)
    multipliers: iterable of k values
    Returns: dict mapping k -> boolean mask (True = outlier)
    Optionally returns bounds if return_bounds=True
    """
    x = pd.Series(arr).dropna().astype(float)
    q1 = x.quantile(0.25)
    q3 = x.quantile(0.75)
    iqr = q3 - q1
    results = {}
    bounds = {}
    for k in multipliers:
        lower = q1 - k * iqr
        upper = q3 + k * iqr
        # Use original index to align with possible NaNs
        mask = ~pd.Series(arr).isna() & ((pd.Series(arr) < lower) | (pd.Series(arr) > upper))
        results[k] = mask.values
        bounds[k] = (lower, upper)
    return (results, bounds) if return_bounds else results

Key points:- Time: O(n) to compute percentiles; Space: O(n) for masks.- Strengths: Simple, fast, robust to extreme values (uses medians/quantiles), non-parametric.- Limitations: For skewed distributions, fixed IQR multipliers may flag many legitimate tail points or miss skewed outliers; for multimodal data, IQR ignores multiple clusters and may treat entire modes as outliers.- Alternatives: Use robust z-scores (median and MAD), transformation (log/Box-Cox) before IQR, model-based methods (Gaussian mixture for multimodal), isolation forest or local outlier factor for complex patterns. Choose method based on domain knowledge and downstream impact.

Unlock Full Question Bank

Get access to hundreds of End To End Data Preprocessing & Exploration interview questions and detailed answers.

Join thousands of developers preparing for their dream job.