Edge Case Identification and Testing Questions

Focuses on systematically finding, reasoning about, and testing edge and corner cases to ensure the correctness and robustness of algorithms and code. Candidates should demonstrate how they clarify ambiguous requirements, enumerate problematic inputs such as empty or null values, single element and duplicate scenarios, negative and out of range values, off by one and boundary conditions, integer overflow and underflow, and very large inputs and scaling limits. Emphasize test driven thinking by mentally testing examples while coding, writing two to three concrete test cases before or after implementation, and creating unit and integration tests that exercise boundary conditions. Cover advanced test approaches when relevant such as property based testing and fuzz testing, techniques for reproducing and debugging edge case failures, and how optimizations or algorithmic changes preserve correctness. Interviewers look for a structured method to enumerate cases, prioritize based on likelihood and severity, and clearly communicate assumptions and test coverage.

MediumSystem Design

99 practiced

Design a canary and smoke-test strategy to catch edge cases when rolling out a new model version: which smoke tests run on 1% of real traffic, which synthetic tests run in pre-production, how you evaluate key metrics for rollback, and how you automate a rollback if canary metrics violate safety thresholds. Include example safety checks that specifically test edge-case inputs.

Sample Answer

Requirements:- Catch correctness/regression and edge-case failures before full rollout- Minimal user impact (start at 1% real traffic)- Fast detection and automated rollback if unsafe- Pre-prod synthetic coverage for rare inputs

High-level architecture:- CI/CD → Pre-prod test harness (synthetic + replay) → Canary gateway (1% real traffic) → Monitoring & Alerting → Canary controller (automated rollback)

Pre-production synthetic tests (run on every model build):- Unit-level validation: schema, type, range, and model output invariants- Distributional tests: compare feature distributions to training set (KS tests)- Adversarial/edge-case suite: - Empty/missing features - Extremely long text, Unicode, emoji, and control chars - Out-of-vocab tokens / unseen categories - Extremely large numeric values, infinity/NaN - Time-zone / locale anomalies- Golden-path replay: run historical transactions with known expected labels and measure accuracy/regression- Performance/load smoke: latency and memory under expected concurrency

Canary (1% real traffic) smoke tests:- Lightweight correctness: sample inputs → check output validity (not null, in-bounds)- Latency P95/P99 and error rate (HTTP 5xx, timeouts)- Business KPIs computed on online labels or delayed feedback proxies (e.g., click-through, conversion rate delta)- Drift detectors: feature drift compared to baseline window- Safety checks for edge-case inputs: run injected edge cases and check responses same as pre-prod

Key metrics & rollback conditions:- Latency: P95 > baseline + 50% OR P99 breach → fail- Error rate: increase > 3σ above baseline or absolute > 0.5% → fail- Model quality: immediate proxy metric drop (e.g., CTR down > 5% relative or AUC drop > 0.02) → fail- Anomaly flags: new NaN outputs, high percentile of identical outputs (>10%), or invalid classes produced → fail- Drift/Distribution: KS p-value < 0.01 on key features → warn/conditional rollback

Automation:- Canary controller polls metrics stream (Prometheus/Kafka) and evaluates rules every minute.- If any hard threshold violated, controller: 1) Pauses routing to canary 2) Shifts traffic back to previous model version (automated config change via feature flag / service mesh) 3) Triggers alert to on-call with diagnostics bundle (request samples, stack traces, metric deltas) 4) Creates rollback ticket and marks build as failed in CI- Soft warnings ramp down canary (1% → 0.1%) and require human review.

Example safety checks for edge-case inputs:- Null/missing: send 100 requests with missing critical fields; assert no 5xx, outputs default or safe class, and confidence not > 0.9 (indicating overconfident guess)- Long-text fuzz: send text >10x typical length; assert latency < 2x baseline and no truncation error- Unseen categorical: send rare category; assert model returns calibrated low confidence or fallback behavior- Numeric overflow: very large floats; assert model yields safe output and no NaNs- Repeated identical input flood: 1k identical requests; assert no degradation or stateful leakage

Why this approach:- Synthetic pre-prod catches deterministic bugs and rare edge cases cheaply- 1% real traffic exposes unforeseen production distributions- Automated thresholds + safe rollback minimize user impact while surfacing real regressions for investigation.

MediumTechnical

85 practiced

Class imbalance is an important edge scenario. Design tests to ensure evaluation metrics are correct and informative under extreme class imbalance. Provide example test datasets and expected metric outcomes for these cases: no positive labels in ground truth, a single positive among 1,000,000 negatives, and perfect predictions. Discuss which metrics (precision, recall, F1, AUROC, AUPRC) are informative or misleading in these settings and how to encode these tests in automation.

Sample Answer

Approach: create deterministic small and large synthetic datasets covering edge cases, compute standard metrics with robust handling (treat undefined as 0 or np.nan per policy), and assert expected values or properties (monotonicity, bounds). Use sklearn for reproducibility.

Example test datasets & expected outcomes:1) No positives in ground truth- y_true = [0,0,0,0]- y_score = [0.1,0.9,0.2,0.4] or y_pred all zerosExpected:- Precision: undefined (no predicted positives) → treat as 0 or assert sklearn returns 0 with zero_division=0- Recall: 0 (no true positives)- F1: 0- AUROC: 0.5 if scores vary (model no-skill); if no positive labels sklearn raises error — assert raised or handle.- AUPRC: undefined (no positives) — often 0 or undefined; assert behavior explicitly.

2) Single positive among 1,000,000 negatives- y_true length = 1_000_001, one 1 at index 0- y_score all zeros except score[0]=1.0Case A: perfect ranking (positive highest)Expected:- Precision@k (k=1): 1.0- Recall: 1.0- F1: 1.0- AUROC: ~1.0 (perfect)- AUPRC: ~1.0Case B: model gives low score to positiveExpected:- Precision ~0 (very low)- Recall 0- AUROC can still be ~0.5 if scores random — misleading because class imbalance makes AUROC insensitive.

3) Perfect predictions- y_true = [1,0,1,0], y_pred = [1,0,1,0], y_score alignedExpected: precision=recall=F1=1.0, AUROC=1.0, AUPRC=1.0.

Which metrics informative vs misleading:- Precision/Recall/F1: informative when you evaluate at decision threshold or top-k; F1 may hide trade-offs — report precision and recall separately.- AUROC: can be misleading under extreme imbalance; it measures ranking across negatives and will remain high even if positive is ranked slightly above many negatives. Not sensitive to performance at top of ranking.- AUPRC: more informative for rare positives because it focuses on positive class performance; prefer AUPRC and precision@k.- For "no positives" cases, AUPRC and recall are undefined — tests must encode expected behavior (raise, NaN, or defined fallback).

Automation encoding (example using sklearn + pytest):

python

from sklearn.metrics import precision_score, recall_score, f1_score, roc_auc_score, average_precision_score
import numpy as np
import pytest

def test_no_positives():
    y_true = np.zeros(4, dtype=int)
    y_pred = np.zeros(4, dtype=int)
    assert precision_score(y_true, y_pred, zero_division=0) == 0
    assert recall_score(y_true, y_pred) == 0
    assert f1_score(y_true, y_pred, zero_division=0) == 0
    with pytest.raises(ValueError):
        roc_auc_score(y_true, np.array([0.1,0.2,0.3,0.4]))
    assert np.isnan(average_precision_score(y_true, np.array([0.1,0.2,0.3,0.4]))) or average_precision_score(y_true, np.array([0.1,0.2,0.3,0.4]))==0

def test_single_positive_perfect():
    N=1_000_001
    y_true = np.zeros(N, int); y_true[0]=1
    scores = np.zeros(N); scores[0]=1.0
    assert roc_auc_score(y_true, scores) == 1.0
    assert average_precision_score(y_true, scores) == 1.0

Best practices:- Explicitly decide how to handle undefined metrics (raise vs fallback).- Assert metric ranges and monotonic properties rather than exact floats when large datasets cause numerical noise (use approx).- Include precision@k and calibration checks for practical relevance.

MediumTechnical

71 practiced

Given a labelled dataset with timestamps, describe the concrete tests and checks you would run to detect label leakage before training. Include automated checks (for example, verifying no feature has greater predictive power on future labels than past labels), exploratory checks (feature-time correlations), and sample SQL or pandas checks to identify features with timestamps after the label timestamp. Describe what to do if you find potential leakage.

Sample Answer

Start with clear definition: label leakage = any feature computed using information not available at prediction time (including future timestamps). Approach: automated checks, exploratory checks, and targeted queries.

Automated checks- Temporal feature importance: train two models — one using only features with timestamp <= label_time (past-only) and one allowing all features. If “all” model greatly outperforms past-only, suspect leakage.- Lead/lag predictive test: compute feature predictive score on labels shifted forward/backward in time. A feature that predicts future labels much better than past labels is suspicious.- Timestamp consistency: ensure feature_timestamp <= label_timestamp for each row.

Exploratory checks- Plot correlation/importance vs. time delta (feature_time - label_time). Look for spikes near positive deltas.- Grouped summaries: check mean/variance of features for small positive deltas.

Pandas checks

python

# assumes df has feature_ts, label_ts columns per feature or dict of timestamps
df['delta'] = (df['feature_ts'] - df['label_ts']).dt.total_seconds()
leak_rows = df[df['delta'] > 0]
leak_fraction = len(leak_rows)/len(df)

SQL check (per feature):

sql

SELECT COUNT(*) AS leak_count,
       COUNT(*)*1.0/ (SELECT COUNT(*) FROM table) AS leak_frac
FROM table
WHERE feature_timestamp > label_timestamp;

Lead/lag predictive test (pandas/sklearn): train model to predict label shifted by +k and -k days and compare AUCs.

If leakage found- Trace source and confirm whether data legitimately uses future info.- Remove or truncate offending features (e.g., aggregate only up to label_time).- Engineer proper lookback windows and use rolling/expanding aggregations aligned to label_time.- Re-evaluate model; add automated CI checks to block future leakage (timestamp assertions, unit tests on feature pipelines).

EasyTechnical

101 practiced

Describe three realistic off-by-one or boundary errors that commonly appear in data science workflows (for example inclusive vs exclusive window endpoints in rolling aggregates; splitting time-series at boundaries that leak the last training row; index vs label-based slicing mismatch). For each example explain a minimal reproducible test (input and expected output) that would catch the error and a unit test you would write to prevent regression.

Sample Answer

1) Inclusive vs exclusive rolling-window endpoints- Error: Pandas rolling with closed='left' vs default includes/excludes current row, causing aggregates to leak current target.- Minimal reproducible test: Input series: [1,2,3,4], window=2, expected rolling sum with excluding current (previous 2): [NaN,1,3,5]- Unit test (pytest):

python

import pandas as pd
import numpy as np

def test_rolling_excludes_current():
    s = pd.Series([1,2,3,4])
    # window of 2 looking at prior values only
    res = s.rolling(2, closed='left').sum()
    expected = pd.Series([np.nan,1.0,3.0,5.0])
    pd.testing.assert_series_equal(res.reset_index(drop=True), expected)

2) Train/validation split leakage at boundary (last training row repeated)- Error: Splitting by index positions with inclusive endpoint duplicates the last training timestamp in validation.- Test: Input timestamps ['2020-01-01','2020-01-02','2020-01-03'], split_date='2020-01-02' Expected train timestamps ['2020-01-01'], val ['2020-01-02','2020-01-03']- Unit test:

python

def test_time_split_exclusive_train():
    df = pd.DataFrame({'t':pd.to_datetime(['2020-01-01','2020-01-02','2020-01-03'])})
    train = df[df.t < pd.Timestamp('2020-01-02')]
    val = df[df.t >= pd.Timestamp('2020-01-02')]
    assert list(train.t.dt.date) == [pd.Timestamp('2020-01-01').date()]
    assert len(train) + len(val) == len(df)

3) Index-based vs label-based slicing mismatch- Error: Using iloc vs loc on a DatetimeIndex causes off-by-one when labels are inclusive.- Test: Index: dates 1..5; slice loc['3':'4'] -> should include both, iloc[2:4] -> includes same two; ensure functions use same semantics.- Unit test:

python

def test_loc_vs_iloc_equivalence_for_integer_labels():
    idx = pd.date_range('2021-01-01', periods=5)
    s = pd.Series(range(5), index=idx)
    a = s.loc[idx[2]:idx[3]]
    b = s.iloc[2:4]
    pd.testing.assert_series_equal(a, b)

Each test is minimal, deterministic, and prevents regressions by codifying intended inclusive/exclusive semantics.

MediumTechnical

69 practiced

When you replace a simple correctness-first implementation with a vectorized or approximate version for performance, how do you ensure the optimized version preserves correctness across edge cases? Describe a test strategy that includes equivalence tests on representative inputs, property tests on invariants, tolerance-based numerical comparisons, and regression tests against historical outputs.

Sample Answer

Start with the correctness-first implementation as the ground truth and treat the optimized (vectorized/approximate) version as the candidate. My test strategy has four pillars: equivalence, property-based, tolerance-aware numeric checks, and regression tests.

1) Equivalence tests on representative inputs- Build a curated suite: typical cases, edge cases (empty, singletons, extreme values, NaNs/infs), boundary sizes, and pathological distributions.- Run the candidate vs ground-truth and assert identical outputs where exactness is required (e.g., categorical results) or within tolerance for floats.

2) Property (invariant) tests- Encode invariants: monotonicity, conservation (sum/count), idempotence, commutativity where applicable.- Use property-based testing (Hypothesis) to generate randomized inputs constrained to realistic domains and verify invariants hold.

3) Tolerance-based numerical comparisons- Use relative and absolute tolerances depending on scale: - assert |a-b| <= atol + rtol * |ground_truth|- For approximate algorithms, test distributional similarity (KS test), summary statistics (mean/std), and downstream impact (model AUC delta).Example comparison snippet:

python

import numpy as np

def close_enough(a, b, rtol=1e-6, atol=1e-9):
    return np.all(np.isclose(a, b, rtol=rtol, atol=atol, equal_nan=True))

4) Regression tests against historical outputs- Snapshot canonical inputs and outputs (including seeds for RNG) and include as CI fixtures.- Record performance metrics and fail builds if correctness regresses or if numerical drift exceeds historical bounds.

Additional practices- Seed RNGs, fix float dtype, and test multiple vectorization backends (NumPy, Numba).- Run fuzz tests and long-running stochastic checks nightly.- When failing, add minimal counterexample to the regression corpus.

This layered approach ensures the optimized version is fast while preserving correctness and measurable, acceptable approximation.

Unlock Full Question Bank

Get access to hundreds of Edge Case Identification and Testing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.