Edge Case Identification and Testing Questions

Focuses on systematically finding, reasoning about, and testing edge and corner cases to ensure the correctness and robustness of algorithms and code. Candidates should demonstrate how they clarify ambiguous requirements, enumerate problematic inputs such as empty or null values, single element and duplicate scenarios, negative and out of range values, off by one and boundary conditions, integer overflow and underflow, and very large inputs and scaling limits. Emphasize test driven thinking by mentally testing examples while coding, writing two to three concrete test cases before or after implementation, and creating unit and integration tests that exercise boundary conditions. Cover advanced test approaches when relevant such as property based testing and fuzz testing, techniques for reproducing and debugging edge case failures, and how optimizations or algorithmic changes preserve correctness. Interviewers look for a structured method to enumerate cases, prioritize based on likelihood and severity, and clearly communicate assumptions and test coverage.

HardSystem Design

95 practiced

For a multi-stage streaming inference pipeline (feature extraction -> model A -> model B -> aggregator), design integration and chaos tests that simulate partial failures: delayed upstream features, dropped messages, backpressure, and out-of-order arrival. Specify assertions that validate correctness of aggregated outputs under these failures and tests that measure latency and graceful degradation.

Sample Answer

Requirements & constraints:- Functional: pipeline stages: Feature Extractor → Model A → Model B → Aggregator produce per-key aggregated predictions.- Non-functional: tolerate delayed, dropped, reordered messages; provide bounded staleness, backpressure handling, and graceful degradation with SLOs (latency, accuracy).- Test goals: validate correctness under partial failures, measure latency and degradation.

High-level test architecture:- Use streaming platform (Kafka / Pulsar) for durable event bus, and a stream processing framework (Flink / Kafka Streams) to run the pipeline in test mode. Deploy each stage as a containerized service so we can inject faults.- Test harness: synthetic traffic generator that creates canonical input traces with ground-truth expected aggregates and timestamps. Observability: distributed tracing, per-event unique IDs, metrics exporter (Prometheus), and test assertion component that subscribes to final aggregator topic.

Core integration & chaos test cases (how to inject + expected behavior):

1) Delayed upstream features- Injection: introduce tail-latency by adding artificial sleep in Feature Extractor for a subset of keys (e.g., 10% keys delayed by 500ms–5s).- Mechanisms under test: watermarks & event-time windows, buffering window, late-arrival handling policy (emit-with-late-updates vs drop).- Assertions: - For keys delayed within configured allowed-lateness (L): final aggregator emits initial result within latency SLO and then emits a corrected update within allowed-lateness. Assert: corrected result equals ground truth; number of corrected updates <= threshold. - For delays > L: assert aggregator marks result as partial/stale (flag) and accuracy degrades gracefully (measured drop < X%).

2) Dropped messages (loss)- Injection: randomly drop N% of messages between Feature Extractor → Model A or Model A → Model B (simulate network loss).- Mechanisms: at-least-once vs exactly-once semantics, deduplication, retry/backoff.- Assertions: - If system guarantees at-least-once with retries: final aggregated value equals ground truth within error tolerance. - If permanent drop: aggregator outputs with confidence/coverage metadata; assert coverage fraction ≈ (1 – drop_rate) within ±epsilon and aggregated metric degrades proportionally. - Measure counts: input_count - output_count = dropped_count; system retries logged.

3) Backpressure propagation- Injection: slow Model B (CPU throttling) causing downstream queues to fill; set upstream producers to limited buffer sizes.- Mechanisms: Reactive backpressure (Flink reactive scaling), upstream throttling, circuit-breakers, bounded queues with overflow policy.- Assertions: - System should maintain stability: no unbounded memory growth (monitor heap / queue length). - Latency percentiles: P50, P95, P99 increase but remain within SLO thresholds or trigger graceful-degradation path (e.g., switch Model B to cheap fallback model). - Assert fallback engaged when queue-length > threshold and aggregated outputs produced with fallback tag.

4) Out-of-order arrival- Injection: shuffle event order for a subset of keys (simulate reordering across partitions).- Mechanisms: event-time processing, sequence numbers, idempotent processing, watermark advancement rules.- Assertions: - With correct watermarking and allowed-lateness, final aggregates match ground truth; no double-counting. Verify by per-event unique ID dedup table size = expected. - If reordering exceeds buffer window: aggregator emits partial then corrected—assert corrections count and final correctness.

Test specifics and pass criteria:- Inputs: deterministic trace with N keys and per-key expected aggregated value (e.g., sum/probability ensemble), each event includes event_time, key, seq_id.- Run scenarios with controlled randomness seeds. For each scenario collect: emitted events (with timestamp), correction flags, coverage/confidence metadata, queue lengths, CPU/memory.- Assertions: - Correctness: final aggregator value per key within tolerance δ of ground-truth (e.g., L2 error < 0.01) after convergence window T_conv. - Ordering/idempotence: no duplicate contributions; verify by checking each seq_id applied at most once. - Staleness: percentage of keys that had no update within latency SLO <= allowed_failure_rate. - Graceful degradation: when faults exceed thresholds, system must switch to fallback and maintain minimum utility metric (e.g., aggregated accuracy >= baseline_A). - Resource safety: memory & queue length never exceed configured caps.

Latency & degradation measurement:- Measure end-to-end latency per event (ingest_time → final_emit_time) and emit time of corrections.- Report P50/P95/P99 before, during, after fault injection.- Define SLOs and thresholds: e.g., normal P95 < 200ms; under backpressure P95 < 2s; P99 spikes allowed but must be bounded.- Degradation metrics: - Accuracy vs latency curve: plot model ensemble accuracy as function of induced delay/drop. - Availability: fraction of keys with updated aggregates in T_window. - Recovery time: time to return to baseline post-fault.

Test automation & practices:- Run chaos tests in CI with small-scale cluster, and regularly in staging with production-like load.- Use fault injection tools: Chaos Mesh (k8s), Toxiproxy (network faults), Istio fault injection, or built-in test harness that toggles behavior flags.- Collect traces and metrics; make assertions deterministic by using fixed random seeds and bounded variance.- Add canary tests: progressive increase of fault intensity and automatic gating if metrics cross thresholds.

Trade-offs & notes:- Larger allowed-lateness buffers improve correctness under reordering/delay but increase memory and end-to-end latency.- Exactly-once strong semantics reduce correctness risk but add throughput/complexity cost.- Design test pass criteria to reflect acceptable business degradation: e.g., slight accuracy drop acceptable in favor of meeting latency SLO or vice versa.

This test suite validates functional correctness, resilience, observability, and graceful degradation for the multi-stage streaming inference pipeline under partial failures.

EasyTechnical

94 practiced

Write two pytest unit tests (pseudocode acceptable) for a training pipeline function train(dataset) that must raise ValueError when the dataset is empty and log a warning when the dataset contains only a single class label. Explain how tests should assert both functional behavior and logged telemetry.

Sample Answer

Approach: use pytest to assert (1) train(dataset) raises ValueError for empty dataset, and (2) train logs a WARNING when all labels are the same. Use pytest.raises for the exception and pytest's caplog fixture to capture logging output (or monkeypatch the logger). Tests below are pseudocode but executable with minimal adaptation.

python

# test_training_pipeline.py
import pytest
from my_training_module import train

def test_train_raises_on_empty_dataset():
    empty_dataset = []  # or DataFrame with zero rows
    with pytest.raises(ValueError) as excinfo:
        train(empty_dataset)
    assert "empty" in str(excinfo.value).lower()  # functional assertion

def test_train_logs_warning_for_single_class(caplog):
    # dataset where every example has label 0
    single_class_dataset = [
        {"features": [0.1, 0.2], "label": 0},
        {"features": [0.3, 0.4], "label": 0},
    ]
    caplog.clear()
    with caplog.at_level("WARNING"):
        train(single_class_dataset)
    # telemetry assertion: there should be at least one warning mentioning 'single class' or similar
    warnings = [r.message for r in caplog.records if r.levelname == "WARNING"]
    assert any("single class" in str(m).lower() or "only one class" in str(m).lower() for m in warnings)

Key points:- Use pytest.raises to validate functional error handling.- Use caplog to capture and inspect logged telemetry; set appropriate level and search messages.- For more robust checks, assert logger name, exact message, or a metric/telemetry call via monkeypatch.

EasyTechnical

87 practiced

List common off-by-one and boundary errors that occur when handling padded sequences (for example, calculating effective length, attention masks, slicing windows). For each of three examples, describe a short unit test and an integration test demonstrating the bug and its fix.

Sample Answer

Common off-by-one and boundary errors with padded sequences typically arise when you mix token indices, lengths, and exclusive vs inclusive ranges. Below are three concrete examples, each with a short unit test and an integration test that demonstrate the bug and its fix.

1) Incorrect effective length (counting pad)- Bug: effective_length = seq.size(1) - tokens.count(pad) — but using <= or wrong count can off-by-one when sequences have no pads or all pads.- Unit test: input tokens [A,B,PAD,PAD] expected effective_length=2; also test [A,PAD,PAD, PAD] expected=1 and [PAD, PAD] expected=0. Assert function returns these exact integers.- Fix: compute effective_length = sum(token != PAD for token in seq) (explicit boolean sum) or use first_pad_index with min(index, seq_len).- Integration test: run batch through encoder that masks positions >= effective_length; before fix model attends to one pad position causing higher loss on short sequences. After fix, validate loss decreases slightly on dataset with many short sequences and attention weights on pads are ≈0.

2) Off-by-one in attention mask range (inclusive vs exclusive)- Bug: creating mask with mask[:, :L] = 1 but using L = effective_length - 1, excluding last token; or using <= which includes an extra pad.- Unit test: effective_length=3 -> expected mask [1,1,1,0,...]. Assert mask equals expected for several lengths (0,1,last).- Fix: be explicit: mask = (positions < effective_length).astype(int). Use half-open intervals.- Integration test: compare perplexity on sequences where the last real token is informative. Before fix model ignores last token (worse perplexity); after fix perplexity improves and attention alignment scores focus correctly.

3) Sliding window slicing overflow (end index clamps wrong)- Bug: window_end = min(start + window_size, seq_len - 1) — this subtracts one twice, producing truncated windows when start+window_size == seq_len.- Unit test: seq_len=10, start=5, window_size=5 -> expected slice indices [5:10]; buggy gives [5:9]. Test multiple starts including start=6 where clamp should shrink.- Fix: use window_end = min(start + window_size, seq_len) and slice [start:window_end) using python semantics.- Integration test: run a local attention implementation that aggregates over sliding windows; before fix some tokens never see full context leading to degraded F1 on span extraction near sequence ends. After fix, verify F1 improves and gradient flow reaches end tokens.

General testing tips:- Always test edge cases: zero-length, full-length, length exactly equal to window or batch dims.- Use property tests (randomized lengths) to catch off-by-one.- Make masked assertions on attention sums and gradient non-NaN to detect silent inclusion of pads.

EasyTechnical

99 practiced

Explain what an "edge case" is in the context of AI systems and model pipelines. Give three concrete examples that commonly break ML systems: one for NLP (e.g., long/empty/malformed text), one for computer vision (e.g., extremely dark image or alpha channel), and one for tabular data (e.g., NaNs, extreme outliers, swapped columns). For each example, briefly explain why it matters and outline the first two automated tests you would write to detect it.

Sample Answer

An edge case in AI/model pipelines is an input or situation that’s rare, atypical, or outside training distribution and therefore likely to cause failures or unpredictable behavior. Detecting and testing for edge cases prevents silent degradation, safety issues, and costly downstream errors.

1) NLP — Empty / extremely long / malformed text- Why it matters: Models can throw tokenization errors, exceed context windows, or produce garbage when input is empty or malformed (e.g., binary blobs, broken encoding).- Tests: 1. Unit test: feed empty string, whitespace-only, and null; assert graceful fallback (validation error, default reply, or no-op) and no exceptions. 2. Boundary test: feed strings around max_context_length+/- and malformed UTF-8; assert truncation policy, predictable truncation marker, and no OOM.

2) Computer Vision — Extremely dark/transparent images (alpha channel)- Why it matters: Preprocessing (normalization, color-space conversion) or augmentations may fail; model confidence can be meaningless for near-black or alpha-only inputs.- Tests: 1. Pixel-distribution test: synthetic image of all zeros and near-zero variance; assert preprocessing doesn't divide by zero and outputs expected tensor stats. 2. Channel-shape test: image with alpha channel or unexpected channel count; assert detection and either proper conversion (drop/merge alpha) or rejection.

3) Tabular Data — NaNs, extreme outliers, swapped/misordered columns- Why it matters: Imputation, scaling, and feature mapping assume schema; NaNs/outliers skew models; swapped columns produce wrong semantics.- Tests: 1. Schema/shape test: input with columns renamed/swapped or extra/missing columns; assert schema validation failure and clear error. 2. Distribution test: columns with >X% NaNs or values outside historical quantiles (e.g., >1e6); assert alerts and blocked training/inference or automatic fallback (impute/clipping).

These tests should be integrated into CI and monitoring to catch regressions and data-drift early.

MediumTechnical

99 practiced

You fine-tuned a transformer and suspect an off-by-one error in label alignment between tokenized inputs and output labels. Propose a set of unit tests and small synthetic examples to detect misalignment for cases like subword tokenization, sequence truncation, and special tokens. Show a minimal example input-output pair that would reveal the bug.

Sample Answer

Approach summary: create small deterministic tokenization examples and assert label length and per-token label mapping. Tests cover subword splitting (word -> multiple tokens), truncation (labels truncated or shifted), and special tokens (CLS/SEP added). Use a real tokenizer (HuggingFace) or a minimal mock to reproduce off-by-one.

Example unit tests (pytest-style):

python

from transformers import AutoTokenizer
import pytest

tok = AutoTokenizer.from_pretrained("bert-base-uncased")

def align_labels(tokens, word_labels):
    # Implementation under test: maps word-level labels to token-level labels
    # e.g., propagate first subword label to following subwords or use -100 for them
    ...

def tokens_from_text(text):
    return tok(text, return_offsets_mapping=True, add_special_tokens=True)

def test_subword_alignment():
    text = "playing"
    # "play" + "##ing" -> two tokens
    enc = tok(text, return_offsets_mapping=True, add_special_tokens=True)
    labels = ["VERB"]  # one word-level label
    token_labels = align_labels(enc, labels)
    # Expect same label on first subtoken, and -100 (or same) on continuation depending policy
    assert token_labels[1] == "VERB" or token_labels[1] == -100

def test_special_tokens_shift():
    text = "John"
    enc = tok(text, add_special_tokens=True)
    labels = ["B-PER"]
    token_labels = align_labels(enc, labels)
    # ensure label aligns to token index of "John", not off by one due to [CLS]
    cls_index = 0
    assert token_labels[cls_index] in (None, -100)

def test_truncation_behavior():
    text = "A " + "word " * 1000
    enc = tok(text, truncation=True, max_length=10, return_overflowing_tokens=False)
    labels = ["O"] * (len(text.split()))
    token_labels = align_labels(enc, labels)
    assert len(token_labels) == enc['input_ids'].__len__()

Minimal input-output pair that reveals an off-by-one (subword + special token):- Text: "Unhappy"- Tokenization (bert-base-uncased): ["[CLS]", "un", "##happy", "[SEP]"]- Word label: ["ADJ"]If implementation mistakenly shifts labels by one, you might see labels aligned as:["[CLS]": "ADJ", "un": -100, "##happy": -100, "[SEP]": -100]Correct alignment should map "ADJ" -> token index 1 ("un") (or index 2 depending whether you keep CLS). A failing assertion: token_labels[1] == "ADJ" will catch off-by-one.

Why these tests catch bugs:- Subword test ensures multi-token words don't drop or shift labels.- Special-token test ensures you don't assign labels to [CLS]/[SEP] or shift everything by their presence.- Truncation test ensures labels/truncation are consistent and lengths match.

Best practice: run with both policies (propagate labels to subwords vs mark as -100) and assert behavior explicitly. Mock tokenizer offsets for deterministic unit tests.

Unlock Full Question Bank

Get access to hundreds of Edge Case Identification and Testing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.