Software Testing and Assertions Questions

Core software testing and debugging practices, including designing tests that exercise normal, edge, boundary, and invalid inputs, writing clear and maintainable unit tests and integration tests, and applying debugging techniques to trace and fix defects. Candidates should demonstrate how to reason about correctness, create reproducible minimal failing examples, and verify solutions before marking them complete. This topic also covers writing effective assertions and verification statements within tests: choosing appropriate assertion methods, composing multiple assertions safely, producing descriptive assertion messages that aid debugging, and structuring tests for clarity and failure isolation. Familiarity with test design principles such as test case selection, test granularity, test data management, and test automation best practices is expected.

HardTechnical

63 practiced

Propose an approach to generate adversarial examples as part of CI to test robustness while keeping runtime manageable. Describe fast attack methods (e.g., FGSM), sampling strategies (small seed set), and fail criteria (e.g., drop in accuracy beyond epsilon). Explain how to avoid flakiness and limit compute.

Sample Answer

Approach (high-level): add a lightweight adversarial-robustness stage in CI that runs fast, deterministic attacks on a small, representative seed set every push and a larger budget nightly. Use single-step/low-iteration attacks (FGSM, one- or two-step variants) for speed; reserve stronger attacks (multi-step PGD) to scheduled runs. Fail CI only when degradation is statistically meaningful.

Fast attack methods:- FGSM (single-step): x_adv = x + ε * sign(∇_x L); very fast (one backward pass).- Randomized FGSM / R-FGSM: add small random start then FGSM to catch gradient masking.- Lightweight iterative (2–5 iters) with small step size for nightly/regression runs.

Sampling strategy:- Small seed set: 50–200 examples per CI push, stratified by label/confidence/edge cases (low-confidence, boundary, high-loss). Choose examples once per model version and rotate monthly.- Two-tier schedule: - Push-level (smoke): 32–64 examples, FGSM ε tuned (e.g., 2–8/255 for images; scaled for embedding norm in NLP). - Nightly/regression: 200 examples, R-FGSM or 3-5 step iterative.

Fail criteria and thresholds:- Primary metric: model accuracy on adv set vs clean baseline. Fail if: - Absolute drop > delta (e.g., drop ≥ 5 percentage points), OR - Relative drop > p% (e.g., ≥ 10%), OR - Accuracy on adv set < minimum threshold (e.g., < 80% of clean).- Use paired tests or run N repeats to ensure significance: require failure observed in 2/3 deterministic runs to avoid flakiness.

Avoiding flakiness:- Deterministic pipeline: fix RNG seeds for attacks and batching; log seeds.- Freeze model weights snapshot used for CI; compare against stored baseline metrics.- Use same preprocessing and hardware type in CI to reduce nondeterminism.- Require reproducibility: store adversarial examples and gradients for triage.- Statistical smoothing: run small number of repeats (e.g., 3) and use median or majority to decide.

Limiting compute:- Use single backward pass per sample (FGSM) for push-level.- Batch adv generation on GPU; cap total samples per job to budget (e.g., 2k forward/backwards per job).- Precompute adversarial perturbations for static seed sets when model architecture unchanged; or compute on cheaper nodes/spot instances.- Early-exit: stop attack generation if model already fails fast (e.g., misclassified after first pass).- Prioritize examples by tier (most impactful first) so failing early triggers exit.

Operational considerations:- Track trends: store per-example clean vs adv metrics to detect slow regressions.- Alerting: only block CI for clear failures; otherwise create ticket with severity based on magnitude and test tier.- Governance: maintain epsilon and seed sets as part of model release notes; tune thresholds per model/task.

Why this works:- FGSM and R-FGSM give good coverage cheaply; stratified small seed sets find regressions without huge cost. Determinism and statistical rules reduce flakiness. Two-tier scheduling balances fast feedback and deeper nightly checks.

HardTechnical

62 practiced

Design an automated process that creates a reproducible minimal failing example for flaky or failing training jobs. Describe what artifacts to capture (seed, code commit hash, small dataset slice, model checkpoint, environment), how to reduce size of artifacts, and an automated script that attempts to reproduce and validate the problem.

Sample Answer

High-level approach: capture a compact, deterministic snapshot of everything needed to reproduce the failure, then automatically reduce artifacts (dataset, model, randomness) to a minimal failing example and run reproducibility attempts in isolated environments until validated or deemed non-reproducible.

Artifacts to capture- Code: git commit hash + patch of local uncommitted changes (git diff).- Config: full training config/flags (JSON/YAML).- Randomness: RNG seeds for Python, NumPy, framework (torch.manual_seed, tf.random.set_seed), CUDA deterministic flags.- Dataset slice: original input pointers + export of the minimal subset used in the failing run (see reduction).- Model state: latest checkpoint and a small checkpoint if available.- Logs & metrics: stdout/stderr, framework logs, stack traces, profiler traces.- Environment: OS, python version, package versions (pip freeze), GPU driver, CUDA/cuDNN versions, Dockerfile or container image digest.- Hardware trace: GPU memory, power, host load if available.

Artifact reduction strategies- Binary-search dataset slicing: run on progressively smaller slices (half-size, quarter) to find smallest subset that still fails.- Input delta debugging: remove single examples or features and re-run to find minimal trigger set.- Model size reduction: train shorter epochs, smaller batch, or freeze layers to isolate.- Deterministic replay: ensure all randomness fixed; if flaky, introduce repeated runs to detect nondeterminism.- Compress artifacts: use dataset serialization (TFRecord/Parquet) and gzip; store diffs not full repos.

Automated reproduction script (outline)- Creates clean container (Docker) from captured environment or pulls image.- Checks out commit and applies patch.- Restores packages (pip install -r requirements.txt).- Loads minimal dataset slice and checkpoint.- Sets all RNG seeds and deterministic flags.- Runs training script for N attempts (e.g., 5) capturing exit code, logs, and stack traces.- Validates failure by checking error patterns or metric divergence.

Example reproducibility runner (Python wrapper)

python

#!/usr/bin/env python3
import subprocess, json, os, shutil, sys, time

CONF = "repro_config.json"  # contains commit, patch, seeds, dataset_path, docker_image
with open(CONF) as f:
    cfg = json.load(f)

def run_in_container(cmd, mounts):
    base = ["docker", "run", "--rm"]
    for src, dst in mounts.items():
        base += ["-v", f"{src}:{dst}"]
    base += [cfg["docker_image"]] + cmd
    return subprocess.run(base, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# prepare workspace
subprocess.run(["git", "checkout", cfg["commit"]])
if cfg.get("patch"):
    subprocess.run(["git", "apply", cfg["patch"]])

# attempts
for attempt in range(cfg.get("attempts",3)):
    env = {"PYTHONHASHSEED": str(cfg["seed"]),
           "TORCH_DETERMINISTIC": "1"}
    mounts = {os.path.abspath(cfg["workspace"]): "/workspace",
              os.path.abspath(cfg["dataset"]): "/data"}
    cmd = ["python", "/workspace/train.py", "--config", "/workspace/config.yaml", "--data", "/data/min_slice"]
    r = run_in_container(cmd, mounts)
    open(f"run_{attempt}.log","wb").write(r.stdout+r.stderr)
    if r.returncode != 0:
        print("Failure reproduced on attempt", attempt)
        sys.exit(0)
print("No failure reproduced")

Validation and CI integration- Trigger this runner from CI when flaky failure detected (sentry/alert hook).- If reproduced, capture final minimal artifacts and open an automated issue with artifacts attached and steps to reproduce.- If unreproducible after N attempts, mark flaky and schedule longer isolation runs with richer telemetry (profiling, hardware counters).

Trade-offs and best practices- Aim for smallest deterministic artifact set to speed debugging; but preserve provenance (commit, seed).- Use containers to remove host differences; if hardware-dependent, collect hardware telemetry.- Automate TTL for stored artifacts and use artifact storage (S3) with content-addressed keys.

This process yields deterministic, small, shareable reproductions that speed root-cause analysis and reduce noise from environmental nondeterminism.

EasyTechnical

47 practiced

You have a simple PyTorch model's forward method below. Write minimal tests (using pytest) asserting output shape, dtype, that values are finite (no NaN/Inf), and that a forward pass with `torch.no_grad()` does not require gradients.

python

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.lin = nn.Linear(in_dim, out_dim)
    def forward(self, x):
        return self.lin(x)

Specify the test inputs and assertions.

Sample Answer

python

import torch
import pytest
from simple_model import SimpleModel  # or define SimpleModel inline

# Test parameters
IN_DIM = 8
OUT_DIM = 3
BATCH = 5
DTYPE = torch.float32

def make_model_and_input():
    torch.manual_seed(0)
    model = SimpleModel(IN_DIM, OUT_DIM)
    # typical float input, does not require grad by default
    x = torch.randn(BATCH, IN_DIM, dtype=DTYPE)
    return model, x

def test_forward_shape_dtype_and_finite():
    model, x = make_model_and_input()
    out = model(x)

    # shape
    assert out.shape == (BATCH, OUT_DIM), f"expected shape {(BATCH, OUT_DIM)}, got {out.shape}"

    # dtype
    assert out.dtype == DTYPE, f"expected dtype {DTYPE}, got {out.dtype}"

    # values finite (no NaN / Inf)
    assert torch.isfinite(out).all().item(), "output contains NaN or Inf"

def test_forward_no_grad_does_not_require_grad():
    model, x = make_model_and_input()

    # sanity check: with grad enabled, outputs usually require grad because model params require grad
    out_with_grad = model(x)
    assert out_with_grad.requires_grad, "expected output to require grad when not in no_grad()"

    # now run under no_grad and assert no gradients are required
    with torch.no_grad():
        out = model(x)
    assert not out.requires_grad, "output should not require grad inside torch.no_grad()"

MediumTechnical

45 practiced

Multiple assertions in a single test can hide the true cause of failure. Describe techniques and patterns (e.g., pytest subtests/parametrization, small single-concern tests, assertion aggregation) you would use to ensure failure isolation and maintainable tests. Provide a short code example demonstrating one pattern.

Sample Answer

Multiple assertions in one test make it hard to know which condition failed and why. For AI systems—where inputs, preprocessing, model inference, and metrics can all fail—isolating failures improves debugging speed and test reliability. Techniques and patterns:

- Small single-concern tests: each test asserts one responsibility (e.g., preprocessing, model forward pass, metric calculation).- Parametrization: use pytest.mark.parametrize to run the same assertion across many cases; each case shows separately in test reports.- Subtests (pytest-subtests): group related checks but isolate failures per case.- Assertion aggregation: collect multiple assertion results and report them together (use pytest-check, or custom aggregator) when you need to verify many independent properties but still want to see all failures.- Descriptive assertions & helper functions: wrap complex comparisons in helpers that raise clear messages (e.g., compare_embeddings() with cosine threshold).- Fixtures and deterministic inputs: reduce flaky tests by fixing RNG seeds and using small synthetic data.

Example — pytest parametrization for model inference cases:

python

import numpy as np
import pytest

@pytest.mark.parametrize("input_vec,expected_label", [
    (np.array([0.0, 1.0]), "positive"),
    (np.array([1.0, 0.0]), "negative"),
    (np.array([0.5, 0.5]), "neutral"),
])
def test_model_predict_simple(model, input_vec, expected_label):
    """Each param is a separate test case—failure isolates to that case."""
    pred = model.predict(input_vec)  # deterministic small model fixture
    assert pred == expected_label, f"input={input_vec.tolist()} produced={pred}"

When many property checks are required for a single input (e.g., shape, dtype, value ranges), either split into multiple tiny tests or use an aggregator that records all failed properties so you can see them together without hiding root causes.

HardTechnical

46 practiced

A third-party tokenization library update caused a subtle regression in downstream model outputs. Describe how you would write tests to protect against such indirect dependency regressions in the future. Include ideas like pinned contracts, fuzz/regression suites, and integration tests that cover end-to-end tokenization behavior.

Sample Answer

I would build a layered test strategy that catches indirect regressions from dependency updates early, and couples automated checks with runtime monitoring.

Approach summary- Define a strict tokenization contract (inputs → token ids, offsets, special tokens) and pin a small set of canonical examples (golden files).- Add unit/contract tests that assert exact behavior for those examples.- Add fuzz/regression suites that cover edge cases, randomness, and distributional samples; record expected properties or outputs (not necessarily exact ids).- Add end-to-end integration tests that run model inference using tokenized inputs and validate downstream model outputs or metrics within tolerances.- Enforce CI gating, canary updates, and production monitoring/alerts.

Concrete tests1) Contract/unit tests (exact matching)- Pick representative sentences, multilingual, edge cases (emoji, control chars, long sequences).- Assert token ids, token strings, offsets, and presence of special tokens.

python

def test_tokenizer_contract(tokenizer):
    # canonical example pinned to repo
    text = "Hello, world! 👋\n"
    expected_tokens = ["Hello", ",", " world", "!", " 👋", "\n"]
    expected_ids = [15496, 11, 995, 0, 50256, 13]  # example pinned IDs
    toks = tokenizer.tokenize(text)
    assert toks == expected_tokens
    assert tokenizer.encode(text) == expected_ids

2) Regression/fuzz suite (property-based + golden model outputs)- Use Hypothesis or randomized corpus to generate inputs; assert invariants: - decode(encode(x)) reconstructs normalized form - offsets point into original text - token count monotonic with appended chars- Run a nightly regression that records token distributions and diffs versus baseline; fail if unexpected changes exceed thresholds.

3) End-to-end integration tests (model-level)- Feed canonical inputs through tokenization → model → output. Assert: - Top-k logits for key tokens remain within relative tolerance - Generated text for a seeded sampling matches golden outputs or pass metrics (BLEU/ROUGE) thresholds

python

def test_e2e_tokenization_effect(model, tokenizer):
    text = "Translate: Hello"
    seed = 42
    tokens = tokenizer.encode(text)
    logits = model.forward(tokens)
    # compare key-token logits against pinned baseline with small epsilon
    assert abs(logits[0, tokenizer.vocab["Hello"]] - baseline_logits) < 1e-3

Operational policies- Pin minimum supported version range but run weekly tests against newer versions in CI (matrix) to detect regressions before upgrading.- Use library-of-record: store golden files (tokens, ids, logits) in version control with clear provenance.- Canary deployment: upgrade in a staging environment, run full regression suite and shadow traffic checks.- Add runtime safeguards: when tokenizer library changes, run lightweight startup check validating token contract and abort if violated.

Monitoring & alerting- Log tokenization stats (vocab hits, unknown tokens, average token length) and drift metrics; feed into dashboards and set alerts on sudden shifts.- Post-deploy A/B tests comparing model outputs to baseline metrics; auto-roll-back on regression.

Trade-offs and justification- Exact pinned outputs catch subtle breaking changes but increase maintenance when intentional tokenizer improvements occur—mitigate by review process for updating golden files.- Property-based tests and toleranced model checks balance brittleness and sensitivity.- CI + canary + monitoring provides fast detection and human review before user impact.

This layered approach gives early, automated detection of indirect regressions while allowing controlled, auditable upgrades.

Unlock Full Question Bank

Get access to hundreds of Software Testing and Assertions interview questions and detailed answers.

Join thousands of developers preparing for their dream job.