Debugging and Troubleshooting AI Systems Questions

Covers systematic approaches to find and fix failures in machine learning and artificial intelligence systems. Topics include common failure modes such as poor data quality, incorrect preprocessing, label errors, data leakage, training instability, vanishing or exploding gradients, numerical precision issues, overfitting and underfitting, optimizer and hyperparameter problems, model capacity mismatch, implementation bugs, hardware and memory failures, and production environment issues. Skills and techniques include data validation and exploratory data analysis, unit tests and reproducible experiments, sanity checks and simplified models, gradient checks and plotting training dynamics, visualizing predictions and errors, ablation studies and feature importance analysis, logging and instrumentation, profiling for latency and memory, isolating components with canary or shadow deployments, rollback and mitigation strategies, monitoring for concept drift, and applying root cause analysis until the underlying cause is found. Interviewers assess the candidate on their debugging process, ability to isolate issues, use of tools and metrics for diagnosis, trade offs in fixes, and how they prevent similar failures in future iterations.

MediumTechnical

33 practiced

Your model degraded immediately after a schema change. Design an ablation study plan to determine which feature(s) or transformation(s) caused the regression. Include experimental design, statistical tests to detect significant change, and how to prioritize features for rollback or modification.

Sample Answer

Situation: After a schema change, model performance (e.g., validation AUC and production conversion rate) dropped immediately. We need an ablation study to identify which feature(s) or transformation(s) introduced the regression.

Plan (high level):1. Recreate baseline and candidate variants - Snapshot the pre-change model + data pipeline (baseline). - Build a reproducible pipeline that can turn each feature/transformation ON/OFF independently.

2. Define metrics and dataset - Primary metric: the one that degraded (e.g., AUC or business KPI). - Secondary: calibration, precision/recall, latency. - Use the same evaluation set as production traffic just before the change; if unavailable, use a stratified holdout representing production distribution.

3. Experimental design - Factorial ablation: test each changed feature/transformation individually by reverting it to prior behavior while keeping others at new state (one-at-a-time ablation). If number of changes is small (<10), also run pairwise combos to detect interactions. - For N changes where N large, use a staged approach: binary search grouping (split changes into halves) to quickly localize problematic group, then drill down. - For each variant, run deterministic scoring on the fixed evaluation set and, if feasible, a small randomized shadow test in production traffic.

4. Statistical testing - Use paired tests since outputs are computed on same examples: - For scalar metrics (AUC): use DeLong test for AUC or bootstrap paired difference to compute 95% CI. - For rates (conversion): use McNemar’s test for paired binary outcomes or bootstrap for difference in proportions. - For continuous losses: paired t-test or Wilcoxon signed-rank if non-normal. - Correct for multiple comparisons (Benjamini-Hochberg FDR or Bonferroni if conservative). - Require practical significance threshold (e.g., delta > 0.5% absolute or business-defined) plus statistical significance.

5. Prioritization for rollback or fix - Rank features by impact size × confidence: (mean metric delta) × (1 / p-value) or prefer larger effect with tight CI. - High priority: single feature whose revert restores most of the drop with strong stat. significance — roll back transformation or revert schema change for that feature immediately (can be gated via feature flag). - Medium: features with moderate impact or interaction effects — create patch (fix transformation or re-train with new schema) and A/B test in shadow. - Low: negligible or uncertain effects — monitor and collect more data.

Example workflow- Suppose 6 changed fields. Start by scoring 6 one-at-a-time variants. Feature B when reverted recovers 80% of lost AUC (bootstrap delta = +0.012, 95% CI [+0.008,+0.016], p<0.001 after FDR). Action: toggle feature flag to roll B back in prod, start root-cause (e.g., wrong parsing of nulls), retrain model using corrected transformation, and schedule canary release.

Safeguards and follow-up- Use shadow inference or canary rollout for any rollback.- Log examples where predictions changed most for error analysis.- Add unit/integration tests for schema-to-transformation mapping and monitor drift alerts for future schema changes.

This approach balances speed (one-at-a-time, binary grouping) with statistical rigor (paired tests, bootstrap, multiple-test correction) so you can rapidly identify, prioritize, and remediate the offending feature or transformation.

EasyTechnical

38 practiced

Implement in Python a function compute_class_metrics(y_true, y_pred) that returns precision, recall, and F1 score for binary classification. Assume y_true and y_pred are lists or 1D numpy arrays of 0/1. Do not use sklearn; your solution should run in O(n). Example: y_true=[0,1,1,0,1], y_pred=[0,1,0,0,1] => precision=..., recall=..., f1=...

Sample Answer

Approach: iterate once through y_true and y_pred, count true positives (tp), predicted positives (pp), and actual positives (ap). Compute precision = tp/pp, recall = tp/ap, and F1 = 2*precision*recall/(precision+recall). Handle division-by-zero when pp or ap are zero.

python

def compute_class_metrics(y_true, y_pred):
    """
    Compute precision, recall, and F1 for binary labels (0/1).
    Accepts lists or 1D numpy arrays. Runs in O(n).
    Returns (precision, recall, f1) as floats.
    """
    # Convert to simple iterables (if numpy arrays)
    try:
        iterator_true = iter(y_true)
        iterator_pred = iter(y_pred)
    except TypeError:
        # Fallback: assume they are numpy arrays supporting .tolist()
        y_true = list(y_true)
        y_pred = list(y_pred)
        iterator_true = iter(y_true)
        iterator_pred = iter(y_pred)

    tp = 0        # true positives
    pp = 0        # predicted positives
    ap = 0        # actual positives

    for t, p in zip(y_true, y_pred):
        if p == 1:
            pp += 1
            if t == 1:
                tp += 1
        if t == 1:
            ap += 1

    precision = tp / pp if pp > 0 else 0.0
    recall = tp / ap if ap > 0 else 0.0
    if precision + recall == 0:
        f1 = 0.0
    else:
        f1 = 2 * precision * recall / (precision + recall)

    return precision, recall, f1

Key points:- Single pass O(n) time, O(1) extra space.- Handles empty inputs and cases with zero predicted/actual positives without raising exceptions.Example: y_true=[0,1,1,0,1], y_pred=[0,1,0,0,1] => tp=2, pp=2, ap=3 => precision=1.0, recall=0.6666667, f1=0.8.

MediumTechnical

34 practiced

Explain influence functions and how they can be applied to trace which training examples most affected a particular prediction. Discuss computational challenges, approximations (Hessian-vector products), and practical limitations when applying influence functions to deep neural networks.

HardTechnical

40 practiced

Design and describe an algorithm (pseudo-code acceptable) that identifies likely label flips by training K diverse models and selecting examples with high disagreement and high confidence. Explain the scoring metric, hyperparameters to tune, compute cost, and how you would present candidates to human labelers for review.

Sample Answer

Approach summary:Train K diverse classifiers on the same labeled dataset (diversity via architectures, random seeds, training folds, augmentations). For each training example, compute each model’s predicted class and confidence (softmax probability). Score examples by combining inter-model disagreement and average max-confidence to prioritize likely label flips: high disagreement + high confidence.

Pseudo-code (Python-like):

python

def detect_label_flips(models, dataset, beta=1.0, gamma=0.5):
    # models: list of K trained models
    # dataset: list of (x, y)
    # beta: weight for confidence in score
    # gamma: exponent to amplify disagreement
    scores = []
    for i, (x, y_true) in enumerate(dataset):
        preds = []        # predicted class
        confs = []        # predicted max-prob
        for m in models:
            p = m.predict_proba(x)        # vector over classes
            c = argmax(p)
            preds.append(c)
            confs.append(max(p))
        # disagreement: normalized entropy over model votes
        vote_counts = Counter(preds)
        vote_probs = [v/len(models) for v in vote_counts.values()]
        disagreement = -sum(p*log(p) for p in vote_probs) / log(len(vote_probs))
        avg_conf = mean(confs)
        # Score: prefer high disagreement and high average confidence
        score = (disagreement ** gamma) * (avg_conf ** beta)
        scores.append((i, y_true, score, preds, confs))
    return sorted(scores, key=lambda t: t[2], reverse=True)

Scoring metric rationale:- Disagreement: normalized vote-entropy captures how split models are about label.- avg_conf: high confidence means models are confident in (possibly incorrect) prediction; combining favors cases where models confidently agree on a label different from the annotated one or where models strongly disagree while confident.- Exponents/weights (gamma, beta) let you emphasize disagreement vs confidence.

Hyperparameters to tune:- K (number of models): trade-off between robustness and compute (typical 5–20).- Diversity mechanisms: architecture pool, data folds, augmentation strength.- beta, gamma: relative importance of confidence vs disagreement (grid search on validation noise-injection).- Thresholds: top-N or score cutoff for human review.- Per-model temperature scaling or calibration to make confidences comparable.

Compute cost:- Training: K × cost of single model training. Can reduce via lightweight or snapshot ensembles, distillation, or training smaller diverse models.- Inference: K forward passes per example; can batch and parallelize. For dataset of N examples, cost ~ O(KN) inference.- Memory: store K models; inference memory scales with largest model.

Presentation to human labelers:- Show original example, current label, top voted label(s) by models, per-model confidences, vote distribution, score, and brief explanation (e.g., "4/5 models predict 'cat' avg_conf=0.92; original='dog'"). Prioritize by score and group by predicted-label vs original-label (confident consensus flips vs high-entropy disagreements). Provide UI actions: accept model-consensus flip, mark ambiguous (needs more context), add comment, or escalate. Include small batch context: similar examples and model explanations (saliency or nearest neighbors) to speed decisions and ensure consistency.

EasyTechnical

39 practiced

Describe common signs that a labeled dataset contains label errors (noisy or incorrect labels). Include quantitative checks you would run (loss distribution, per-example confidence, disagreement across models), and explain how you would prioritize relabeling or filtering when manual relabeling is expensive.

Sample Answer

Common signs of label errors- High loss / low confidence on examples that should be easy: samples with large training loss or consistently low predicted probability for the labelled class often indicate wrong labels.- Loss or confidence distribution heavy tails: a small subset with much higher loss or lower confidence than the bulk.- Persistent disagreement across models/checkpoints: examples where ensembles, different architectures, or checkpoints predict different labels or consistently predict a class different from the assigned label.- Label-feature mismatch: examples whose features are out-of-distribution for their label (use t-SNE / UMAP or class-conditional feature density).- High annotator disagreement or metadata signals: borderline cases, rapid annotator time, or known low-quality sources.

Quantitative checks to run- Per-example training loss and average validation loss contribution — rank examples by mean loss across epochs.- Per-example confidence: mean and variance of softmax probability for the labeled class across epochs / augmentations.- Ensemble disagreement: fraction of models that predict a different class; compute entropy of predicted class distribution.- Influence/Forgetfulness: examples that are frequently “forgotten” (correct -> incorrect during training) often indicate noise.- Cross-validation flips: label flips across K-folds indicate instability.

Prioritization when relabeling is expensive1. Tier the dataset: top 1–5% highest-loss / highest-disagreement as “high priority” for manual review.2. Use active relabeling: present high-uncertainty, high-impact examples (those influencing model predictions or business metrics) first.3. Aggregate signals: only relabel examples flagged by multiple checks (loss + ensemble disagreement + feature outlier) to reduce wasted effort.4. Automated filtering/soft-labeling: for medium-confidence suspect examples, use ensemble majority vote or soft labels instead of immediate human relabeling.5. Bootstrapped relabeling: relabel small batches, retrain, and re-evaluate to see if noise signal persists — stop when marginal benefit falls below cost.6. Track and measure: maintain relabeling ROI metrics (model performance delta per relabel hour) and prioritize classes/segments with highest ROI.

These steps let you focus scarce human effort where label corrections yield the biggest model improvements.

Unlock Full Question Bank

Get access to hundreds of Debugging and Troubleshooting AI Systems interview questions and detailed answers.

Join thousands of developers preparing for their dream job.