Image Model Pipelines Questions

End to end design and implementation of computer vision pipelines for image classification, object detection, and segmentation. Topics include dataset collection and annotation formats, data splitting and augmentation strategies, label schemas for classification and bounding boxes and masks for detection and segmentation, and training workflows including transfer learning and backbone selection. Candidates should understand model families and examples such as You Only Look Once, Faster Region Convolutional Neural Network, and Mask Region Convolutional Neural Network, relevant loss functions, anchor and bounding box handling, mask prediction for instance segmentation, and post processing techniques such as non maximum suppression. Evaluation and validation practices should be covered, including accuracy for classification, precision and recall, mean average precision for detection, and intersection over union for segmentation, plus cross validation and handling class imbalance. Operational considerations include inference latency and throughput, model quantization and optimization, deployment pipelines, monitoring model performance in production, continual learning and data drift handling, and tradeoffs between accuracy, speed, and resource usage.

MediumTechnical

60 practiced

Implement a Python function to compute IoU between two axis-aligned bounding boxes. Boxes are tuples (x_min, y_min, x_max, y_max). Your function should return a float IoU in [0,1]. Provide code and a brief complexity analysis.

Sample Answer

To compute Intersection over Union (IoU) for two axis-aligned boxes (x_min, y_min, x_max, y_max), compute the coordinates of the intersection rectangle, its area, each box area, then IoU = intersection_area / union_area. Handle non-overlapping and degenerate boxes (zero or negative area).

python

def iou(boxA, boxB):
    """
    Compute IoU between two axis-aligned bounding boxes.
    Each box is (x_min, y_min, x_max, y_max).
    Returns float in [0,1].
    """
    ax1, ay1, ax2, ay2 = boxA
    bx1, by1, bx2, by2 = boxB

    # Intersection coordinates
    ix1 = max(ax1, bx1)
    iy1 = max(ay1, by1)
    ix2 = min(ax2, bx2)
    iy2 = min(ay2, by2)

    # Intersection dimensions (if negative => no overlap)
    iw = max(0.0, ix2 - ix1)
    ih = max(0.0, iy2 - iy1)
    inter_area = iw * ih

    # Areas of boxes (if invalid boxes given, area will be 0)
    areaA = max(0.0, ax2 - ax1) * max(0.0, ay2 - ay1)
    areaB = max(0.0, bx2 - bx1) * max(0.0, by2 - by1)

    union = areaA + areaB - inter_area
    if union == 0.0:
        return 0.0  # both boxes degenerate
    return inter_area / union

Key points:- Uses clamp to zero for non-overlap and degenerate boxes.- Returns 0.0 when union is zero to avoid division by zero.

Time complexity: O(1) — constant work and arithmetic. Space complexity: O(1) — constant extra memory.

EasyTechnical

59 practiced

Compare the COCO, Pascal VOC, and YOLO annotation formats for object detection and instance segmentation. For each format, describe: file structure, how bounding boxes and masks/polygons are represented, support for multiple instances and categories, and common pitfalls when converting between formats. Example (informal):

- Pascal VOC XML: <annotation> ... <object><name>cat</name><bndbox><xmin>10</xmin>...</bndbox></object>- YOLO txt line: class_id x_center_norm y_center_norm width_norm height_norm- COCO (json-like): annotations: [{bbox:[x,y,w,h], segmentation:[[x1,y1,...]], category_id:k}]

Which format would you choose for a large-scale instance segmentation project and why?

Sample Answer

Pascal VOC- File structure: one XML per image (annotation root with size, filename, and repeated <object> entries).- Boxes/masks: bounding boxes as <bndbox> with integer xmin,ymin,xmax,ymax. No native mask/polygon support (some projects add separate mask files).- Multi-instance/categories: supports multiple <object> tags and a <name> per object.- Pitfalls: inconsistent coordinate origin/1‑ vs 0‑based conventions; mixed image sizes; no masks—people improvise with separate PNG masks which breaks tooling.

YOLO (Darknet .txt)- File structure: one plain .txt per image, each line per object.- Boxes/masks: bbox only, format: class_id x_center_norm y_center_norm width_norm height_norm (normalized to [0,1]). No mask/polygon support.- Multi-instance/categories: supports many lines (instances); category by integer ID—requires consistent class mapping file.- Pitfalls: normalization mistakes when resizing images; rounding/precision loss; class ID mismatches; can't represent segmentation.

COCO- File structure: single JSON (or split train/val) with images[], annotations[], categories[], plus optional licenses/infos.- Boxes/masks: bbox [x,y,width,height] (absolute, top-left origin). segmentation supports polygons (list of coordinate lists) and RLE for binary masks.- Multi-instance/categories: first-class support for multiple instances, crowd annotations (iscrowd), and hierarchical categories.- Pitfalls: differing coordinate conventions when converting (e.g., Pascal uses xmax inclusive), polygon winding/format, RLE encoding, category id/name mismatches, and large JSON memory footprint.

Which to choose?For large-scale instance segmentation pick COCO. It natively supports polygons and RLE masks, instance-level metadata (iscrowd, area), standardized category taxonomy, and broad ecosystem/tooling (Detectron2, COCO API). Its single-file layout simplifies dataset splits and evaluation. Be mindful to standardize coordinate origins, category IDs, and use RLE for dense masks to save space.

MediumTechnical

65 practiced

You observe severe class imbalance in your detection dataset: a few classes have many instances while others are rare. Compare and contrast practical approaches (resampling, class-balanced loss, focal loss, synthetic data augmentation, oversampling hard examples) to address this imbalance. Provide pros/cons and how you'd evaluate success.

Sample Answer

Situation: When facing severe class imbalance in an object-detection dataset, I compare several practical strategies and choose a mix based on the problem, data quality, and deployment constraints.

Resampling (undersampling/oversampling)- Pros: Simple; balances class frequencies directly; undersampling reduces training time.- Cons: Undersampling discards useful data (hurts majority-class performance); naive oversampling (duplication) can overfit rare classes.- When to use: Moderate imbalance, large majority-class dataset, fast prototyping.

Class-balanced loss (effective number weighting)- Pros: Theoretically accounts for class frequency using effective sample counts; easy to plug into existing loss.- Cons: Doesn’t change data distribution; may under-correct when rare-class variance is high.- When to use: Detection where sampling is hard (anchor-based detectors) and you want principled weighting.

Focal loss- Pros: Down-weights easy negatives and focuses on hard/ambiguous examples; widely used in dense detectors.- Cons: Sensitive to hyperparameters (gamma, alpha); may still bias toward frequent hard negatives if imbalance extreme.- When to use: Many background/negative examples and many easy negatives swell loss.

Synthetic data augmentation (rendering, domain randomization)- Pros: Increases diversity of rare classes, can create edge cases and annotations cheaply, reduces overfitting.- Cons: Domain gap to real data; costly to build realistic synthetic pipelines.- When to use: Very few real examples; when you can simulate realistic contexts.

Oversampling hard examples (hard example mining)- Pros: Focus on informative samples (false positives/negatives); improves model robustness.- Cons: Needs careful scheduling to avoid noise amplification; mining for rare positives may be unstable early.- When to use: After warm-up training when model can identify hard cases.

Practical combo and workflow- Start with class-balanced loss or focal loss + controlled oversampling of rare positives (augment rather than duplicate).- Add synthetic augmentation for under-represented poses/environments if feasible.- Use online hard example mining or focal loss to focus learning on mistakes.- Avoid aggressive undersampling of majority unless compute/label quality forces it.

How I’d evaluate success- Use per-class Average Precision (mAP) and mean Average Precision but report per-class AP and harmonic mean to reveal rare-class performance.- Precision-Recall curves, per-class F1, and recall at fixed precision (or vice versa) for operational thresholds.- Confusion analysis (which classes are confused) and calibration checks.- Validation on a holdout real set, plus targeted hard-case test set (including synthetic if used) to detect domain gaps.- Monitor overfitting to duplicated examples (train vs val gap) and stability across seeds.

Key principle: combine loss-level corrections with intelligent data augmentation and targeted mining; validate with class-wise metrics and hard-case tests rather than aggregate scores alone.

HardSystem Design

61 practiced

Design a CI/CD and evaluation pipeline to automatically test new model checkpoints for robustness before deploying to production. Include checks for accuracy on holdout sets, distributional shift tests, fairness checks across demographic subgroups, synthetic adversarial tests, and criteria for promoting a model. How would you automate alerts for regressions?

Sample Answer

Requirements & constraints:- Automatic testing of new checkpoints for accuracy, robustness, fairness, distributional shift, and adversarial resilience.- Gate model promotion (automated + human-in-loop) and support canary rollout, rollback, and alerts.

High-level architecture:- CI (GitHub Actions/Jenkins) → Model build & unit tests → Artifact registry (container + weights) → Evaluation pipeline (Kubeflow/Airflow) → Metrics DB/Feature store + Model Registry (MLflow or Vertex AI) → Monitoring & alerting (Prometheus/Grafana, Evidently/WhyLabs) → Deployment (K8s/Serverless) with canary.

Core pipeline stages:1. Pre-checks (CI): unit tests, linting, small synthetic smoke tests.2. Holdout evaluation: run full validation suite on held-out test sets; compute accuracy, AUC, calibration. Record in metrics DB.3. Distributional-shift tests: compare training vs. current production / holdout using PSI, KS, MMD, and feature drift detectors; flag features with significant shift.4. Fairness checks: evaluate per-demographic metrics (accuracy, precision/recall, FPR/FNR, AUC); compute parity metrics (demographic parity, equalized odds); run statistical significance tests and visualize per-group calibration.5. Synthetic adversarial tests: run a battery—noise/semantic perturbations, adversarial attacks (FGSM, PGD for differentiable models), prompt/edge-case generation for NLP or image corruptions; measure performance drop and worst-group behavior.6. Robustness & safety heuristics: calibration, confidence thresholds, out-of-distribution detection rate.7. Champion/Challenger: compare candidate vs production; require candidate beats prod on primary metric and passes all hard fairness/robustness gates or shows statistically non-inferior behavior.

Automation & orchestration:- Use declarative pipeline (Kubeflow Pipelines/Airflow) with containerized steps; parallelize heavy tests.- Store artifacts and metrics in MLflow + feature store. Version metadata via MLMD.

Promotion criteria (example):- Primary metric (e.g., AUC) improvement >= 1% OR within ±0.5% with better calibration.- No subgroup shows degradation >2% absolute on key metrics, and fairness constraints (e.g., FPR gap < 3%).- PSI < 0.1 for all critical features or documented mitigation.- Adversarial worst-case drop < X% (policy-defined).- If all gates pass: automatic staging deploy; else hold and create review ticket.

Canary & rollout:- Deploy to canary with 5–20% traffic; monitor metrics in real time for 24–72 hours; automated rollback if production metrics regress beyond thresholds or if alert triggers.

Alerting for regressions:- Push metrics to monitoring (Prometheus + Grafana + Evidently). Define deterministic alerts (threshold breaches) and statistical anomaly alerts (Evidently/WhyLabs drift/anomaly detectors).- Integrate with PagerDuty/Slack/Jira. Alert includes diff report, failed test artifacts, and suggested rollback action. For minor regressions, auto-create ticket; for critical regressions, automated rollback and on-call notification.

Human-in-loop & governance:- Require approvals for production-critical degradation exceptions.- Keep audit logs of evaluations, approvals, and rollbacks in model registry.

Trade-offs:- More gates increase safety but slow deployment; use progressive rollout and parallelized testing to balance.- Heavy adversarial testing adds compute cost—prioritize for high-risk models.

This design produces an automated, auditable CI/CD + evaluation pipeline that enforces accuracy, fairness, and robustness before production promotion while enabling fast, safe rollouts and clear alerting for regressions.

HardTechnical

74 practiced

Propose an efficient, batched Multi-Class NMS algorithm suitable for GPU execution when you have thousands of proposals per image and many images per batch. Outline data structures, GPU-friendly memory layout, and strategies to reduce work (top-k prefiltering, per-level pruning). Discuss complexity and memory trade-offs.

Sample Answer

To build an efficient batched Multi-Class NMS for GPU with thousands of proposals per image and many images per batch:

1) Approach summary- Run parallel prefiltering (per-image, per-class) to keep top-K scores per class and optionally across classes; then perform class-wise NMS in parallel on GPU with tile-based IoU suppression and final gather. Use per-FPN-level pruning to reduce candidates early.

2) Data structures & memory layout- Proposals: float32 arrays in SoA for coalesced loads: - boxes: [N_total,4] stored as four contiguous arrays x1[], y1[], x2[], y2[] (or Nimages×Nprop flattened) - scores: [N_total] per-class stored as [N_images, N_classes, N_props] flattened with classes innermost for quick class scan - class_ids: uint16 output- Index arrays: int32 indices per candidate- Bitmask suppression map: 64-bit masks per box tile for fast popcount/and operationsAll buffers preallocated per-batch to avoid allocation overhead.

3) GPU-friendly kernels- Kernel A (Top-K prefilter): per (image,class) block: compute top-K indices using parallel segmented radix/top-k heap in shared memory. Output candidate list and scores.- Kernel B (Per-level pruning): if using FPN, run prefilter per-level keeping M per level; concatenate results.- Kernel C (Parallel block-NMS): work per (image,class) threadblock. Sort candidates by score. Partition candidates into tiles of 64; compute IoU matrix between tile pairs (using shared memory) and produce 64-bit masks indicating suppressions. Use bitset reduction to remove suppressed boxes iteratively (classic Fast NMS bitset trick). Each thread handles one box.- Kernel D (Gather & final top-K per image across classes): merge remaining boxes across classes (can use segmented top-k) to produce final detections.

Include a minimal CUDA-like pseudocode for the NMS loop:

cpp

// pseudocode sketch
__global__ void block_nms_kernel(boxs_x1, x2, y1,y2, scores, indices, out_mask){
  // each block -> one image-class; load candidate indices into shared mem
  // sort descending (radix/bucket)
  // tile loop: for i_tile in tiles:
  //   load tile boxes to shared mem
  //   for j in boxes in tile (parallel):
  //     compute IoU with other boxes, set bitmask
  //   atomic updates to suppression bitsets
  // write survivors to output
}

4) Strategies to reduce work- Top-K prefilter per-class (K=100..1000 depending) reduces N dramatically.- Per-level FPN pruning keeps balanced candidates and leverages locality.- Score thresholding before top-K.- Use class-agnostic NMS for correlated classes to reduce repeated IoU work.- Early exit when few candidates remain.

5) Complexity & memory trade-offs- Without pruning: per-class NMS worst-case O(N^2) IoU checks; with tiling and bitsets, practical cost ~O(N * N_tile) with N_tile << N.- Top-K reduces candidates from P to K: compute cost O(P log K) for top-k step (parallelizable) then O(K^2 / 64) bit-ops for NMS.- Memory: bitmask storage O((K/64)*K) bits per class → modest; shared memory per block requires careful tuning; preallocated global buffers increase memory but avoid allocation overhead.- Latency vs memory: larger K increases recall but quadratically increases compute/memory; choose K and tile sizes tuned to GPU SM count and shared memory.

6) Practical tips- Favor SoA and align to 128B for coalescing.- Use warp-level primitives (shuffle, ballot) and __popcll for masks.- Fuse kernels when possible (prefilter + NMS) to reduce global memory traffic.- Profile per dataset: adjust K, level pruning thresholds, and tile size.

This design yields highly parallel, memory-efficient GPU NMS suitable for thousands of proposals per image and large batches while trading off small additional memory for big compute reductions.

Unlock Full Question Bank

Get access to hundreds of Image Model Pipelines interview questions and detailed answers.

Join thousands of developers preparing for their dream job.