Computer Vision Fundamentals Questions

Core concepts and methods in computer vision with an emphasis on both traditional image processing and modern deep learning approaches. Candidates should understand how images are represented as matrices or tensors, common preprocessing steps and augmentation techniques to improve generalization, and fundamentals of convolutional neural networks including convolution operations, receptive fields, pooling, and normalization. Familiarity with common vision tasks such as image classification, object detection, semantic and instance segmentation, and key model design patterns is expected. Candidates should know common vision architectures and families such as residual networks and Visual Geometry Group style networks, the role of pretrained models and transfer learning, how to fine tune models for new tasks, and practical tooling including image processing libraries and deep learning frameworks for training and inference. Evaluation may include trade offs between accuracy, latency, and resource usage for deployment.

HardSystem Design

47 practiced

Design a distributed training approach to train a large vision transformer on a billion-image dataset using 256 GPUs. Discuss data parallelism, model parallelism (tensor/pipeline), optimizer state sharding, mixed precision, checkpointing strategy, and how to handle stragglers and node failures for robustness.

Sample Answer

Requirements:- Train a large Vision Transformer (VIT) on ~1B images using 256 GPUs.- Target throughput, reasonable training time, memory limits, fault tolerance, and correctness.- Support mixed precision and scalable optimizer state.

High-level architecture:- 256 GPUs grouped into 32 nodes × 8 GPUs (or 16 nodes ×16 GPUs). Use NCCL over RDMA for all-reduce and point-to-point.- Hybrid parallelism: pipeline + tensor model-parallelism inside model, data-parallel across replicas. - Split the big model into P pipeline stages (e.g., 8–16 stages) mapped to GPU groups. - Within each stage, apply tensor model parallelism (Megatron-style) to split large linear layers across g GPUs per stage. - Create R data-parallel replicas (R = total_gpus / (P * g)). Example: with 256 GPUs, P=8, g=2 → R=16 replicas.

Data pipeline and sharding:- Sharded dataset across storage nodes; each worker reads local shards to avoid bottlenecks.- Use an epoch-based deterministic shuffling with distributed samplers; prefetch and decode on CPU, augmentation on host or GPU where appropriate.- Maintain per-replica mini-batch size to fit GPU memory; global batch = per_replica_batch * R.

Optimizer state sharding:- Use ZeRO-style optimizer sharding (DeepSpeed/FSDP): stage 2 or 3. - Shard optimizer states (momentum, Adam's moment estimates) across data-parallel ranks to reduce memory. - Combine with model-parallel tensor sharding so each GPU only stores its parameter shard + local optimizer shard.- Gradient communication: accumulate local grads, perform intra-stage reduce-scatter for tensor-parallel grads, then all-reduce across data-parallel replicas (or use reduce-scatter + all-gather per ZeRO).

Mixed precision:- Use AMP (FP16 or BF16) for compute with a master FP32 copy for parameters, or use optimizer-state sharding to store master params sharded in FP32.- Use dynamic loss scaling to avoid underflow.- Prefer BF16 if hardware supports it (stability without scaling).

Checkpointing strategy:- Multi-level checkpointing: - Frequent lightweight checkpoints: shard-only metadata + optimizer shards and RNG state written asynchronously (every N steps). - Periodic full checkpoints: reconstruct full model/optimizer state on fewer nodes and write combined checkpoint to reliable object storage. - Use incremental checkpoints: only changed shards or versioned deltas.- Save pipeline and tensor partition mapping in metadata so recovery reconstructs exact mapping.- Ensure checkpoint consistency by pausing training briefly at barrier; use asynchronous background uploads to storage to reduce pause time.

Handling stragglers & node failures:- Stragglers: - Monitor per-GPU throughput and OS-level telemetry; detect slow nodes. - Use elastic training: allow rebalancing of data shards; pause pipeline briefly to migrate pipeline stage to spare GPU if available. - Use micro-batching and gradient accumulation to tolerate transient slowdowns.- Node failures: - Elastic/resumable training using sharded checkpoints + leader election: - On failure, remaining GPUs either (a) shrink world size and continue (re-shard optimizer and model via ZeRO elastic), or (b) replace failed node with spare and restore shards from last checkpoint. - Use fault-tolerant all-reduce (e.g., NCCL + timeout + reinitialize) or switch to ring-reduce with retry. - Automate restart: orchestrator (Kubernetes/Slurm) reassigns, restores state from last consistent checkpoint and resumes at last step. - Keep multiple recent checkpoints to reduce lost work.

Scalability, trade-offs and rationale:- Hybrid parallelism balances memory (model-parallel) and throughput (data-parallel). Pipeline parallelism improves utilization for very deep transformers; tensor-parallelism handles large linear layers.- ZeRO stage 2/3 reduces memory footprint at cost of more communication — acceptable on 256 GPUs with high-bandwidth interconnect.- Mixed precision gives 2-3x speed/memory benefits; BF16 preferred for stability.- Checkpoint frequency trades runtime overhead vs. rework on failure; prefer shard-level frequent checkpoints + occasional full checkpoint.- Elastic training complexity is higher but required at this scale to avoid losing days of compute.

Key metrics to monitor:- Samples/sec (per replica and global), GPU utilization, communication vs compute ratio, memory headroom, checkpoint write latency, time-to-recovery on simulated failures.

This design provides high throughput, manageable memory use, and robustness via sharded state and elastic recovery, making training on 1B images with 256 GPUs feasible.

MediumTechnical

63 practiced

You are tasked with establishing a high-quality image annotation workflow for bounding boxes and polygons. Describe labeling instructions, quality control procedures (review, consensus, gold tasks), annotation formats, and tooling integrations to ensure consistent labels at scale.

Sample Answer

Approach/framework:- Define scope & success metrics (precision/recall targets, IoU thresholds, throughput, labeler throughput per hour).- Create a canonical labeling guide, implement tooling with QA gates, and continuous monitoring with metrics and retraining loops.

Labeling instructions (must be unambiguous):- Task scope: object classes (with taxonomy), when to mark “occluded”, “truncated”, “ignore”, and hierarchical labels.- Bounding boxes: tight axis-aligned boxes with 1–2 px padding; include minimum size threshold; mark occlusion and visibility percent.- Polygons: vertices clockwise, minimal vertices to follow contour (<=5 px vertex tolerance), snapping rules for holes and overlapping objects.- Examples: 10 positive, 10 negative, corner cases (partial objects, reflections, mannequins) with annotated images and rubric checklist.- Do/Don't gallery and short checklist at top of UI.

Quality control procedures:- Peer review: every annotation reviewed by a second labeler; disagreements flagged.- Consensus: if reviewer disagrees, send to arbiter; consensus reached when 2/3 agree or arbiter decision.- Gold tasks: insert 5–15% hidden gold images per labeler with known ground truth to measure ongoing accuracy; fail rate triggers retraining or lockout.- Statistical sampling audit: daily random sample of batches reviewed by senior annotator (5–10%).- Metrics: per-labeler precision/recall, IoU distribution, time per task, disagreement rate; set SLAs.- Drift detection: monitor model-in-the-loop performance vs. labels; high model-label divergence triggers re-evaluation.

Annotation formats and metadata:- Store canonical format: COCO JSON for polygons/segmentation and Pascal VOC/COCO bounding boxes; include metadata fields: labeler_id, reviewer_id, timestamps, version, visibility, occlusion, difficulty flag, source image id.- Provide export adapters (COCO, VOC, TFRecord) and schema validation.

Tooling integrations:- Use labeling platform (Label Studio/Scale/Cloud AutoML) with: - Custom UI widgets enforcing vertex ordering, duplicate detection, min box size validation. - Real-time validation rules (IoU overlap thresholds, class consistency). - Model-assisted pre-annotations (active learning): model suggestions shown; labeler accepts/edits — record accept/edit rates. - Automated workflows: gold insertion, reviewer routing, consensus arbitration queues. - CI pipeline: pre-commit validators for exported labels + automated unit tests on sample images. - Monitoring dashboards (Grafana) for labeler performance, gold-test pass rates, and drift alerts. - Versioning: dataset version control (DVC or internal), snapshot exports for training reproducibility.

Operationalizing at scale:- Onboarding program with quiz-based certification on gold tasks.- Retraining cadence: weekly analysis of errors → update guide and gold set.- Feedback loop to model teams: high-disagreement classes prioritized for model fine-tuning.This combination ensures consistent, auditable, and scalable high-quality bounding box and polygon annotations.

MediumTechnical

58 practiced

Explain self-supervised pretraining methods (e.g., SimCLR, MoCo, DINO) for computer vision. How do contrastive and clustering-based approaches learn useful representations, and when might self-supervised pretraining outperform supervised ImageNet pretraining for downstream tasks?

Sample Answer

Self-supervised pretraining creates useful visual representations without labels by defining proxy tasks. Two major families are contrastive and clustering/distillation methods.

Contrastive methods (SimCLR, MoCo):- Core idea: pull together representations of two augmented views of the same image (positives) and push apart views of other images (negatives).- SimCLR: uses strong augmentations, a large batch size, and a projection head; optimizes NT-Xent contrastive loss (temperature-scaled cosine similarity).- MoCo: addresses the need for many negatives by maintaining a dynamic memory bank / queue and a momentum-updated encoder for consistent keys, enabling smaller batches.- Why it learns useful features: forcing invariance to augmentations while preserving discriminative information yields features that cluster semantic content.

Clustering / distillation methods (DINO, SwAV):- SwAV: performs online clustering of image views and enforces consistency between cluster assignments of different views (swapped prediction), avoiding explicit negative pairs.- DINO: uses a teacher–student setup (momentum teacher) and matches output distributions without labels; emergent properties include attention to objects and localization.- Why it works: these objectives encourage grouping semantically similar images and learning stable prototypes or targets, often producing features good for transfer.

When it can beat supervised ImageNet pretraining:- Domain shift: target data differs from ImageNet (medical, satellite, industrial), so label biases in supervised models hurt transfer.- Fine-grained or dense tasks: self-supervised features often retain more low/mid-level information useful for detection, segmentation, or fine-grained classification.- Limited labeled data: SSL pretraining plus few-shot finetuning often outperforms supervised pretraining.- Robustness and privacy: SSL can be trained on in-domain unlabeled data, handling label noise and proprietary datasets.

Practical notes:- Choose contrastive if negatives are easy to sample and you can scale; use MoCo variants for memory efficiency.- Use clustering/distillation (DINO/SwAV) when negatives are noisy or for better emergent localization.- Always validate with a linear probe and task-specific finetuning; augmentation strategy and projection head matter a lot.

MediumTechnical

58 practiced

Compare classic image pyramid approaches with Feature Pyramid Networks (FPN) for handling multi-scale object detection. Explain computational and memory trade-offs, inference-time costs, and situations where an image pyramid is still advantageous.

MediumTechnical

48 practiced

In a face-detection system where false negatives are critical, propose strategies to reduce false negatives at training and inference time without causing unacceptable spike in false positives. Discuss threshold tuning, loss weighting, cascaded detectors, and post-processing techniques.

Sample Answer

Start by clarifying objective and constraints: minimize false negatives (missed faces) because they’re critical, while keeping false positives (FP) within an operationally acceptable range (specify target FP rate or cost per image).

1) Training-time strategies- Loss weighting: increase weight for positive examples (faces), or use focal loss with low gamma to emphasize hard positives. Example: cross-entropy * w_pos where w_pos tuned so recall improves without exploding FP.- Hard example mining: mine false-negative-like hard positives (occluded, extreme poses) and oversample them. Use Online Hard Example Mining (OHEM) to bias training to tough positives.- Data augmentation: synthetically create small faces, occlusions, low-light, blur to teach model to detect edge cases.- Multi-task supervision: add auxiliary heads (landmarks, segmentation) so feature maps are richer for subtle faces.- Class imbalance handling: use balanced sampler or reduced negative-to-positive ratio in minibatches so positives influence gradients more.

2) Inference-time strategies- Threshold tuning with validation curves: sweep detector score threshold and plot Recall vs FP per image (FPI). Choose operating point that meets required FN cost. Use per-size or per-anchor thresholds: lower threshold for small-face bins.- Cascaded detectors: first-stage high-recall, low-precision lightweight detector (low threshold) produces proposals; second-stage stronger classifier filters FPs. This improves recall while controlling FP via a stronger verifier.- Multi-scale and multi-model ensemble: run detector at multiple scales or combine models specialized for small/occluded faces, then fuse proposals (non-max suppression tuned conservatively).- Post-processing: - Adaptive NMS: use softer-NMS or score-aware NMS to avoid suppressing true overlapping faces; set IoU thresholds per scale. - Contextual filtering: use temporal consistency (video) or face-tracking to recover missed detections across frames. - Calibration and score fusion: calibrate scores (Platt/ isotonic) and combine with auxiliary signals (landmark confidence) before final thresholding.- Monitoring and feedback loop: collect real-world FN examples, re-train periodically.

3) Trade-offs and practical tips- Tuning: use ROC/PR-style plots and cost-weighted metrics (e.g., weighted F1 where FN penalty high) rather than accuracy.- Avoid runaway FP: progressively test thresholds on held-out operational data; use cascaded verifier to catch FP introduced by aggressive thresholds.- Efficiency: make first stage cheap (mobile backbone); heavier verifier only runs on proposals.- Safety nets: for critical cases, allow human-in-the-loop review for low-confidence negatives.

Expected result: combining loss re-weighting + hard mining and augmented positives raises recall; cascaded detector + calibrated thresholding + soft-NMS controls FP. Iterate with metrics (Recall @ fixed FPI) to reach acceptable trade-off.

Unlock Full Question Bank

Get access to hundreds of Computer Vision Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.