Computer Vision Questions

Start with a risk-driven plan: define threat models (attacker goals: untargeted/targeted, white/grey/black-box), operational constraints (latency, accuracy), and safety requirements (acceptable degradation, SLAs).

Generating adversarial examples for testing

• Use a mix of gradient-based (FGSM, PGD, BIM), optimization-based (CW), and transfer attacks (simulate black-box by training surrogate models).
• Apply physical-world perturbations: print-and-photograph, lighting, occlusion, geometric transforms.
• Use automated pipelines (foolbox, ART) to generate datasets at varying strengths and measure drop in metrics (accuracy, calibration, top-k).

Defenses and hardening

• Adversarial training: include strong projected-gradient examples (PGD) during training with curriculum (start small epsilon → increase). Balance clean accuracy via mixed mini-batches and regularization (weight decay, label smoothing).
• Input preprocessing / sanitization: randomized resizing & padding, JPEG compression, bit-depth reduction, denoising (median, bilateral) — test against adaptive attacks.
• Model-level improvements: use robust architectures (feature denoising blocks, small Lipschitz constants), ensemble models, and model calibration (temperature scaling).
• Runtime detection: anomaly detectors on internal activations (Mahalanobis distance), prediction confidence+entropy thresholds, input reconstruction error (autoencoder/denoising AE). Combine detectors with abstain/reject/fallback policies.
• Certification: where applicable, use randomized smoothing to obtain probabilistic l2 robustness guarantees; report certified radii and trade-offs with clean accuracy.

Monitoring and production strategy

• Continuous monitoring: log inputs, predictions, confidences, activation statistics, and input metadata (timestamp, source, sensor state). Compute streaming metrics: accuracy (if labels available), confidence drift, PSI/KS on features, and model uncertainty trends.
• Drift detection: use multivariate drift detectors (MMD, KL, Wasserstein) and lightweight univariate checks per feature; monitor latent-space distributions too.
• Alerting and response: define thresholds triggering investigation, rollback, or fallback models. Implement safe-fail: route low-confidence or anomalous inputs to human review or conservative models.
• Retraining pipeline: automate dataset curation including flagged anomalous inputs and adversarial examples, validate with holdout test sets and out-of-distribution scenarios.

Evaluation and governance

• Maintain an adversarial test-suite and run it in CI/CD (regression on robustness metrics).
• Document threat model, mitigation decisions, trade-offs, and update SLAs.
• Periodically perform red-team exercises and external audits.

This combined strategy balances empirical hardening (adversarial training, preprocessing), detection and operational controls, and formal guarantees (where possible) together with monitoring and incident workflows to keep the vision model robust in production.

Situation: With only 5,000 labeled images across 10 classes, data scarcity and overfitting are the main concerns. Below I compare the three strategies, give practical recommendations for which layers to freeze/unfreeze, learning rates/schedules, and augmentation choices (PyTorch-ready ideas).

Summary recommendation: start with (c) as a fast baseline, then (b) (fine-tune) as the likely best-performing approach. (a) (train from scratch) is usually last-resort unless your domain is very different from ImageNet.

(a) Train ResNet from scratch

• Pros: full flexibility to learn domain-specific features; no pretrained bias.
• Cons: needs lots of data; high risk of overfitting; long training time.
• Layers: all trainable (nothing to freeze).
• LR/schedule: typical starting point SGD w/ momentum: lr=0.1 (batch-normalized architectures can use 0.1 with large batch), momentum=0.9, weight_decay=1e-4; use cosine annealing or step decay over 100+ epochs. With small dataset, lower lr (0.01) and more regularization may help.
• Augmentation: heavy augmentation required (see below). Use aggressive regularization (dropout in head if added), label smoothing, early stopping, cross-validation.

(b) Fine-tune a pretrained ResNet (recommended)

• Pros: leverages learned low/mid-level features from large corpus — usually best sample efficiency and accuracy.
• Cons: needs careful tuning of LRs and which layers to unfreeze; might retain unwanted biases if domain differs.
• Layers to freeze/unfreeze:
  • Option 1 (safe): Freeze backbone (except BatchNorm behavior), replace final FC with new classifier, train head for 5–10 epochs; then unfreeze last block(s) (e.g., layer4) and train both head + those blocks.
  • Option 2 (aggressive): Unfreeze all layers but use discriminative LRs (lower for early layers, higher for head).
  • For BatchNorm: either keep running stats (eval mode) if freezing, or fine-tune BN params if unfreezing to adapt statistics—use small lr.
• LR/schedule:
  • Head-only: lr_head ∼ 1e-3 to 5e-3 (AdamW) or 0.01 (SGD). Train 5–20 epochs.
  • When unfreezing backbone: backbone_lr ∼ head_lr * 0.1 → backbone_lr in [1e-4, 1e-5] for AdamW, or 1e-3→1e-4 for SGD.
  • Use cosine annealing or linear warmup then decay. Total epochs 20–80 depending on overfitting.
• Regularization: weight_decay 1e-4–1e-2, label smoothing 0.1.
• Why: pretrained filters capture edges/textures; slight fine-tuning adapts representations to target classes without destroying generic features.

(c) Pretrained ResNet as fixed feature extractor + linear classifier

• Pros: fastest, low compute, robust baseline; low overfitting risk.
• Cons: can't adapt features to domain-specific cues; may be suboptimal vs fine-tuning.
• Layers: freeze entire backbone; replace and train single linear layer (or small MLP).
• LR/schedule: lr for classifier ∼ 1e-2–1e-3 (AdamW/SGD), train 10–30 epochs with decay; early stopping.
• When to use: if compute or labels are limited, or as sanity check.

Data augmentation (apply in all cases, heavier for scratch):

• Basic: RandomResizedCrop, RandomHorizontalFlip, Normalize (ImageNet mean/std if using pretrained).
• Color/appearance: ColorJitter (brightness/contrast/saturation), RandomGrayscale.
• Stronger methods: RandAugment or AutoAugment, Cutout/CutMix, MixUp (helps generalization).
• Advice: for fine-tuning, keep augmentations moderate; for training from scratch, use aggressive augmentation + regularization.

Practical workflow

1. Baseline: extract features (c), train linear classifier, get quick metric.
2. Fine-tune head only (b), monitor val loss/accuracy.
3. Unfreeze last block(s) and fine-tune with discriminative lr.
4. If still poor and you have more labeled data or domain mismatch, consider training from scratch with very aggressive augmentation and larger regularization.

Validation / tips

• Use stratified k-fold or repeated splits to assess variance on small data.
• Monitor overfitting (train/val gap); if gap large, freeze more layers or add augmentation.
• Save best checkpoints and use test-time augmentation (TTA) if helpful.

Data collection & labeling

• Define a clear policy-violation taxonomy (prohibited content, misleading claims, restricted items, image-text mismatch) with examples and severity labels.
• Collect balanced samples from product listings + synthetic augmentations (blurred logos, text overlays, translations). Pull metadata (category, seller history).
• Labeling strategy: multi-stage. Start with rule-based weak labels (keyword, OCR matches, image classifiers) to bootstrap; run active learning to surface uncertain cases for human annotation. Use layered annotation: primary label + annotator rationale and bounding boxes/text spans. Measure inter-annotator agreement and adjudicate edge cases via SMEs. Maintain a holdout “adversarial” test set with label flips and real-world noise.

Architecture choices

• Late fusion (independent encoders → classifier) is simpler, robust to missing modalities, and cheaper to train; good baseline.
• Early fusion / cross-attention (e.g., ViLT, VisualBERT, FLAVA, or cross-attention head on top of CLIP encoders) better captures image–text interactions (e.g., image contradicts description). Use cross-attention layers that attend image regions to text tokens for nuanced reasoning.
• Trade-offs: cross-attention increases inference cost and latency but improves fine-grained violation detection.

Pretraining vs fine-tuning

• Leverage pretrained vision-language models (CLIP, BEiT/ResNet + BERT, or multimodal transformers). Strategy: freeze base encoders and train a lightweight cross-attention/classification head first; then fine-tune selectively (last N layers) if dataset size allows. Use domain-adaptive pretraining on in-house image-caption pairs to reduce domain shift. Regularize with augmentation and mixup to avoid overfitting.

Interpretability

• Local explanations: Grad-CAM / integrated gradients for image regions; attention rollout for cross-attention to map tokens ↔ image patches.
• Text explanations: extract salient tokens (LIME/SHAP on text) and generate short rationales with a seq2seq explainer trained on annotator rationales.
• Concept-level: TCAV to surface human-understandable concepts (e.g., “brand logo”, “restricted symbol”).
• Counterfactuals: perturb text or mask image regions to show decision flips. Package explanations in UI: highlighted words, bounding boxes, and confidence + rationale sentence.

Deployment & inference cost

• Options to reduce cost: quantization (INT8), distillation to smaller multimodal student models, pruning, model sharding, and GPU/CPU routing. Cache embeddings for repeated listings. Batch and async inference for offline pipelines; real-time pipelines use distilled or late-fusion unimodal fallbacks.
• Missing modalities: design graceful degradation—if image missing, fall back to text-only classifier; if text missing, image-only model. Use a gating model to detect modality reliability (OCR confidence, image quality) and route to appropriate model. Combine unimodal predictions via calibrated ensembling when both present.
• Monitoring: drift detection on input distribution, periodic reannotation of false positives/negatives, latency/SLA tracking, human-in-the-loop review for high-risk decisions.

Evaluation & governance

• Measure precision/recall by violation class, AUC, and human-AI agreement. Monitor downstream business metrics (removal accuracy, appeal reversal rate). Ensure audit logs with inputs, model outputs, and explanations for compliance and appeals.

This approach balances detection accuracy, interpretability, cost, and operational robustness for production policy enforcement.

Situation: You need a reliable image classifier for a niche product with only a few hundred labeled images, but you can leverage large generic datasets.

Strategy (high-level):

• Start with transfer learning from a strong pretrained backbone (ResNet50, EfficientNet-B3, or a vision transformer pretrained on ImageNet or ImageNet-21k).
• Combine transfer learning with domain-adaptive techniques and aggressive, realistic data augmentation so the model sees plausible intra-class variation.

Concrete plan:

1. Baseline & preprocessing

• Baseline: train a simple classifier on top of frozen pretrained features (global avg pool → FC) to get a quick benchmark.
• Normalize inputs with the same stats as pretraining; resize/crop to model input.

2. Data augmentation (critical for few-shot)

• Photometric: brightness/contrast/saturation, color jitter, random grayscale.
• Geometric: random crop/resize, small rotations, horizontal flips, perspective jitter.
• Domain-specific: simulate lighting/background/product orientations you expect.
• Use MixUp/CutMix and AutoAugment/RandAugment to expand effective data.

3. Freezing vs fine-tuning

• Phase 1: freeze backbone, train classifier head with a moderate LR (e.g., 1e-3) until head converges.
• Phase 2: unfreeze last block(s) of the backbone (layer-wise) and fine-tune with a lower LR (1/10 to 1/100 of head LR) and weight decay. For ViT or large CNNs, unfreeze progressively: last conv block → last 2 blocks → entire network only if validation improves.
• Use discriminative learning rates: smaller for earlier layers, larger for later layers.
• Early stopping and checkpointing to avoid overfitting.

4. Domain adaptation

• If labeled target data is small but you have unlabeled in-domain images, use unsupervised domain adaptation: fine-tune with consistency regularization (e.g., FixMatch), pseudo-labeling with confidence thresholds, or adversarial feature alignment (DANN) to reduce domain gap.
• If you have synthetic images or product CAD renders, perform domain randomization and then adversarial adaptation or feature matching.

5. Regularization & sample-efficiency

• Use strong weight decay, dropout in classifier head, label smoothing.
• Consider few-shot specialized methods (ProtoNets, fine-tuned meta-learning) if classes or shots are extremely low.
• Ensembling of multiple augmentation-trained checkpoints for production robustness.

Risks of domain mismatch

• Pretrained features may focus on textures/objects not present in niche product images (backgrounds, illumination). This can cause negative transfer.
• Overfitting to augmentation artifacts or spurious correlations (e.g., box color).
• Evaluate domain shift by visualizing feature embeddings (t-SNE/UMAP) for source vs target and by monitoring generalization gap.

How to evaluate transfer learning is helping

• Compare against baselines: (A) training from scratch, (B) frozen-head only, (C) progressively fine-tuned. Use same train/val splits and augmentation.
• Metrics: accuracy, precision/recall, F1, calibration (ECE), and class-wise recall for imbalanced classes. Report confidence intervals via bootstrapping.
• Learning curves: plot performance vs number of labeled examples. Transfer should yield higher performance and steeper gains at small sample counts.
• Ablation: remove pretraining or domain-adaptation components to quantify contribution.
• Statistical test: paired bootstrap or McNemar’s test to confirm improvements aren’t due to chance.

Production considerations

• Monitor drift in deployment; collect hard negatives and mislabeled samples to iteratively fine-tune.
• If latency is constrained, distill the fine-tuned model to a smaller student model (knowledge distillation).
• Maintain a retraining pipeline with labeled+pseudo-labeled data, continuous evaluation, and human-in-the-loop verification for low-confidence predictions.

This approach balances quick wins (frozen head), careful fine-tuning to avoid overfitting, domain adaptation when unlabeled in-domain data exists, and rigorous evaluation to prove transfer learning is beneficial.

Situation: Build a TensorFlow pipeline to classify listing images as "professional" vs "amateur" and surface host-facing feedback.

Pipeline overview:

1. Data and splits

• Shuffle then split: 80% train, 10% val, 10% test stratified by label and by host to avoid leakage (ensure same host’s images appear in only one split).
• Maintain a held-out temporal test set (e.g., last month) for production validation.

2. Preprocessing & augmentation

• Resize to model input (e.g., 224 or 320). Normalize per ImageNet mean/std.
• Augment (only in training): random crop/resize, horizontal flip, color jitter (brightness/contrast/saturation), random rotation, slight perspective warp, gaussian blur. Use tf.data pipelines with tf.image and tf.keras.preprocessing layers for performance.

3. Transfer learning

• Use EfficientNet-B0/B2 or MobileNetV3 backbone (trade-off accuracy vs latency). Initialize with ImageNet weights, remove top, add global avg pool → dropout → dense(1, sigmoid). Fine-tune top blocks after initial frozen training.
  Example model skeleton:

import tensorflow as tf
base = tf.keras.applications.EfficientNetB0(include_top=False, weights='imagenet', input_shape=(224,224,3))
x = tf.keras.layers.GlobalAveragePooling2D()(base.output)
x = tf.keras.layers.Dropout(0.3)(x)
out = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(base.input, out)

4. Handling class imbalance

• Analyze class ratio. Options: class weights in loss, focal loss, or modest upsampling of minority. Prefer class weights + focal loss to avoid overfitting artifacts.
• Use metric-aware thresholding on validation set to control precision/recall trade-offs.

5. Evaluation metrics

• ROC AUC for overall separability.
• Precision@k (e.g., top 1%, top 5%) relevant for surfacing high-confidence “professional” examples — compute by ranking model scores and measuring precision in top-k fraction per host or overall.
• Precision, recall, F1, PR AUC. Calibrate probabilities (Platt or isotonic) for interpretable scores.

6. Inference & deployment

• For upload-time scoring: deploy a lightweight TFLite/Mobile model on client or a low-latency TF-Serving / FastAPI endpoint in inference clusters. Use batching for server inference, autoscaling, and GPU/TPU for heavy throughput.
• Return score + explainability: top contributing signals via Grad-CAM heatmap thumbnails and a short textual checklist (e.g., "low lighting", "cluttered background", "low resolution") derived from secondary small classifiers or heuristics.
• For hosts: if score below threshold, provide targeted tips and show exemplar high-scoring images.

7. Feedback loop & continuous improvement

• Instrument actions: which images hosts replace, override labels, bookings conversion uplift per image, manual QA labels via crowdsourcing.
• Store post-deployment signals: host edits, reuploads, manual reviews, and downstream metrics (clicks, bookings). Use these to:
  • Retrain periodically (weekly or monthly) with recent data.
  • Use human-in-the-loop for ambiguous/low-confidence predictions to expand labeled set.
  • Run online A/B tests for UI changes and thresholding logic.
• Monitor drift: data distribution, model score distributions per region/device; set alerts for metric degradation.

Trade-offs & considerations:

• Latency vs accuracy when choosing backbone.
• Explainability vs compute (Grad-CAM expensive; use cached or asynchronous generation).
• Privacy: process images securely and follow host consent.

This pipeline provides a robust, production-ready approach: performant transfer learning, careful imbalance handling, business-aligned metrics (precision@k), low-latency deployment, and a feedback loop to continually improve model quality.