Computer Vision Questions

Requirements & constraints:

• 20 FPS on mid-tier phones (e.g., 2–4 CPU cores, limited memory), good small-object detection, low power. Real-time latency budget ≈ 50 ms/frame including pre/post-processing.

Backbone selection:

• Use a mobile-optimized backbone: EfficientNet-Lite or MobileNetV3-Large. EfficientNet-Lite (B0-B2) gives better accuracy-per-FLOP; MobileNetV3 is faster on some NPUs. Pick based on target device profiling. Use depthwise separable convs, squeeze-and-excitation where supported.

Feature Pyramid & multiscale choices:

• Lightweight FPN (top-down + lateral) on 3–4 levels (e.g., P3–P6) to keep small-object features (higher-res P3). Use BiFPN-lite (weighted fast fusion) if budget allows for better fusion with minimal overhead.
• Preserve higher spatial resolution: avoid aggressive stride >16 for earliest layers—keep stride-8 feature map for small objects.

Detection head: anchor vs anchor-free

• Prefer an anchor-free head (e.g., FCOS or CenterNet-style) to reduce post-processing and anchor tuning; simpler, fewer priors, often faster.
• If using anchors (for legacy or dataset reasons), use sparse, scale-aware anchors with fewer aspect ratios and per-level priors.

Multiscale detection strategies:

• Detect on P3 (high-res) for small objects, P4/P5 for medium/large. Use task-specific scale-aware losses and focal loss to handle class imbalance for small objects.
• Use feature upsampling and context modules (lightweight dilated conv or depthwise ASPP-lite) on P3 to increase receptive field without loss of resolution.

Latency reduction while preserving small-object accuracy:

• Quantize to int8 with calibration-aware training; prefer post-training quantization if accuracy drop small, otherwise quantization-aware training.
• Use operator fusion and NNAPI/Metal delegates to leverage hardware acceleration.
• Distillation: train a smaller mobile model with a larger teacher focusing on small-object examples and hard negatives.
• Prune channels with structured pruning then fine-tune; keep channels in early high-res layers to protect small-object features.
• Reduce post-processing cost: NMS with GPU/accelerator or soft-NMS with class-wise thresholds; limit candidate proposals by pre-filtering with confidence threshold.
• Input-size tradeoff: use multi-resolution pipeline—process full-resolution frame at lower frequency, or tile/ROI-patch high-res crops where motion detection indicates likely small objects.

Evaluation & deployment:

• Profile on target phones, measure end-to-end latency including I/O. Iterate backbone/FP layer choices and quantization. Track AP on small-object subset and FPS; use Pareto frontier to choose final model.

Situation: Choosing between CNNs and Vision Transformers (ViT) for a 224x224 classification dataset of ~200k labeled images.

Short answer / recommendation:

• Both families can work; CNNs are safer for smaller-to-moderate datasets and constrained compute. ViTs shine when you can leverage large-scale pretraining (or heavy data augmentation/self-supervised pretraining) and want state-of-the-art accuracy, especially with transfer from large datasets.

Comparison by axis:

1. Transfer learning behavior

• CNNs: pretrained conv backbones (ImageNet) transfer robustly; few-shot / fine-tuning often effective with modest compute. Layer-wise fine-tuning strategies work well.
• ViT: benefits strongly from large-scale pretraining (supervised or self-supervised). Off-the-shelf ViT pretrained on large corpora transfers well, but training ViT from scratch on 200k may underperform.

2. Data requirements

• CNNs: lower data hunger due to built-in locality/inductive bias.
• ViT: higher data requirement unless pretrained; needs heavy augmentations (RandAug, MixUp) and regularization.

3. Robustness

• CNNs: more robust to small shifts/occlusions due to locality and pooling; but can be vulnerable to texture biases.
• ViT: often more robust to global corruptions and adversarially different patterns if pretrained, but can be less stable without sufficient data/regularization.

4. Training and inference compute

• CNNs: generally cheaper to train and faster inference on edge/CPU with optimized conv kernels. Lower memory for similar accuracy.
• ViT: Transformer blocks have quadratic token attention cost; at 224x224 with 16-patch (14x14 tokens) it's manageable, but ViTs typically require more FLOPs and memory and benefit from hardware with optimized matrix-multiply and larger batch sizes.

5. Production preferences

• Choose CNN if: latency, memory, and deterministic performance are priorities; you want simpler fine-tuning and smaller models for edge.
• Choose ViT if: you can use a well-pretrained checkpoint, need top-end accuracy, plan for continued scaling, or your infra supports larger GPU/TPU inference and you accept slightly higher latency.

Practical tips

• If using ViT, start from large pretrained weights (ImageNet-21k, CLIP, or DINO), use strong augmentations, tune optimizer/weight decay and learning-rate schedules.
• For CNNs, try EfficientNet/ResNet variants and apply transfer learning with gradual unfreezing and label smoothing.
• Consider hybrid: convolutional stem + transformer body or ensemble to get benefits of both.

Conclusion: With ~200k labels, prefer CNNs for simplicity and cost-sensitive deployment; use ViT when you have access to high-quality pretrained models and compute budget, or if maximal accuracy matters and you can invest in tuning.

Start by clarifying objectives: maximize genuine-user convenience (low false reject rate, FRR) while minimizing impersonation risk (low false accept rate, FAR). That trade-off drives dataset, model, system, and privacy choices.

Dataset & training:

• Collect large, diverse data across age, skin tones, facial hair, glasses, masks, lighting, poses, and capture devices. Include hard negatives (look-alikes, family members) and synthetic augmentations (illumination, blur) to improve robustness.
• Label for identity, capture conditions, and spoof types. Use balanced class sampling and long-tail augmentation; maintain privacy-compliant consent and on-device enrollment when possible.

Liveness / presentation-attack detection (PAD):

• Combine multi-modal signals: IR depth (structured light/ToF), stereo/SLAM-based depth, and active challenge-response (blink, head turn). Use a two-stage pipeline: lightweight on-device CNN for fast heuristic checks, then a stronger PAD model or sensor fusion if uncertain.
• Train PAD on real spoofs (print/replay/3D mask) plus synthetic adversarial examples. Use temporal models (LSTM/3D-CNN) to capture micro-movements.

Threshold tuning:

• Evaluate ROC/DET curves, choose operating point by target FAR (e.g., 1e-6) and acceptable FRR. For consumer face unlock, prioritize low FRR at modest FAR; for payments, raise threshold.
• Use per-user adaptive thresholds: calibrate during enrollment to account for intra-user variability; use cohort-based normalization (e.g., compare score to distribution of non-matching scores).
• Continuously monitor production metrics, run A/B tests when changing thresholds, and implement fallback auth when ambiguous.

On-device template privacy:

• Never store raw images. Extract embedding templates via the face-recognition network and store only encrypted templates in a secure enclave/TEE.
• Use cancelable biometrics / keyed hashing: apply a user-specific salt/transformation before storing so templates are revocable.
• Protect with hardware-backed keys (secure element); perform matching inside enclave so templates and raw embeddings never leave device.
• For stronger guarantees, consider differential privacy during model updates and use federated learning with secure aggregation to improve models without centralizing biometric data.

System & operational considerations:

• Optimize models for latency/power via pruning, quantization, and knowledge distillation to meet edge constraints.
• Design for graceful degradation: fast heuristic check -> full model -> alternative auth.
• Maintain attack monitoring, periodic adversarial testing, and a secure update pipeline (signed models) to respond to new spoof types.

This combined ML + system approach balances convenience and security by aligning data collection, multi-modal detection, conservative thresholding for sensitive flows, on-device protected templates, and continuous monitoring.

Requirements & constraints

• Functional: extract printed and handwritten text from images on-device across iPhone/iPad/Mac.
• Non-functional: strong privacy (user data stays private), low latency for interactive use (<200–300ms for live preview), high accuracy for printed & varied handwriting, minimal battery/CPU impact.

High-level trade-offs Apple faced

• Privacy vs. Accuracy: Cloud models can be larger and more accurate but leak data; on-device preserves privacy but limits model size/compute.
• Latency vs. Accuracy: Larger models give better accuracy but increase inference time and battery usage. Real-time UX requires low-latency models or staged processing.
• Device heterogeneity: Newer devices have NPUs/TPUs allowing bigger models; older devices require smaller, faster models—trade-off between consistent UX and per-device optimization.

Design to extend Live Text for robust handwriting recognition

1. Multi-tier model architecture (progressive enhancement)

• Tiny on-device model for real-time preview/interaction (low-latency): lightweight CNN backbone (MobileNetV3 / EfficientNet-Lite) + small sequence model (temporal depthwise separable convs or tiny Transformer encoder) for immediate OCR; optimized: 4–8MB, quantized int8, <150ms on-device.
• Larger on-device model for high-quality local inference on capable devices: CRNN / Vision Transformer hybrid (~20–50MB, int8/FP16), uses NPU acceleration.
• Cloud fallback (optional, opt-in): full transformer-based OCR (large-scale seq2seq with language model) for extremely ambiguous handwriting or long documents.

2. When to fall back to cloud

• Confidence thresholds: model outputs calibrated uncertainty scores (softmax entropy, Monte Carlo dropout or small Bayesian head). If confidence < threshold AND user has opted into cloud-enhanced processing, upload only encrypted, user-consented snippets; otherwise keep local.
• Progressive upload: send minimal crops or embeddings, not full images. Use selective redaction (blur sensitive regions) before upload.
• Offline-first defaults: cloud fallback only when user opts in or for non-sensitive data.

3. Privacy & data strategies

• On-device-first: default inference and personalization stored locally.
• Federated Learning (FL): send model updates (gradients or delta weights) rather than raw images; use secure aggregation so server sees only aggregated updates.
• Differential privacy (DP): apply DP noise to updates and use clipping to bound contribution.
• Local fine-tuning: allow per-user personalization (e.g., frequent contacts, handwriting style) with tiny adapter modules (LoRA-style) stored on-device; merge adapters via server-side aggregation with FL+DP to improve global model without raw data.
• Data minimization: if cloud is used, limit uploads to minimal crops, metadata stripped, encrypted in transit and at rest, with clear UI consent.

4. Model selection & optimization

• Architecture: printed text—lightweight CRNN or CTC-based model; handwriting—transformer-based sequence-to-sequence with visual encoder (ViT/ConvNeXt-lite) + decoder with attention improves long-form handwriting recognition.
• Training: curriculum from synthetic to real: start with synthetic handwriting augmentation (stroke variation, slant, ligatures), then fine-tune on labeled real data. Use data augmentation: elastic distortions, blur, background noise, multi-lingual scripts.
• Compression: knowledge distillation from large teacher to small student; pruning, quantization-aware training, weight clustering, and structured pruning to reduce latency on constrained hardware.
• Early-exit networks: multiple classifier heads enabling quick confident outputs for easy cases and deeper computation only when needed.

5. Battery & latency optimizations

• Hardware acceleration: leverage Core ML + Neural Engine; compile separate kernels per device class.
• Dynamic compute scaling: adjust model size based on battery, thermal state, and user settings (low-power mode).
• Asynchronous processing & caching: precompute embeddings for frames; reuse results across frames; batch small tasks; cancel in-flight heavy inference when new frames arrive.
• Sparse inference & attention sparsity: reduce compute for long sequences.

6. Monitoring, metrics & safety

• On-device telemetry (opt-in, aggregated): track accuracy, confidence calibration, latency, power usage.
• A/B experiments via federated evaluation to test model updates without centralizing raw data.
• Fallback UX: indicate when recognition is low-confidence or when cloud-enhanced result produced (transparency).

Example flow (user scenario)

• Live camera points at handwritten note → tiny on-device model produces instant selectable text with 95%+ accuracy on printed text, 80–90% on clear handwriting. UI shows selection; if user taps “Copy” and confidence low, prompt: “Improve accuracy by sending to Apple cloud?” If user consents, send small encrypted crop for cloud model; return refined transcription using large seq2seq model and optional language-model correction.

Key trade-offs revisited

• Preserve privacy by default with on-device-first, but provide optional cloud for edge cases requiring heavy models.
• Use model compression, distillation and hardware acceleration to balance accuracy vs. latency and battery.
• Use FL + DP + secure aggregation to improve models globally without raw data centralization.

This design balances Apple’s priorities: strong privacy defaults, progressively improved accuracy, and responsive low-power UX across device classes.

I'll describe each defense, how it works, and practical trade-offs in compute, accuracy, and deployment complexity for a production image-classification pipeline.

Adversarial training

• What: Augment training with adversarial examples (e.g., PGD) so model learns robust decision boundaries.
• Pros: Empirically strongest empirical robustness to attacks seen during training.
• Compute: Training cost multiplies (PGD k-steps per batch → k× forward/backward passes). Larger memory/longer GPU time; inference cost unchanged.
• Accuracy: Clean accuracy typically drops (robustness–accuracy tradeoff); careful tuning (trade-off hyperparameter, schedule) mitigates loss.
• Deployment complexity: Moderate — model artifact is standard; CI/CD and monitoring unchanged. Need continual retraining when threat model evolves.

Input preprocessing / denoising

• What: Apply transformations (median filtering, JPEG compression, learned denoisers, autoencoders) to remove adversarial perturbations prior to classification.
• Pros: Simple to add as preprocessing; can recover some clean accuracy.
• Compute: Low to moderate per-request latency (cheap filters) or higher for neural denoisers (additional inference).
• Accuracy: Can hurt clean accuracy and attackers can adapt (expectation-over-transforms). Not certified; brittle to adaptive attacks.
• Deployment complexity: Low for deterministic transforms; higher if serving extra neural models (scaling, latency budgets).

Randomized smoothing (certified robustness)

• What: Create a smoothed classifier via averaging predictions over Gaussian-noised inputs to obtain provable L2 robustness radius.
• Pros: Provides statistical certificates (guaranteed robustness up to radius) — valuable for high-assurance settings.
• Compute: High at inference (many Monte Carlo samples to estimate class probabilities); training cost moderate if using Gaussian augmentation.
• Accuracy: Certified radius trades off with clean accuracy; tight certificates often small for high-dim images.
• Deployment complexity: High — needs batched/noise-parallel inference, careful RNG, and latency/throughput engineering; exposing certificates to stakeholders requires explanation.

Detection networks

• What: Train separate detector to flag adversarial inputs (statistical tests, auxiliary classifier, uncertainty models).
• Pros: Can block or route suspicious inputs for human review; adds defense-in-depth.
• Compute: Extra model inference and logging; modest per-request cost.
• Accuracy: False positives/negatives problematic — may degrade user experience; attackers can evade detectors.
• Deployment complexity: High operational complexity: thresholds, feedback loops, human-in-the-loop workflows, and continual calibration against adaptive attacks.

Practical notes

• Combine defenses: adversarial training + preprocessing or detection gives layered protection but compounds costs and potential accuracy loss.
• Threat modeling: Choose based on attacker capabilities (white-box vs black-box), acceptable latency, and regulatory needs (certified guarantees vs empirical robustness).
• Monitoring & response: Production must include attack detection, shadow-testing of robust models, and retraining pipelines; invest in adversarial testing (A/B with adaptive attacks) before rollout.