On-Device and Edge ML Questions

Approach summary
I would combine model compression (pruning + distillation), aggressive quantization, and hardware-aware compilation to meet <50MB and <50ms while preserving accuracy.

Concrete techniques

• Tree pruning / sparsification
  • Remove low-gain leaves and merge near-equal leaf values; limit max depth and leaf count. Trade-off: reduces size and inference ops but may lose fine-grained decisions—use validation loss and SHAP to guide pruning thresholds.
• Quantization
  • Post-training int8 quantization of tree scores and thresholds; or 4-bit per-leaf packing with per-tree scaling. Trade-off: small accuracy drop vs large size and cache benefits.
• Distillation
  • Train a smaller GBDT or a neural surrogate (tiny MLP or decision-rules net) on teacher predictions to recover accuracy lost by pruning/quantization.
• Model compilation & operator fusion
  • Use TVM/XGBoost’s Lite/ONNX + platform-specific backends (ARM NEON, RISC-V intrinsics) to inline comparisons and vectorize predictions; enable branchless evaluation where possible.
• Storage/layout
  • Pack trees in contiguous arrays, align to cache lines, compress with run-length or delta encoding and decompress fast at load.

Trade-offs

• Aggressive compression (4-bit + heavy pruning) → smallest size/fastest latency, higher inference error. Distillation recovers some accuracy at cost of additional training.
• Compilation and vectorization improve latency with no accuracy loss but require device-specific builds and maintenance.

Testing strategy

• Correctness: unit tests comparing teacher vs compressed model predictions (MSE, top-k parity), bit-exact checks for quantized ops on representative inputs.
• Performance: microbenchmarks measuring P50/P95 latency and memory peak on each device; profile CPU counters (cache misses, branch mispredictions).
• Robustness: evaluate distributional shifts, measure calibration (ECE), and run A/B or shadow deployment on a subset of devices.
• CI: automated compression pipeline with regression thresholds (max allowed accuracy drop, latency target) and per-target artifact generation.

This plan balances practical constraints, recovers accuracy via distillation, and ensures per-device validation and maintainability.

Requirements & overview
Design a federated learning (FL) system that trains a periodic global model from mobile devices while preserving privacy, tolerating heterogeneous compute/network, enabling local personalization, and defending against malicious updates.

Client selection

• Use stratified sampling: balance demographics, device capability, and recent activity to reduce bias.
• Prefer hybrid scheduling: a pool of eligible devices (battery, connectivity, CPU) with randomized selection each round to improve convergence and fairness.
• Weight updates by local data size or estimated contribution (importance sampling).

Communication strategy

• Periodic synchronous rounds with soft deadlines (e.g., 1–2 minute window). Allow opportunistic asynchronous updates as fallbacks.
• Reduce bandwidth via sketching/Top-k sparsification, 8-bit quantization, and delta compression; use resumable uploads and retry.
• Split models into core (small) + personalization heads to reduce transfer.

Secure aggregation & privacy

• Run an MPC-based secure aggregation protocol so server only observes aggregated sum (e.g., Bonawitz et al.), combined with client-side DP (Gaussian noise calibrated to epsilon) applied before upload.
• Use secure key management and dropout-resilient masking; audit cryptographic parameters.
• Consider hybrid privacy: stronger DP for sensitive features, lower noise for utility-critical layers.

Local personalization vs global

• Multi-task approach: global backbone + local heads; or meta-learning (Per-FedAvg/FOMAML) for fast personalization.
• Trade-offs: more personalization improves per-user accuracy but reduces global generalizability. Use periodic global fine-tuning and periodic model distillation from global to local heads.

Handling stragglers

• Use soft timeouts and partial aggregation: accept top-K timely clients; employ asynchronous aggregation with staleness-aware weighting (decay older updates).
• Allow checkpointing and incremental updates to avoid re-sends.

Poisoning & malicious updates

• Server-side robust aggregation: median/trimmed-mean, Krum, coordinate-wise clipping, and model update norm clipping.
• Anomaly detection: monitor cosine similarity, loss on holdout validation set, per-client contribution history; quarantine suspicious clients and run rollback.
• Use secure attribution logs and periodic verifiable audits; combine with reputation scoring and adaptive client selection.

Metrics & monitoring

• Track global validation, per-segment drop, utility/privacy trade-offs (epsilon), bandwidth, and client failure rates. Run A/B experiments for defense techniques.

This design balances privacy, efficiency, and robustness while enabling practical personalization for mobile-scale FL.

Clarify requirements & constraints

• 30 FPS on mobile CPU → per-frame budget ~33 ms P95. Model <50 MB on-device. Real-time detection (likely single-class or multi-class?), acceptable accuracy/latency trade-off, supported device families (Android ARM, iOS).

High-level architecture

• On-device inference pipeline: Camera → Preprocess (resize, normalize) → Model (backbone + detection head) → Postprocess (NMS, smoothing) → App logic.
• Offline/cloud: heavy training, architecture search, analytics, model store, telemetry.

Model family & specific variants

• Start with mobile-first detectors: MobileNetV3/Small or EfficientNet-lite backbones + SSD-lite or YOLOv5n / YOLOv8n / NanoDet. Consider Detectors:
  • MobileNetV3-S + SSD-lite (fast, small)
  • EfficientNet-Lite0 + SSD or CenterNet-lite
  • YOLOv5n / YOLOv8n for higher accuracy if still <50MB
• Use NAS/compound scaling to pick width/height to meet size/latency.

Optimizations

• Quantization: Post-training quantization (int8) with calibration; if accuracy drops, use quantization-aware training.
• Pruning: Structured channel pruning targeting backbone to reduce FLOPs; retrain after pruning.
• Operator fusion: Fuse conv+bn+activation; use NNAPI/Metal/HAL delegates for fused ops.
• Kernel optimizations: Use platform-specific kernels (ARM Compute Library, XNNPACK).
• Input resolution & batching: Reduce resolution (e.g., 320x320) and use single-frame inference pipeline.
• Model distillation: Teacher-student to retain accuracy.

Deployment & validation plan

• Offline: Automated benchmark suite across representative devices measuring latency P50/P95, memory, power, accuracy (mAP, per-class), and failure modes. CI that runs quantized/pruned builds.
• On-device A/B: staged rollout with telemetry (inference time histograms, top-k errors, user metrics). Canary on low % then ramp.
• Validation: Unit tests for output ranges, E2E tests with synthetic and real-world datasets, regression tests on accuracy and latency.

Fallback & runtime behavior

• Graceful degradations: If P95 exceeds target, fallback to lighter model (smaller variant) or reduce resolution/skip frames (process every nth frame) while preserving UI responsiveness.
• Fail-safe: If model fails, send placeholder UI and log for offline collection.
• Dynamic adaptation: Runtime profiler chooses model variant per-device and under thermal/CPU load.

Trade-offs & why

• Mobile-first backbones + lightweight heads meet size/latency constraints; quantization and pruning give biggest gains with minimal accuracy loss when combined with QAT and distillation. Offline cloud handles heavy ops and continuous improvement; runtime fallbacks maintain user experience.