On-Device and Edge ML Questions

Requirements & constraints:

• On-device model variants compared across iOS users while complying with Apple privacy (no persistent device identifiers, limited telemetry, user consent).
• Low bandwidth; preserve utility for statistical tests despite added noise.

Assignment to buckets:

• Use client-side deterministic, hashed bucketing derived from an ephemeral per-installation ID that Apple permits (e.g., identifierForVendor is not permitted for telemetry — instead generate a random installation ID stored in Keychain with user consent). Compute bucket = HMAC_SHA256(install_id, experiment_salt) mod N. This gives stable assignment, uniformity, and prevents server-driven reassignments. Include versioning so reassignment requires salt change.

Local metrics collection:

• Collect only event-level, non-PII metrics aggregated locally per epoch:
  • Counts (inference count, exposures)
  • Performance signals (latency ms histograms, success/failure booleans, scalar model score distributions)
  • UX outcomes (task completion boolean, time-to-complete)
• Keep raw logs off-device. Aggregate into per-device summaries (sums, counts, histogram buckets, means) over a chosen window (e.g., daily).

Privacy-preserving aggregation:

• Apply local differential privacy (LDP) or secure aggregation:
  • Preferred: Secure Aggregation + anonymized upload: clients encrypt per-device summaries and upload; server can only decrypt aggregate if threshold T of users contributed (protects small cohorts).
  • If Secure Aggregation unavailable, apply LDP (e.g., randomized response for booleans, Laplace or Gaussian noise for sums/means) with ε tuned for acceptable utility.
• Strip or never collect PII; use only install_id hashed client-side; server stores only experiment bucket and aggregated stats.
• Include metadata: device model, iOS major version, app version — but only coarse-grained bins to avoid fingerprinting.

Server-side aggregation & analysis:

• For secure-aggregated uploads, compute global sums, counts, and histograms per bucket and cohort strata (OS bin, app version bin).
• If LDP used, apply bias-correction to de-noise estimates and compute variances including added noise variance.
• Estimate metrics: mean conversion, CTR, latency percentiles (from histograms).

Statistical significance with sampling and privacy noise:

• Use pre-specified primary metric and minimum detectable effect (MDE). Compute power and required sample size accounting for:
  • Sampling fraction (only consenting users)
  • Added noise variance from LDP or thresholding from secure aggregation.
• For tests:
  • Use two-sample tests with adjusted variance: Var_total = Var_sampling + Var_privacy_noise.
  • For proportions: z-test with corrected standard error; for means: t-test with noise-corrected SE or bootstrap resampling on aggregated data.
  • If histograms, use delta-method to approximate percentile variances or use quantile bootstrap on server-aggregates.
• Control for multiple comparisons (Bonferroni or Benjamini-Hochberg) and use sequential testing corrections (alpha-spending / O’Brien-Fleming) for continuous monitoring.
• Report confidence intervals that reflect both sampling and privacy noise; require that effect size exceed combined noise floor before declaring significance.

Operational best practices:

• Pre-register experiments: metric, hypothesis, cohorts, analysis plan.
• Run experiments long enough to reach required effective N after discounting noise.
• Monitor data quality, cohort balance, and consent rates; fallback to longer windows if noise too large.
• Provide tooling to simulate privacy noise and power before launch.

This design balances stable client-side assignment, minimal local aggregation, privacy via secure aggregation or LDP, and statistically sound inference by accounting for added noise in variance estimates and power calculations.

Requirements:

• Functional: predict on-device with low latency, privacy, offline support.
• Non-functional: small binary size, low memory/CPU, energy-efficient, ability to update/rollback.

High-level lifecycle:

1. Training & export

• Train in PyTorch/TensorFlow; instrument training to produce deterministic checkpoints, export to interoperable formats (TorchScript/ONNX/TF SavedModel).
• Validate outputs on representative mobile inputs and measure latency/accuracy on CPU.

2. Conversion to Core ML (.mlmodel)

• Use coremltools to convert from ONNX/TorchScript/TensorFlow to .mlmodel. Specify input/output shapes, precision (fp32 → fp16/INT8), and metadata (version, signature).
• Run unit tests comparing outputs to reference to ensure numerical parity.

3. Compilation to .mlmodelc

• Compile with coremlcompiler for target iOS version/arch to produce optimized .mlmodelc. This optimizes operator kernels and bundles metadata for runtime.
• Test on physical devices across CPU/GPU/Neural Engine configurations.

4. Packaging & delivery

• Embed in app bundle for simplest, offline-first delivery; good for small stable models and faster startup.
• Use on-demand download (App Store-hosted assets or secure CDN) for large models or frequent updates. Implement integrity checks (signatures, checksums) and atomic swaps.

5. Versioning & rollback

• Embed semantic version + model hash in metadata. Serve version manifest from backend.
• Client-side: keep current and previous model copies; download new model to temp location, validate, then switch atomically. If validation fails or performance regresses, rollback to previous version.
• Use staged rollouts and A/B tests; monitor key metrics and device telemetry.

Key mobile constraints & strategies:

• Size: quantize to fp16/INT8, prune, distill; compress with packaging and lazy-loading.
• Latency/CPU: use Core ML acceleration, operator fusion, and batching where appropriate.
• Memory: stream inputs, limit peak activation memory, prefer smaller architectures.
• Energy: prefer Neural Engine, reduce frequency of inference, cache results.
• Security: sign models, encrypt downloads, respect user privacy.
• Observability: collect lightweight telemetry and simulated benchmarks to detect regressions.

Trade-offs: aggressive quantization reduces size but can reduce accuracy; runtime downloads allow agility but add network/SEC complexity.

High-level comparison (standard conv vs depthwise-separable conv (DWConv) vs inverted residual / MobileNet-style):

FLOPs & parameters

• Standard conv: high FLOPs and params: for Cin×Cout kernel K×K: K^2·Cin·Cout per output spatial location.
• Depthwise-separable: depthwise K^2·Cin + pointwise 1·Cin·Cout ≈ (1/Cout + 1/K^2) fraction of standard — typically 6–9× fewer FLOPs and several× fewer params.
• Inverted residual (with linear bottleneck + expansion): uses DWConv inside a residual block with small pointwise-expansion factors. Achieves similar FLOP/param savings to DW-sep but with better representational power.

Memory access patterns & practical throughput

• Standard conv: dense GEMM-friendly (im2col→GEMM) — high compute utilization on GPU, efficient cache reuse, lower memory-bound behavior.
• DWConv: memory-bound — reads/writes per channel with little reuse; poor utilization on GPUs unless fused/optimized. On CPU with good multi-threading and vectorization, DWConv can be decent but still memory-dominant.
• Inverted residual: mixes pointwise (GEMM-friendly) and DWConv (memory-bound). Efficiency depends on operator fusion (expand→dw→project) and runtime kernels.

Actual latency (CPU vs GPU)

• GPU: standard conv often faster per FLOP because GPUs favor dense matrix ops. DW-sep and inverted blocks give FLOP reduction but may not proportionally reduce latency unless kernels are optimized or fused; on mobile GPUs (Adreno/Apple) vendor libraries help.
• CPU / NPU / DSP: DW-sep and inverted residuals usually yield large real latency wins on mobile NPUs or optimized inference engines (NNAPI, TFLite delegates). On generic x86 CPU, speedup depends on optimized kernels (AVX, oneDNN); sometimes standard conv wins if DWConv isn't well-optimized.
• Edge accelerators: benefit most from lower arithmetic and memory bandwidth of DW/inverted designs.

Accuracy per FLOP

• DW-sep reduces FLOPs but can reduce representational capacity; inverted residuals recover accuracy by expansion and residual connections, giving higher accuracy per FLOP than naive DW-sep in many cases (MobileNetV2/V3 show good trade-offs).
• For same FLOPs, well-designed inverted residual networks usually outperform pure DW-sep networks.

When to choose each in production

• Standard conv: when model size/compute not constrained (server/GPU inference), or when target runtime has very optimized dense conv kernels and accuracy needs demand full capacity.
• Depthwise-separable: when extreme FLOP/param reduction required and target runtime has good DWConv support (mobile NPUs, vendor-optimized libraries). Good for ultra-light models where top accuracy can be sacrificed.
• Inverted residual (MobileNet family): default choice for edge/mobile — better accuracy-vs-efficiency, residual stability, and practical latency when using fused kernels or optimized runtimes. Choose when you need best accuracy per FLOP under tight resource/latency budgets.

Operational considerations

• Benchmark on target hardware with the production runtime (TFLite, ONNX Runtime, vendor SDK). FLOPs/params are proxies — actual latency depends on kernel support, memory bandwidth, batching, and I/O.
• Use operator fusion, quantization (int8), and delegate backends to unlock real-world gains; inverted residuals often quantize and fuse better than naive DW-sep.
• Validate accuracy vs latency trade-offs with A/B tests; prioritize deterministic latency and energy consumption for battery-constrained devices.

Answer:

I’d take a three-pronged approach: measure latency, memory, and power on-device with Xcode/Instruments; capture Core ML runtime details and execution device choices; and run offline benchmarks with coremltools/Python to iterate quickly.

On-device (Xcode + Instruments)

• Use the Core ML instrument (Templates → Core ML) first — it shows model invocation events, layer timings, and which accelerator was used (CPU/GPU/ANE).
• Time Profiler for CPU call stacks and hotspots if model runs on CPU.
• Allocations and VM Tracker / Memory Graph to measure peak live bytes, allocations by stack, and find memory leaks.
• Energy Log to measure energy impact during inference (combined with repeated runs to get stable numbers).
• Metal System Trace if GPU execution is used (helps inspect GPU queue times).
• Add os_signpost (os_signpost API) around model predict calls in your app to mark start/end; these appear in Instruments and let you measure end-to-end latency including pre/post-processing.

Capturing Core ML runtime details

• Enable Core ML logging in the app: use MLModelConfiguration’s computeUnits to force CPU/GPU/ANE and log configuration.
• Use the Core ML instrument to inspect per-layer timings and device selection. For lower-level info, enable unified logging for "com.apple.coreml" via Console or Console.app to see runtime messages.
• Test with different computeUnits (CPU_ONLY, CPU_AND_NEEDS_GPU, ALL) to compare.

Offline / desktop benchmarking (coremltools & Python)

• Use coremltools to load/inspect models (model spec, layer shapes) and convert/quantize: coremltools.models.MLModel for model metadata.
• For fast latency estimates on macOS, run predictions with MLModel.predict in a tight loop and measure with time.perf_counter() or timeit, including warm-up iterations.
• Use numpy inputs with realistic shapes; measure mean, p50/p90/p99, and stddev.
• Use psutil to monitor host memory during offline runs. For power on macOS, use powermetrics (requires privileges) or Activity Monitor for rough checks.
• Alternative: export to ONNX and benchmark with onnxruntime (cpu/gpu) and pyinstrument/timeit for cross-framework comparisons.

Best practices / caveats

• Warm up the model (several inferences) before measuring to avoid lazy initialization cost.
• Run tests under controlled thermal state and with airplane mode to reduce variability.
• Measure many iterations and report median and tail latencies (p50/p90/p99), peak RSS, and energy-per-inference.
• Use MetricKit in production to collect real-device telemetry (crash/metrics) for long-term monitoring.

Structured (filter/channel) pruning vs unstructured (sparse weights) — short answer:

• Structured pruning usually gives more reliable real-world latency wins on mobile accelerators (ANE, mobile GPU) because it produces smaller dense tensors that map directly to existing fast dense kernels.
• Unstructured sparsity often yields high theoretical FLOP reduction but only translates to actual latency improvements if the runtime and hardware provide optimized sparse kernels or support structured sparse patterns (block/N:M sparsity).

Why structured pruning often beats unstructured for real latency

• Hardware & kernels: Mobile GPUs and NPUs/TPUs are optimized for dense matrix/tensor ops (GEMM, conv). Removing whole channels/filters reduces tensor shapes so those same optimized dense kernels run less work — a straight win.
• Memory & bandwidth: Channel pruning reduces activation and parameter memory footprint in contiguous blocks, improving cache behavior and lowering memory traffic.
• Predictability: Scheduler and memory layouts remain regular so vendor runtimes (Core ML, Metal, NNAPI, XNNPACK) exploit vectorization, SIMD, and tensor cores without special-case code paths.
• Unstructured sparsity, unless the hardware/runtime has specialized sparse kernels, leaves irregular memory access and control overhead; the overhead often wipes out FLOP savings for fine-grained sparsity.

When unstructured sparsity can help

• If hardware supports sparse acceleration (sparse arithmetic units or dedicated sparse kernels) or the sparsity follows a regular pattern (block sparsity, N:M like 2:4), you can get real speedups and memory savings. Also useful when you need maximal parameter reduction while retaining accuracy and can tolerate limited runtime support.

Toolchain / runtime support required

1. Structured pruning (practical path)

• Prune channels/filters at training time or via structured regularizers. Fine-tune to recover accuracy.
• Export and recompile model so layer shapes are reduced. That means:
  • Use framework tooling to rewrite model graph (pruned channels removed) — e.g., PyTorch pruning + TorchScript/ONNX export; TensorFlow Model Optimization + SavedModel conversion.
  • Re-run quantization-aware training or post-training quantization on the resulting smaller graph.
  • Deploy with vendor runtimes that use optimized dense kernels: Core ML/ANE, Metal Performance Shaders, NNAPI-backed drivers, XNNPACK on CPU, or mobile GPU runtimes. These will give near-linear latency gains proportional to reduced compute and memory.

2. Unstructured sparsity (what you need)

• Prefer structured sparse patterns (block sparsity or hardware-supported N:M). They strike a balance between pruning granularity and runtime efficiency.
• Runtime/compiler support required:
  • Sparse kernels in the runtime (cuSPARSE-like, but on mobile: specialized NN runtimes or vendor SDKs). Without this, sparse matrices are often slower.
  • Graph compilers that lower to sparse ops with efficient memory layouts (TVM with sparse schedules, Glow with sparse support, ONNX Runtime with sparse kernels, or vendor SDKs providing sparse convolution).
  • Support for sparse formats (CSR/CSC/blocked) and fused operators to minimize indirection overhead.
• Toolchain flow: train with magnitude pruning or L1 regularization, fine-tune with sparsity schedule, export sparsity metadata, compile with a runtime/compiler that recognizes and optimizes sparse ops. Consider converting to block/N:M sparsity during training for better runtime support.

Trade-offs & recommendations

• If target is typical mobile GPU/ANE/NNAPI with standard runtimes: prefer structured pruning first — predictable latency wins, easier toolchain. Combine with quantization and operator fusion for maximal benefit.
• If you control the runtime (custom runtime or the hardware supports sparse acceleration): use unstructured or block sparsity for higher compression; ensure toolchain (compiler + kernels) supports sparse execution.
• Always measure end-to-end latency on target hardware (including memory, IO, and pre/post processing), not just FLOPs or parameter counts. Profile to find bottlenecks (memory-bound vs compute-bound) and choose pruning strategy accordingly.

Concrete example:

• For an image classification conv-net on iPhone (ANE): removing 30% of filters in early/mid layers and re-exporting to Core ML typically reduces latency proportionally. Applying 80% unstructured pruning without Apple-provided sparse kernels often yields little to no latency improvement despite lower parameter count.

Bottom line: structured pruning is the practical first choice for mobile latency reduction; unstructured sparsity can win only when the runtime/hardware explicitly supports efficient sparse execution or when using regular sparse patterns (block/N:M) that compilers can optimize.