Mobile Build and Release Engineering Questions

Brief answer (what anti-rollback is)
Anti-rollback prevents acceptance/boot of older, potentially-vulnerable firmware images. Implementations typically use signed metadata and some monotonic state that only advances.

Signed version numbers

• Firmware image contains a version field signed by the vendor’s private key. Bootloader verifies signature and checks version >= stored allowed version.
• Simple to implement in bootloader; works with software-only devices.
• Weakness: stored allowed version must be protected from tampering (otherwise attacker rolls it back).

Monotonic counters

• A hardware monotonic counter (or secure monotonic storage) is incremented on successful update and compared to image version. Counters can be non-volatile and one-way.
• Stronger than unsigned storage because counters resist decrement. Good for devices with TPM/RTC-backed monotonic NV counters.
• Trade-offs: limited counter width (wrap/overflow), wear on NVM, need for atomic update on power loss.

Hardware-backed eFuses / Secure Elements

• eFuses can permanently record a max-version (or revoke keys); secure elements store sealed monotonic counters and keys. These are tamper-resistant and survive reboots/resets.
• Most robust: attacker cannot reset eFuses or extract keys. Secure elements can handle secure boot, crypto ops, and monotonic storage.
• Trade-offs: cost, limited flexibility (eFuses are one-time), complex provisioning and field recovery.

Key rotation & recovery

• If vendor rotates signing keys, bootloader must support key migration: either keep old keys to verify already-deployed images, use an update path signed by both keys, or embed a secure key-rotation protocol in hardware.
• Recovery complexity: if a monotonic counter was advanced past a bad release, you cannot roll back even to trusted older image—need dual-bank rollback-safe designs or a signed “rollback-allow” recovery sealed by secure element. eFuses make recovery very hard.

Practical recommendations for embedded systems

• Use signed versions plus a protected monotonic counter in secure element if available.
• Implement dual-bank firmware with atomic swap and verify-before-commit so failed updates don’t advance monotonic state.
• Plan key rotation: support key chains or vendor-signed rotation manifests stored/verified by secure hardware.
• Balance cost vs security: eFuses/SEs for high-risk devices; software protections for low-cost constrained devices.

Brief definition
Atomic update semantics**:** the firmware update either completes fully and becomes the new running image, or the system remains exactly in the previous known-good state. No intermediate partial state should be bootable.

Partial-write failure example

• During an OTA flashing of main firmware region, power drops after 60% of bytes written. Bootloader tries to run that partially written image: CRC/signature fails or, worse, corrupted vector table causes undefined behavior and brick.

Two architectural patterns to enforce atomicity

1. Dual-bank (A/B) with validated switch

• Maintain two full firmware partitions: active (A) and staging (B).
• Write new image to B, verify signature + CRC, run self-tests while still booted on A.
• Atomically flip a single metadata flag or boot pointer (stored with redundancy and CRC) to mark B as active.
• Bootloader reads pointer; if B fails CRC, it falls back to A.
• Pros: simple rollback, safe even if power fails during write. Cons: needs extra flash.

2. Copy-then-commit with bootloader-led verification (transactional update)

• Staged write with per-chunk checksums and a journal/state machine in a small protected area (e.g., internal flash or EEPROM).
• Steps: write chunk -> mark chunk committed -> after all chunks, write “commit” marker atomically (single-word write with ECC or mirrored copies).
• Bootloader treats image as valid only if commit marker and full image CRC/signature pass; otherwise it replays journal or uses backup.
• Pros: works with single-bank constrained devices; smaller overhead. Requires careful atomic markers and wear management.

Implementation notes / best practices

• Always verify signatures/CRC before switching.
• Use redundant metadata (mirrors + inverted copies) and single-word atomic flags.
• Keep a minimal ROM bootloader that can always fallback and perform recovery.

Overview / goals
Design a protocol for chunked firmware/image transfer using a Merkle tree so multiple chunks download in parallel, resume after interruption, and verification uses minimal RAM (suitable for microcontrollers).

Manifest (helper data)

• Contains: image size, chunk size, total chunks N, root hash, hash-algorithm, and a compact Merkle helper table: for each chunk index i, the list of sibling hashes on the path to root (or a pointer to a small per-level compacted array). Store helper entries in canonical order so manifest is small and streamable.
• Include optional bitmap checkpoint (sparse) describing already-applied chunks and a chunk sequence nonce/version.

Chunk format & download

• Each chunk = payload bytes + chunk index + chunk payload hash (leaf hash).
• Device requests ranges in parallel (HTTP/CoAP GET range or P2P). Server can serve any chunk independently.

Verification on receipt (minimal RAM)

• Device computes leaf hash of chunk immediately (uses small buffer = chunk size or smaller if streaming hash).
• Using the manifest sibling hashes for that index, iteratively compute parent hashes up to root:
  • For each level: hash = H( left || right ) with left/right order from index bit.
  • Only need to keep current running hash and one sibling hash from manifest per level → O(1) RAM beyond chunk buffer.
• Compare computed root to manifest root; accept chunk as “verified leaf”. Mark bitmap entry true (persisted in flash).

Full image verification & finalization

• After all chunk bits set, device can either:
  • Reconstruct streaming write to final image while recomputing internal Merkle (same O(1) extra RAM), or
  • Verify by computing a rolling hash over the entire written image if manifest includes an overall image hash (recommended). This requires only a small buffer for streaming.

Resumability & partial re-download

• Persistent bitmap allows skipping already-verified chunks across reboots.
• Support range requests and server-side prioritization: re-request missing chunks first.
• For power-loss mid-chunk: partial chunk discarded; only fully-verified chunks marked.

Integrity & low-memory tradeoffs

• Chunk size tuned to available RAM; use streaming hash (e.g., SHA-256 incremental) so buffer << chunk size.
• Manifest sibling lists grow as O(N log N) but can be micro-optimized:
  • Use sparse storage: compute some levels on server and include only upper-level hashes; or
  • Use k-ary Merkle (larger fanout) to reduce depth and manifest size at cost of larger sibling per level.

Example parameters

• 1 MB image, 4 KB chunk → 256 chunks, depth 8. Manifest sibling hashes ~256 * 8 * 32 bytes naive; compress by storing per-level arrays or using inclusion proofs on demand.

This protocol keeps RAM low (streaming hashes + O(1) per-level state), supports parallel downloads and resumability via persistent chunk bitmap and per-chunk inclusion proofs in the manifest.

Clarify goals (embedded lens)
Support 100k simultaneously managed constrained devices with staged rollouts, delta patches to minimize egress, reliable resume/verification, and per‑device rate/bandwidth controls.

High-level components

• Artifact Store: S3 (or equivalent object store) holding full images and delta artifacts; objects signed and versioned.
• Delta Generator Service: serverless or CI job that produces binary deltas (bsdiff/courgette/Google’s courgette-like) between versions, stores metadata (source hash, delta size).
• CDN + Edge Cache: global CDN serving artifacts to reduce origin egress; use signed, short‑lived URLs for access control.
• Update Server / Orchestrator: central service that manages rollout policies, issues signed URLs/tokens, tracks device state, and coordinates canaries.
• Device Registry: DB (Cassandra/Postgres) keyed by device id, current fw, hardware variant, cohort tags, connectivity profile, last checkin.
• Telemetry & Rollback Service: collects install success/fail metrics for automated rollbacks.

Delta flow

• Build pipeline: when a new image is produced, for each supported base version produce delta artifacts to that image (prioritize most common bases).
• Store: full image + deltas in artifact store; generate manifest listing required base hashes and delta chains.
• Device fetch: device sends manifest with current version; update server selects smallest delta chain (prefer single-step delta), returns signed URL(s) to CDN.
• Resume/verify: use chunked transfer (HTTP Range or resumable protocol); device verifies signature/hash and applies delta in safe partition (A/B or swap slots).

Canary & grouping

• Devices tagged in registry with metadata: hw variant, geographic region, connectivity class, battery profile.
• Cohorts: define canary cohorts (e.g., 1% sampled across variants and regions) and staged percentages (1 → 5 → 25 → 100) with time windows.
• Selection: orchestrator queries registry using tags + random sampling seeded by deployment id to pick deterministic canaries.
• Monitoring: enforce rollout gates based on failure rate thresholds from telemetry.

Rate limits & bandwidth throttling

• CDN-edge: enforce per-token bandwidth caps and concurrent connection limits via signed URL metadata (e.g., max-bytes-per-second, validity window). This reduces origin egress and central load.
• Update server issues time-windowed tokens limiting total bytes allowed; tokens tied to device id and manifest.
• Server-side policing: API gateway enforces token bucket per device id (leaky bucket with refill rate = device bandwidth cap); rejects/queues requests over limit.
• Device-side behavior: embedded client implements local token bucket, exponential backoff on errors, resume support, and configurable chunk size (e.g., 16–64 KB) to match network/RAM constraints.
• Connection scheduling: orchestrator schedules large downloads during off-peak times per device profile to reduce simultaneous egress peaks.

Why this fits embedded constraints

• Delta patches and CDN minimize bytes transferred (critical for metered links).
• Chunked/resumable transfers and A/B updates protect devices with small RAM/flash and intermittent links.
• Per-device tokens + client-side throttling avoid swamping low-power radios and respect carrier limits.

Operational tradeoffs

• Precomputing deltas for many base versions increases storage/compute; prioritize common bases.
• Short signed URLs improve security but require clock sync and retry logic on devices.
• A/B requires extra flash; fallback: dual-stage write with verification if space constrained.

Overview
Compare three approaches: (A) Full-image encryption at rest & transport, (B) Encrypt sensitive fields only, (C) Transport-layer only. For each: threat model, performance on constrained devices, and product-specific recommendation.

A — Full-image encryption (rest + transport)

• Threat model: adversary with physical access or supply-chain interception aims to extract code/IP, keys, or inject modified firmware.
• Performance: storage overhead for ciphertext, bootloader needed for decryption (secure boot + KDF/HW crypto). CPU cost low if hardware AES exists; otherwise significant RAM/CPU and boot-time latency. Key provisioning complexity.
• Recommendation:
  • Consumer: recommended if IP protection matters or OTA via untrusted networks; use hardware crypto and incremental decryption.
  • Industrial: recommended for sensitive infrastructure; pair with TPM/HSM.
  • Regulated: required for many sectors (medical, automotive); follow standards (e.g., FIPS).

B — Field-level encryption

• Threat model: protects secrets/config only; attacker with full image can still reverse-engineer code or exploit vulnerabilities but cannot read specific data.
• Performance: lower crypto cost, simpler boot; parsing needed to locate/decrypt fields; smaller RAM footprint.
• Recommendation:
  • Consumer: good balance for low-cost devices where full-image crypto is too expensive.
  • Industrial: acceptable for less-critical devices; prefer for configs/keys but combine with code integrity checks.
  • Regulated: usually insufficient alone—must meet regulation-specific requirements.

C — Transport-layer only (e.g., TLS)

• Threat model: defends against network eavesdropping and MitM but leaves firmware at-rest vulnerable to physical extraction or tampering if storage compromised.
• Performance: minimal impact on boot; runtime TLS overhead exists but often already supported; no added storage/boot complexity.
• Recommendation:
  • Consumer: acceptable for low-risk features and small devices if physical access is controlled.
  • Industrial: risky—physical attacks more likely; add at-rest protections.
  • Regulated: generally unacceptable as sole control.

Trade-offs & Practical Guidance

• Use layered defenses: code signing + integrity checks is mandatory regardless.
• If hardware crypto present, favor full-image encryption for high-value devices.
• If constrained, encrypt only sensitive fields and ensure strong access control and signed images.
• Key management is the hardest part—use secure elements, unique device keys, and rotate/update procedures.