IoT and Edge Device System Architecture Questions

Brief overview
End-to-end flow: sensor samples → local buffer → aggregator → protocol translation & auth at gateway → cloud ingestion. Gateway enforces validation, batching and policies to save battery and ensure reliability.

Device (embedded)

• Sampling: wake, read sensors, timestamp (RTC), low-pass/filter, attach sequence number.
• Format: compact binary (CBOR) with CRC16 and device ID.
• Local storage: circular ring buffer in flash (wear-leveling), store minimal fields; sleep between samples to save power.
• Send to gateway via BLE/LoRa/IEEE 802.15.4 using lightweight ACKs; retransmit limited times.

Gateway responsibilities

• Receive packets, ACK locally, maintain per-device queues.
• Local validation:
  • CRC and sequence continuity (detect drops/dup).
  • Timestamp sanity (reject/from-device clock drift > threshold).
  • Schema/field ranges (temperature within plausible limits).
  • Device authentication token (pre-shared key or device cert).
• Aggregation & buffering:
  • Time-windowed aggregation (e.g., 1 min average, min/max, counts) and event prioritization (alarms forwarded immediately).
  • Buffer batches in persistent storage; maintain per-device metadata (last seq, battery level).
• Protocol translation & auth:
  • Translate CBOR -> JSON; CoAP/UDP -> MQTT or HTTPS.
  • Gateway holds TLS client certificate or uses cloud SDK; authenticate via mTLS or OAuth/JWT.
• Upload policy:
  • Upload when internet up and either: aggregation interval elapsed, high-priority event, buffer size threshold reached, or scheduled upload.
  • If connectivity lost: continue local buffering until flash quota; apply oldest-drop policy or downsample when full.
  • Use exponential backoff for retries, compress batches, and respect rate limits.

Cloud ingestion

• Endpoint expects authenticated connection (mTLS/JWT), validates payload schema, sequence numbers, deduplicates, and persists raw + aggregated metrics.
• Send ACK/commands back through gateway for OTA or config changes.

Power & robustness notes

• Favor batching to save radio energy; keep critical alarms high-priority.
• Keep validations simple on device; delegate complex checks to gateway.
• Monitor gateway health and expose diagnostics for embedded firmware to adjust sampling or retransmit.

Clarify constraints

• 10k devices, intermittent connectivity, constrained embedded clients (flash/ram limited), secure transport required (TLS), need targeted OTA.

High-level architecture

• MQTT broker cluster (EMQX / Mosquitto cluster / cloud managed AWS IoT Core or Azure IoT Hub) behind HA load balancer. Use persistent clustered storage (etcd / shared DB) for session/state and horizontal scaling.

Broker selection & scaling

• Use a clustered broker that supports horizontal scaling and MQTT bridge for cross-region. Autoscale worker nodes, monitor connections/latency, and shard by client-id or topic namespace if needed.

Authentication & per-device credentials

• Mutual TLS with per-device X.509 certificates issued by internal CA (rotate certificates). Alternative: per-device long-lived JWTs refreshed via bootstrap. Store device identity & metadata in device registry; map cert CN to device-id.

TLS

• Enforce TLS 1.2/1.3, strong cipher suites, revocation via CRL/OCSP or certificate expiration.

QoS

• Use QoS 0 for high-frequency telemetry where occasional loss is acceptable.
• Use QoS 1 for critical events/commands (device state, OTA control).
• Avoid QoS 2 due to overhead on constrained devices.

Offline buffering (device & gateway)

• Device-side: small circular persistent queue in flash (e.g., N entries or M KB), persist pointers, exponential backoff reconnect. On reboot resume send with QoS 1.
• Gateway (for very constrained devices): act as local MQTT proxy, aggregate messages, keep larger durable queue and replay when WAN up.
• Implement deduplication with monotonic sequence numbers / timestamps.

Topic naming & conventions

• Hierarchical, predictable:
  • telemetry: device/{org}/{device-id}/telemetry
  • state: device/{org}/{device-id}/state
  • commands: device/{org}/{device-id}/cmd
  • ota control: device/{org}/{device-id}/ota/control
  • ota data (blob): device/{org}/{device-id}/ota/data
• Use retained messages for last-known-state; use ACLs on broker to restrict publish/subscribe by topic patterns. Use wildcards for fleet-level subscriptions: device/{org}/+/telemetry

OTA strategy

• Host firmware blobs on secure HTTP(S) server (CDN) signed with private key.
• Control plane via MQTT: publish OTA job to topic fleet/{group}/ota/control with metadata (version, URL, signature, rollout-id).
• Device verifies signature before applying. Support delta/diff updates where possible.
• Targeting: maintain device groups via registry (tags/attributes). Broker ACLs allow devices in group to subscribe to group topics. Use staged rollouts: test group -> canary -> phased percentage using rollout-id and job status topics (device/{id}/ota/status).
• Rollback: devices keep previous image, verify post-checksum/health-report before committing.

Monitoring & security ops

• Telemetry ingest pipeline with backpressure (Kafka), alerting on high reconnects, failed OTAs, cert expiry. Regularly rotate keys, audit logs.

This design balances embedded constraints (small persistent queues, QoS choices), strong security (mTLS per-device), scalable broker architecture, and a safe, targeted OTA process using MQTT for control plus signed blobs for payload.

Situation & goals
Deliver low‑latency cloud→device commands to 500k NAT‑ed IoT devices (embedded clients). Key constraints: devices have limited RAM/CPU, sit behind NATs, need reliable delivery and secure auth.

High‑level approach

• Use MQTT over TLS (or MQTT over WebSocket) with long‑lived TCP/TLS connections initiated by device to server. Devices maintain a single persistent connection to a geographically nearest broker to traverse NATs.

Session management

• Keep device session state (subscription, pending QoS1/2 messages, lastSeen) server‑side.
• Use MQTT cleanSession=false to persist subscriptions and queued messages.
• Heartbeat/keepalive from device (short, e.g. 30s) + server idle timeout. Implement exponential backoff reconnect with jitter on device.

Persistent connection strategies

• Single TCP/TLS connection per device. Use TCP keepalives and application heartbeats to detect NAT timeouts.
• For constrained devices: minimal memory circular buffer for inflight messages; store larger queues on server.
• Implement Last Will for unexpected disconnect handling.

Message routing & partitioning

• Partition clients by clientID/deviceID using consistent hashing to a broker shard. This ensures a given device’s session lands on the same broker.
• Use L4 load balancer with consistent‑hashing (source/clientID) or DNS with geo‑routing to select broker pool.
• For cloud→device, publish to topic routed to the broker shard owning that client’s session. Use a routing tier (stateless frontends) that consults a central session directory or uses the same hashing function.

Broker layer scaling & session state

• Broker layer: stateless frontends + pool of stateful broker nodes (shards). Scale horizontally: add shards and rebalance using consistent hashing to limit churn.
• Session state stored in a fast distributed KV (Redis Cluster with persistence or Cassandra for larger durability). Keep ephemeral connection state in broker memory, persist durable session metadata to KV for failover.
• Use replication (RAFT/Redis Cluster replication, or Cassandra replication factor) to tolerate node failures.
• For rebalancing: implement session migration/hand‑off protocol or keep mapping in external registry so reconnects land on new owner.

Security & authentication

• Device certs (mutual TLS) for high security; or short‑lived JWT signed by provisioning CA. Embed private key securely in device (TPM/secure element where available).
• Per‑message authorization via topics and ACLs. Use OAuth2/JWT refresh flow for long‑lived sessions.
• Encrypt payloads (TLS) and consider payload‑level signatures for critical commands to validate device firmware.

Reliability & latency optimizations

• Use QoS1 for cloud→device commands; QoS2 for critical operations.
• Keep broker pools regional to minimize RTT; use replication/edge brokers for regional fan‑out.
• Prioritize small control messages; compact binary payloads to reduce parse time on embedded devices.

Device considerations

• Implement reconnect/backoff, persistent local pointer to last acked message, and minimal parsing stack (lightweight MQTT client).
• Use sparse wake/sleep scheduling: if deep‑sleeping, broker queues messages; use push wake token to reduce latency if supported.

Tradeoffs

• Keeping persistent sessions requires more server state; mitigated by external KV and sharding.
• mTLS is stronger but more complex provisioning vs JWT easier to rotate.

Result: predictable, low‑latency cloud→device delivery using long‑lived device‑initiated MQTT/TLS connections, consistent hashing for routing, distributed session store for scale and failover, and hardware-aware client implementations for constrained embedded devices.

Clarifying constraint: 32 KB RAM and limited flash — target: model + runtime + working buffers + stack must fit comfortably (reserve ~8–10 KB for OS/stack/RT tasks). Aim for model+activations < ~20 KB.

Model selection

• Choose an extremely small architecture: tiny dense autoencoder (for anomaly score) or a 1D-CNN with depthwise separable convs. Example: encoder 3 dense layers (64→32→8), decoder symmetric; inference only needs encoder+scoring.
• Train offline with dataset augmentation; minimize activation footprint (use smaller batch = 1, short sequence window).

Compression: quantization & pruning

• Post-training integer quantization to 8-bit (full integer or dynamic ranges) — reduces flash and RAM for weights.
• Perform magnitude-based structured pruning (remove neurons/channels) to keep memory alignment friendly.
• Optionally apply weight clustering then run Huffman/DEFLATE for flash storage; ensure runtime supports decompressed format or decompress at install.

Memory mapping & buffer management

• Place weights in flash (XIP) or in read-only section; map as const to avoid copying to RAM.
• Use single in-place activation buffer sized to peak layer activation. Compute peak analytically; reuse buffer across layers.
• Use double-buffering only if DMA sensor→RAM and inference overlapped; otherwise single buffer to save RAM.
• Reserve a small heap for dynamic needs and a guarded stack; use static allocation in C, avoid malloc in critical paths.

Inference runtime

• Use TensorFlow Lite for Microcontrollers or CMSIS-NN optimized kernels (ARM Cortex-M) — both support 8-bit and minimal overhead.
• If CPU lacks DSP, use integer-only kernels and hand-optimized C for core ops.
• Build without exceptions/RTTI, strip symbols, link-time optimize.

Periodic model updates

• A/B firmware partitions or small update partition in flash.
• Deliver model deltas (pruned/quantized) over OTA; validate signature and checksum, stage to update partition, run canary inference tests, then swap pointers/boot flags.
• Keep previous model for rollback on failure.

Fallback/cloud strategy

• Primary: local inference. If confidence score low/uncertain (near threshold) or resource/health flags (low memory, corrupted model), enqueue feature window to send to cloud.
• Implement backoff + batching for uplink to save power. Use secure channel (TLS) and compress features.
• Graceful degrade: if cloud unreachable, flag as “unknown” or use conservative policy (trigger safe action), log events locally for later upload.

Monitoring & safety

• Monitor memory usage, stack watermarks, inference latency. Fail-safe: watchdog reset on hang; model corruption triggers rollback.
• Test edge cases: distribution shift, sensor noise, and verify false-positive/negative trade-offs.

This concrete plan balances model size (8-bit + pruning), minimal RAM activations (in-place buffers), efficient runtime (TFLM/CMSIS-NN), safe OTA model swaps, and a clear cloud-fallback triggered by uncertainty or local failure.

Clarify requirements

• Sample analog sensor at 10 kHz (every 100 μs), run fixed‑point filtering, publish aggregated result every 10 ms (every 100 samples). MCU: no FPU, constrained CPU.

High‑level design

• Use ADC with timer trigger at 10 kHz.
• ADC writes samples to memory via peripheral DMA into a ping‑pong (double) buffer of 100 samples per half‑buffer.
• On DMA half/full transfer interrupt (or peripheral event), schedule deterministic processing.

Buffering & DMA

• Allocate two buffers: bufferA[100], bufferB[100] aligned, in DMA-capable RAM.
• DMA in circular mode: half transfer interrupt at 100 samples / or set transfer size = 100 and toggle buffers manually.
• While DMA fills bufferX, CPU processes bufferY — no sample copy, cache maintenance if present.

Filtering (fixed‑point)

• Use Q15 or Q31 depending dynamic range. Convert ADC uint16 -> signed Q15 centered around 0.
• Implement FIR/IIR in integer C with saturating arithmetic and consistent shift/scaling.
• Example scaling: coefficients in Q15, multiply -> 32-bit accumulator, shift down by 15, saturate to 16-bit.
• Precompute scaling/gain to avoid runtime divisions.

Timing & WCET

• Budget per 10 ms window: processing of 100 samples must finish <10 ms minus publish overhead.
• Measure WCET for per‑sample op or for whole buffer using cycle counters; add margin (e.g., 20%).
• Example: if filter of order 32 takes 40 μs per buffer, total processing 40 μs << 10 ms; worst-case includes interrupts and publish.

Scheduling & jitter bounding

• Use priority: ADC/DMA interrupts high priority but minimal work (clear flag, set event).
• Processing runs in a real-time task or main loop triggered by semaphore from ISR.
• Enforce deterministic execution: disable lower-priority interrupts during critical compute if needed; use fixed execution slots.
• Publish (UART/CAN) in separate lower-priority task; if slow, queue packets or DMA the transmit to avoid blocking.
• Bound jitter by worst-case response time: ISR latency + scheduling latency + WCET. Keep ISR short, use preemption ceiling to guarantee completion before next half-buffer (100 μs).

Validation & tests

• Instrument cycle counters, toggled GPIOs to measure timing and jitter.
• Stress-test with worst-case interrupts and publish path active.
• Verify numerical behavior with unit tests (fixed‑point vectors) and full-system timing tests.

This yields deterministic 10 kHz sampling, safe fixed‑point filtering, and guaranteed 10 ms publishing with bounded jitter.