IoT Systems Architecture and Design Questions

Covers architecture and design of distributed Internet of Things systems and connected embedded devices. Core topics include edge computing patterns, sensor and actuator network topologies, gateway and mesh network architectures, cloud integration and data pipelines, and trade offs between edge processing and cloud processing. Also includes networking protocols commonly used in constrained environments such as WiFi, Bluetooth, ZigBee, and LoRaWAN, plus connectivity strategies for unreliable networks including buffering, retries, offline operation, and data aggregation and filtering. Candidates may be evaluated on device to cloud data flow, scalability considerations from hundreds to millions of devices, performance and power trade offs on resource constrained hardware, deployment patterns for gateways and proxies, and high level fault tolerance and monitoring strategies.

HardTechnical

84 practiced

Design a memory-efficient algorithm and data structure for a gateway with limited RAM to deduplicate and order sensor events that may arrive out of order. Include expected memory bounds per device, eviction policy for old entries, how to checkpoint state to flash between reboots, and how to recover after a crash with minimal duplication to cloud consumers.

Sample Answer

**Approach (brief)**

Use a compact per-device sliding-window sequence-number bitmap stored in RAM, plus an ordered ring buffer of pointers/metadata for events to forward. Sequence numbers identify dupes; window bounds handle out-of-order. Persist minimal state to flash periodically to survive reboots.

**Data structures**

- Per device: - uint32 last_acked_seq; // highest contiguous seq forwarded - uint16 window_size; // e.g., 128 - uint128/bitset seen_bitmap; // window_size bits; bit i = seq (last_acked_seq + 1 + i) seen - circular buffer of event headers (index-only): struct { uint32 seq; uint16 payload_offset; uint16 payload_len; } entries[window_size]- Payload store: append-only flash area or temporary RAM pool (depending on size). For tiny devices keep payload in flash and store offset.

Memory bounds (example window_size = 128):- seen_bitmap = 16 bytes- last_acked_seq = 4 bytes- entries = 128 * 8 bytes = 1024 bytesTotal ≈ 1.04 KB per device + small overhead.

**Algorithm / Actions**

- On incoming event (device_id, seq): - If seq <= last_acked_seq: drop (already forwarded). - delta = seq - (last_acked_seq + 1) - If delta >= window_size: treat as jump — advance last_acked_seq to seq (optionally accept and clear bitmap) or reject depending on policy. - Else check bitmap bit delta: if set -> duplicate, drop; else set bit, store header+payload in ring buffer position (last_acked_seq + 1 + delta). - Attempt to forward in-order: while bitmap bit 0 set, forward payload for seq = last_acked_seq+1, clear bit and advance last_acked_seq, rotate bitmap right by 1 (efficient with two 64-bit ops).

**Eviction policy**

- Sliding-window implicit eviction when last_acked_seq advances: entries older than window are overwritten.- For long gaps or device inactivity: expire device state after TTL (e.g., 24h) and free RAM.

**Checkpointing to flash**

- Periodically (or on clean shutdown) write a compact checkpoint record per device: - { device_id, last_acked_seq, window_size, seen_bitmap, CRC32, monotonic counter }- Use atomic write: write new checkpoint slot with incremented counter; on completion mark valid. Employ wear-leveling and group commits to reduce flash wear.

**Recovery after crash**

- On boot, read latest valid checkpoint per device, reconstruct last_acked_seq and bitmap. Rehydrate any payloads referenced (payload storage must be GC-safe).- Resume dedupe: any events with seq <= last_acked_seq are dropped; events in bitmap already forwarded are dropped; in-flight events that were forwarded before checkpoint but not acknowledged to cloud could cause at-most-once vs at-least-once tradeoff. To minimize duplication: - On forward, tag cloud messages with device_id + seq so consumers dedupe server-side if needed. - Prefer checkpoint-on-forward confirmation or fast periodic checkpoints (e.g., after every N forwards) to bound duplicate window.

**Trade-offs & notes**

- Window size trades memory vs reorder tolerance.- Use bit operations for speed; implement bitmap as 64-bit words.- Ensure flash checkpoint atomicity and CRC to avoid corruption.- Keep messages idempotent at cloud consumer for absolute safety.

This design keeps per-device RAM ~1KB for 128-window, is implementable in C on microcontrollers, supports out-of-order arrival, efficient dedupe, bounded memory, and safe recovery with minimal duplicates.

HardTechnical

78 practiced

Design MAC-layer duty cycling and routing policies for a battery-powered ZigBee mesh that must support sub-second command latency for actuators while preserving multi-year battery life for sensors. Discuss wake-up synchronization, scheduled versus asynchronous MAC strategies, route discovery frequency, and trade-offs between routing table size and energy consumption.

Sample Answer

**Situation & goals (brief)** Design for ZigBee mesh where actuators need <1 s command latency; sensors must last years on battery. Primary levers: MAC duty cycle, wake synchronization, routing policy, and routing state size.

**Wake‑up synchronization** - Use a cluster of periodic beaconed wake windows: actuators and routers run a higher-duty coordinator schedule; sensors stay in deep-sleep and wake on an aligned short window. - Implement low-jitter, RTC-driven scheduling with occasional clock drift correction via beacons (sequence number + timestamp). Correct drift only when needed to minimize radio on-time.

**Scheduled vs asynchronous MAC** - Scheduled (beacon-enabled ZigBee / TSCH-like): deterministic sub‑second latency with predictable on-times. Best for actuators/routers. - Asynchronous (LPL / low power listening): ultra-low average power for sensors but higher and variable latency. Use hybrid: nodes that must receive commands (actuators, parents) use scheduled slots; leaf sensors use asynchronous transmit-to-parent within the parent’s scheduled receive window.

**Route discovery frequency** - Minimize broadcasts: use stable parent-based routing with long route TTLs. Run proactive route refresh on topology change or periodic (e.g., every few hours) for battery sensors. For actuators/routers, permit more frequent updates. Use piggybacked route info in scheduled beacons.

**Routing table size vs energy trade-offs** - Large tables (full mesh) reduce route discovery broadcasts but increase memory and processing (bad for small MCUs). - Small tables (parent-only or default routes) reduce RAM but increase on-demand discovery energy and latency. - Hybrid: maintain next‑hop entries for critical peers (actuators, gateway) and use compressed route caches for occasional nodes. Use route lifetime and LRU eviction tuned to traffic patterns.

**Practical parameters (example)** - Actuator/routers: beacon period 250–500 ms, awake 50–100 ms → <1 s RTT. - Sensors: wake every 5–30 s, transmit quickly to parent’s scheduled RX, sleep; adjust based on event rate.

**Conclusion / trade-offs** Tune parent selection, beacon frequency, and route cache sizes by profiling traffic: prioritize scheduled slots for low-latency paths and asynchronous for deep-sleep sensors to meet both latency and multi-year battery goals.

MediumTechnical

87 practiced

Design algorithms to aggregate and down-sample high-frequency sensor streams on a gateway before sending to the cloud. Discuss lossy and lossless approaches, sliding and tumbling window strategies, sketching or reservoir sampling techniques, compression options, and how to ensure anomalies are preserved for downstream analytics.

Sample Answer

**Approach overview**I’d implement a flexible gateway pipeline that supports both lossless and lossy modes, with lightweight C/C++ modules optimized for limited RAM/CPU and low power. Use ring buffers + interrupts/DMA for ingestion, then windowing or sketch/reservoir modules to produce packets for cloud.

**Lossless options**- Delta encoding + binary packing + LZ4/DEFLATE before transmit to preserve every sample.- Store in on-flash circular log if connectivity delayed.- Trade-off: higher bandwidth and CPU for compression; use streaming compressors (LZ4) to limit memory.

**Lossy options**- Aggregation per window: count, mean, min, max, stddev, last-timestamped sample. Reduces bytes; good for trends.- Downsample by selecting every Nth sample or decimation with anti-alias filter on-device (IIR/FIR).

**Windowing strategies**- Tumbling windows: fixed-length, simple to implement with fixed buffers.- Sliding windows: overlap for smoother stats; implement with incremental updates (keep running sum, sumsq) to avoid O(window) cost.

**Sketching & reservoir**- Use reservoir sampling (k-sample) for uniform subset when preserving random samples is needed; O(1) per sample, O(k) memory.- Use Count-Min or TDigest-like sketches for distribution summaries (quantiles) with bounded memory—helpful for telemetry without raw data.

**Compression & binary formats**- Use compact binary frames (timestamps delta-encoded, 16-bit fixed-point) + LZ4.- If many zeros, use RLE or protobuf with packed fields.

**Anomaly preservation**- Run lightweight anomaly detectors on-gateway: threshold checks, EWMA residual, and variance spikes.- If anomaly detected: 1) tag aggregated packet with anomaly flag + pre/post context, 2) switch to lossless capture for a short buffer window and transmit raw samples, 3) raise high-priority telemetry.- Maintain short circular raw-buffer (e.g., last 5–30s) so anomalies can be shipped with context.

**Implementation notes (embedded)**- Use fixed-point math to avoid FP costs; keep buffers in statically allocated ring arrays.- Use DMA/ISRs for sensor reads, and a low-priority worker thread for aggregation/compression.- Expose config via remote commands (change window size, mode) to balance power/bandwidth.

**Example trade-offs**- Tumbling + mean : simplest, smallest CPU.- Sliding + TDigest: better quantiles but more memory/CPU.- Reservoir + anomaly flag: good when you need representative samples + rare full-resolution capture.

This design preserves anomalies, bounds resource use, and provides configurable fidelity for downstream analytics.

HardSystem Design

64 practiced

Architect an OTA update system for 5 million IoT devices that supports staged rollouts, fast rollback, integrity verification, and fleet-level analytics for canary analysis. Describe package creation, use of CDN or regional caches, delta versus full images, cohort management, telemetry needed to detect regressions, and safety mechanisms to stop a harmful rollout.

Sample Answer

**Clarify requirements & constraints**- 5M devices, intermittent networks, limited flash/ram, A/B partitions preferred, secure boot/TPM on some devices, goal: staged rollouts + automatic rollback + fleet telemetry for canary analysis.

**Package creation**- Build artifacts: full signed image + optional delta patch (bsdiff/ xdelta) generated from each supported baseline. Include metadata: version, baseline hashes, size, required free space, dependencies, signature (Ed25519) and manifest JSON.- Use atomic A/B update with verified swap + bootloader health checks and rollback marker.

**Delta vs full**- Prefer deltas for bandwidth/cost savings when base matches; fall back to full if base mismatch or corruption. Keep server-side logic to select best candidate.

**Distribution: CDN & regional caches**- Origin stores images and manifests; push to multi-region CDN + regional edge caches (or peer caches for constrained networks). Use signed URLs + cache-control. Devices query nearest/region endpoint with device cohort header for staged rollout.

**Cohort & staged rollout**- Cohorts by hardware rev, SKU, fleet tag, risk score. Release plan: canary cohort (0.1%), small rollout (1%), ramp (10%, 50%) with time windows and automatic gating.

**Telemetry for canary analysis**- Telemetry: boot success, boot time, app health, watchdog resets, memory/CPU spikes, sensor anomalies, error logs (sampled), battery impact, network failure rate. Include anonymized device id, firmware version, cohort tag, timestamps.- Aggregate to compute KPIs: crash rate, mean time between reboots (MTBR), error per device, CPU/memory percentiles. Use rolling windows and Bayesian change detection to flag regressions.

**Safety mechanisms**- Client-side: pre-update checks (battery, storage), secure verification, staged apply (download->verify->install->reboot->health-check window), watchdog-enforced rollback to last known-good.- Server-side: automated canary analyzer — if KPIs exceed thresholds or statistical significance detected, trigger pause/rollback for cohorts, revoke manifest via signed “kill-switch” with high-priority push to devices.- Manual override: ops console to abort, push hotfix, and force rollback cohorts.

**Observability & analytics**- Real-time dashboard, cohort comparisons, anomaly alerts, drill-down to device-level traces. Store raw telemetry in time-series DB and events in data lake for forensic analysis.

This design balances safety, bandwidth, and embedded constraints while enabling fast rollback and data-driven rollout decisions.

MediumTechnical

92 practiced

Compare TLS, DTLS, and OSCORE for securing communications of constrained IoT devices. Evaluate handshake overhead, transport suitability (TCP vs UDP), ability to traverse proxies and NATs, memory and CPU cost, and suggest key exchange and session resumption strategies suitable for devices with intermittent connectivity and limited RAM.

Sample Answer

**High-level summary**As an embedded developer, I pick the smallest tool that meets protocol needs: TLS/DTLS provide transport-layer security; OSCORE provides application-layer (CoAP) end-to-end security optimized for constrained nodes. Choice depends on transport (TCP vs UDP), intermediary proxies, and RAM/CPU budgets.

**Handshake overhead**- TLS 1.3: fewer round trips than TLS 1.2 but still heavier (TCP + TLS handshake). Good for reliable connections, higher CPU/memory.- DTLS 1.3: TLS-like crypto over UDP with comparable handshake cost but added retransmission/ordering logic. Better for latency but slightly more complex state machine.- OSCORE: no transport handshake — per-message AEAD using pre-established OSCORE contexts. Minimal runtime handshake cost; setup cost is small (derive a context).

**Transport suitability**- TLS → TCP (HTTP, MQTT over TLS). Use when you need stream semantics.- DTLS → UDP (CoAP over UDP, low-latency). Use for constrained networks or multicast.- OSCORE → Transport-agnostic but designed for CoAP (works over UDP, TCP, SMS); secures payload end-to-end even through CoAP proxies.

**Proxies and NAT traversal**- TLS/DTLS terminate at intermediaries if they act as proxies and need access to plaintext. DTLS/TLS both can be proxied via TLS-terminating proxies; MTU/NAT issues for DTLS/GCM fragmentation.- OSCORE preserves end-to-end confidentiality across CoAP proxies and NATs because proxies only see outer CoAP envelope.

**Memory and CPU cost**- TLS/DTLS: larger RAM (session buffers, certificate stacks) and CPU for handshake (ECDHE). Typical embedded stacks can be tens-hundreds of KB RAM; CPU cost significant without crypto accel.- OSCORE: small RAM footprint (context + replay window + AEAD buffers), low CPU (single AEAD per message). Best for very constrained devices.

**Key exchange & session resumption strategies for constrained/intermittent devices**- Prefer PSK or raw public keys (RPK) over full X.509 where possible to reduce memory and parsing cost.- Use ECDHE (P-256 or X25519) when forward secrecy is required; choose curves supported by hardware.- Use TLS 1.3 / DTLS 1.3 with PSK resumption (pre-shared session tickets) or exported keying material to re-establish quickly.- For DTLS over lossy links, use session tickets stored in flash; rehydrate session state on wake to avoid full handshake.- For OSCORE, provision a master secret and derive new contexts with sequence numbers; store minimal state (current replay window, sender sequence) in NVRAM and refresh on reconnect.- Use 0-RTT cautiously (replay risk); prefer PSK-based 1-RTT resumption if replay protection is a concern.- Always offload crypto to hardware AES/ ECC when available and limit supported ciphers to AEADs like AES-CCM or ChaCha20-Poly1305.

**Practical recommendation**- For CoAP devices behind proxies: OSCORE for end-to-end and minimal runtime cost.- For MQTT or HTTP on constrained devices: DTLS (UDP) if you need low latency; otherwise TLS over TCP with PSK + session tickets and hardware crypto.- Implement session ticket/PSK storage in flash, limit cipher suites, and prefer AEAD + ECC curves matching hardware to minimize RAM/CPU and speed reconnects.

Unlock Full Question Bank

Get access to hundreds of IoT Systems Architecture and Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.