Clarify goals & constraints
- Reliable ingestion from millions of intermittently-connected devices; tolerate offline for hours; bounded device storage; preserve causal ordering per device where required; avoid bursting backend on reconnect.
High-level architecture
- Device agent → Local store-and-forward queue → Gateway (optional) → Ingress API / message broker (e.g., MQTT/AMQP) → Ingestion service → Time-series DB / object storage + processing pipeline.
Store-and-forward on device
- Persistent, append-only log on device (circular with configurable retention).
- Each record: device_id, monotonic sequence number (per-device), wall-clock timestamp, payload, optional causal/transaction id.
- Retry policy: exponential backoff; batch transmissions when online to amortize overhead.
Ordering guarantees
- Provide per-device strict ordering using sequence numbers. Ingress accepts batches with sequence ranges; ingestion validates sequence continuity and stores out-of-order temporarily (small buffer) or requests missing ranges.
- For causal groups spanning devices, use logical timestamps (vector clocks) only where necessary—otherwise avoid global ordering to scale.
Bandwidth & backpressure
- Compress and batch payloads; send diffs for telemetry where possible.
- Device-side rate limiter and adaptive batch size based on current network quality.
- Ingress exposes token-bucket quotas per-device or per-gateway. On reconnect, gateway negotiates a maximum burst (leaky-bucket) to smooth load.
Avoiding backend overload on reconnect
- Reconnect handshake: device/gateway sends annoncement with backlog size and max allowed rate; server responds with an allowed ingest rate and window.
- Server queues large backlogs into durable staging (S3 / Kafka) with rate-limited consumers to process steady-state.
- Prioritization: recent/high-priority measurements first; older low-value data can be downsampled or dropped based on retention policy.
Data retention & trade-offs
- Short device storage → smaller backlog but higher chance of data loss. Longer retention increases device resource need.
- On server: hot storage for recent data (hours-days) in TSDB, cold object store for raw telemetry. Apply TTLs, downsampling, and schema for aggregated summaries.
Eventual consistency
- Use idempotent writes with per-device sequence numbers; reconciliation job replays missing segments from durable staging.
- Expose eventual-read semantics to consumers; provide last-known-state endpoints for quick UI reads.
- Monitoring & alerts for sequence gaps and backlog growth.
Operational considerations
- Instrument metrics: backlog sizes per device/gateway, replays per hour, ingest latency.
- Security: mutual TLS, signed payloads to prevent replay attacks.
- Failure modes & testing: simulate large reconnect storms; circuit-breakers on consumers; autoscale ingestion.
This design balances durability, ordering, bandwidth efficiency, and protects the backend via negotiated rate-limits, staging queues, and prioritized ingestion to achieve eventual consistency without overload.
