Real-Time and Streaming System Design Questions

Approach summary (why it matters)
I’d design layered backpressure combining pull-based control, bounded buffering, shedding, and feedback loops so the system degrades predictably rather than collapsing.

Compare strategies

• Pull-based flow control — consumers poll at capacity; best for tight end-to-end control and avoiding overload but requires producer cooperation (e.g., TCP-like windowing or HTTP long-poll with rate hints).
• Bounded queue buffering — short-term smoothing using fixed-size queues (per-producer, per-tenant). Simple but finite; must be combined with eviction or throttling.
• Shed-load policies — drop or downsample low-value telemetry when overloaded. Effective to preserve core signals.
• Feedback loops — control-plane signals (rate-limit tokens, backoff hints) + adaptive algorithms (additive increase/multiplicative decrease) for dynamic equilibrium.

Concrete policies I’d implement

• Throttling: token-bucket per-producer with global admission token pool. If bucket empty, return 429 with Retry-After or pause notifications. Token refill uses measured consumer throughput.
• Prioritization: classify telemetry (critical, standard, debug). Always accept critical. For standard, allow probabilistic sampling under load; debug only if capacity > 80%. Use weighted queues and priority scheduling.
• Circuit-breaker: track per-consumer processing latency and error rate; open when latency > 95th percentile threshold and error rate > X% for Y seconds. In open state, redirect or rate-limit producers aggressively; half-open with probe requests.

Feedback & control loop

• Use control loop every T seconds: measure consumer throughput, queue depths, drop rates → adjust token refill rates and sampling probabilities via AIMD. Emit backpressure hints via HTTP headers or a control channel (gRPC stream).

Key metrics to monitor

• Ingest rate (events/sec) per producer/tenant
• Consumer processing throughput and concurrency
• Queue depths and tail latencies (p50/p95/p99)
• Drop/shed counts and reasons
• Token-bucket exhaustion events and 429 rates
• Circuit-breaker states and recovery attempts
• Error rates and processing timeouts

Operational notes / trade-offs

• Prefer cooperative pull where possible; otherwise rely on short bounded buffers + clear shedding policies.
• Tune thresholds with canary traffic and simulate burst scenarios.
• Ensure observability and replay paths for shed telemetry (e.g., sample logs) so critical business signals aren’t lost.

Clarify requirements & constraints

• Target latency: microseconds end-to-end for matching; throughput: millions of orders/sec; durability: no loss of trades; geo-redundancy optional; regulatory audit trail required.

High-level architecture

• Kernel-bypass NICs (RDMA/iWARP or DPDK) → FPGA-based pre-filtering (optional) → ultra-low-latency matching engine on locked CPU cores (kernel bypass, busy-poll) → in-memory order book (NUMA-aware) → fast durable sink (NVMe NVM with battery-backed write cache) → async replication to hot DR site.

Core components & responsibilities

• Network: deterministic switches, PTP time sync, separate fabric for market data.
• Compute: bare-metal, real-time tuned Linux, hugepages, CPU affinity.
• Storage: write-ahead log to NVMe RAID for microsecond ack; periodic snapshots.
• Replication: state-machine snapshots + command logs to secondary; warm standby prepared to take over within SLA.

Trade-offs

• Correctness vs latency: sync replication increases latency — use async replicate + commit-to-local durable log to keep ack latency low; accept very short RPO and controlled failover to preserve correctness.
• Hardware cost vs determinism: FPGAs and RDMA increase cost but reduce jitter.

Failure handling

• Fast failover using health heartbeats and orchestrated state transfer; automatic replay of WAL on failover; isolation of noisy neighbors; circuit breakers.

SLAs & measurement

• Metrics: p99/p999 latency, jitter, throughput, successful trades, failover RTO/RPO.
• Tools: hardware timestamping, eBPF probes, dedicated observability pipeline with Prometheus + Grafana + alerting.
• Guarantees: SLOs backed by test harnesses (replay, chaos tests, soak), capacity headroom (50%) and documented runbooks.

Clarify requirements & constraints

• 10M concurrent, 200M/day, peak 100k msg/s, global low-latency (<200ms RTT goal), durable offline delivery, strict ordering where required, multi-DC.

High-level architecture

• Edge Gateways (global PoPs) ←→ Routing/Session Layer ←→ Messaging Backbone (pub/sub) ←→ Storage (hot message store + cold archive) ←→ Presence Service ←→ API/Push/CDC for mobile.

Protocols

• Primary: WebSocket for real-time bi-directional (low latency, backpressure via TCP). HTTP/2 gRPC as fallback for mobile or firewalled networks and for control plane.

Components

• Edge Gateways: TLS termination, auth, rate-limiting, local fanout for nearby recipients.
• Session Router: Maps user session → shard; lightweight consistent-hash + rendezvous hashing.
• Presence Service: Eventually-consistent in-memory (Redis cluster per region) with CRDT-ish last-write-wins + heartbeat TTLs.
• Messaging Backbone: Kafka or Pulsar topics per shard for durability + retention; use partition keys: conversation-id for group/1:1 affinity.
• Delivery Workers: Consume topic, enforce ordering per partition, attempt in-region fanout, fallback cross-region relay.
• Offline Persistence: Hot store (regional Redis + per-user unread queue) + cold in S3 via Kafka Connect.

Partitioning & sharding

• Shard by conversation-id for ordering and locality; assign partitions across Kafka clusters sized to handle 100k msg/s (e.g., 1000 partitions); user-session metadata sharded by user-id for presence routing.

Ordering guarantees

• Strong ordering per conversation: single partition per conversation => single consumer ordering. For very large groups, use sub-sharding (conversation + sequence window) and per-recipient sequence reconciliation with causal vector for conflict detection.
• At-least-once delivery by default; implement deduplication via message-id and idempotent consumer writes to hot store for exactly-once semantics at application layer.

Backpressure & flow control

• Use TCP backpressure; application-level per-connection queue limits and drop/slow strategies. Gateways apply token-bucket rate limits and per-user QoS.

Network partitions & reconnection

• Client reconnect: exponential backoff + sticky session affinity to nearest PoP; on reconnect, client sends last-received message-id per conversation; server uses per-user unread queue to replay delta.
• During inter-DC partition: continue local writes; mark messages with region-timestamp and replicate via async multi-master replication (Kafka MirrorMaker/Pulsar GeoReplication). Resolve conflicts by sequence + last-writer or application merge.
• Monitor with health checks and automated failover of Session Router; degrade to read-only if global consensus needed.

Operational concerns

• Metrics: p99 latency, message loss, consumer lag, reconnection rate.
• Chaos testing for partitions, scale tests for 100k msg/s, and automated on-call runbooks.
• Security: end-to-end encryption optional; TLS, JWT tokens, rate-limits, DDoS protection.

This design provides per-conversation strong ordering, durable storage for offline delivery, WebSocket low-latency delivery with HTTP/2 fallback, and pragmatic partitioning to scale to the specified load while handling partitions and reconnections.