Real-Time and Streaming System Design Questions

Requirements:

• High throughput (hundreds of thousands–millions events/sec), low latency, fault-tolerant, no data loss, independently scalable stages: ingest → parse → enrich → index.

High-level architecture:

• Staged pipeline with durable queues between stages. Each stage runs as many worker instances as needed and consumes/produces to the next queue.

Queue choices:

• Ingest→Parse: durable, distributed persistent queue (Kafka / Pulsar) to absorb spikes and provide retention/replication.
• Parse→Enrich: persistent with configurable retention (Kafka topics / Pulsar); can use tiered storage for cost.
• Enrich→Index: persistent or fast-durable (Kafka + local SSD caching) before indexing sink.
• In-memory queues (bounded in-process buffers) only inside a stage for batching/low-latency handoff; never as the sole copy of unprocessed events.

Backpressure handling:

• Use reactive pull model: consumers poll at consumer-controlled rate. If downstream queue partition lag grows beyond threshold:
  • Slow ingestion by signaling upstream (HTTP 429, token bucket throttling, or pause Kafka producers).
  • Apply adaptive sampling or priority routing for critical events.
  • Autoscale downstream workers based on lag metrics (Kafka consumer lag, queue length, processing latency) before throttling.

Delivery semantics:

• Default: at-least-once using durable queues + idempotent downstream operations.
• Exactly-once where required: use Kafka transactional writes or deduplication keys + idempotent sinks (upserts with unique event IDs) and idempotent enrichment (store processed offsets).
• Design idempotency: assign event IDs at ingest; enrich/index operations must be idempotent (upsert, compare-and-swap) or store processed event IDs in a compact bloom filter/TTL store.

Scaling stages independently:

• Partitioning: use key-based partitioning where logical keys map across topics so related events go to same partition for ordering.
• Autoscaling: metric-driven (CPU, processing latency, consumer lag). Scale horizontally by adding consumers; rebalance partitions.
• Batching: tune batch sizes per stage to maximize throughput while keeping latency bounds.

Failure recovery and no data loss:

• Durable queues with replication and retention ensure data survive node failures.
• Consumer offset management: commit offsets only after successful processing and downstream ack (two-phase commit pattern if needed). For at-least-once, commit post-write to next queue.
• Checkpointing: workers snapshot in-flight progress and retry on restart. Use transactional writes to avoid partial writes.
• Partial failure: if enrichers fail after reading but before writing, on restart they re-read from offset (with at-least-once) and perform idempotent reprocessing. For exactly-once with transactions, incomplete transactions are aborted and retried.
• Monitoring & alerting: track consumer lag, end-to-end latency, processing errors. Automated retries with exponential backoff, dead-letter queue for poison messages.

Trade-offs:

• Exactly-once adds complexity and throughput cost; use only where business-critical.
• Larger retention and replication increases durability but storage/cost.
• Partitioning improves scale but complicates cross-key joins (may need side-store or saga).

This design balances throughput, durability, independent scaling, and recoverability by relying on persistent distributed queues, idempotency, careful offset management, autoscaling, and backpressure controls.

Requirements & constraints:

• Ingest 1M events/sec, E2E processing <500ms, at-least-once or exactly-once semantics (prefer exactly-once for correctness), enrich and materialize results to low-latency store.

High-level architecture:

• Ingestion: Kafka (or Pulsar) fronting producers. Use >200 partitions to parallelize (partition count tuned to throughput & consumer parallelism).
• Buffering: Kafka serves durable buffer and backpressure decoupling.
• Stream processing: Apache Flink (low-latency, mature checkpointing & exactly-once). Deploy with task managers sized for CPU and network throughput.
• State storage: RocksDB state backend (local), with remote durable snapshots to S3/GCS for checkpoints. Keyed state sharded by Kafka partition -> Flink parallelism.
• Materialization: Write enriched events to a low-latency sink (Cassandra/Scylla for wide writes, or Redis/KeyDB for read-heavy materialized views) via idempotent/batched writes.

Data flow:
Producers -> Kafka (partitioned) -> Flink Source (Kafka consumer per subtask) -> map/enrich (external lookup via cached async I/O) -> keyed aggregates/state updates (RocksDB) -> sinks (batched writes)

Latency design decisions:

• Keep per-record processing light: use async enrichment with an in-process LRU cache (e.g., Caffeine) + batch remote lookups. Use Flink async I/O to avoid blocking operators.
• Target per-stage latency budget: Kafka read ~5-20ms, processing ~100-200ms including enrichment, sink write ~50-100ms, plus checkpoint overhead.

Checkpointing & fault recovery:

• Use Flink’s periodic distributed snapshots (e.g., every 2–5s) with incremental checkpoints to S3. Configure minimal checkpoint pause and short timeout (<30s).
• Enable exactly-once sinks where possible (two-phase commit for external stores) or use idempotent writes (upserts with deterministic keys).
• On failure, Flink restores state from latest successful checkpoint + Kafka offsets are restored atomically. Keep retention of checkpoints and Kafka offsets consistent.

Scaling & resource sizing:

• Estimate event size and CPU cost. Example: 1M eps * 200 bytes ~200MB/s ingress. Provide network and disks to sustain ~2–3x headroom.
• Parallelism: set Flink parallelism >= Kafka partition count. Right-size TaskManagers: many cores and enough RAM for RocksDB memtables.

Primary bottlenecks & mitigations:

• Network I/O: use rack-aware placement, NICs 25G+, co-locate compute and state where possible; employ batching and compression.
• State size and RocksDB I/O: tune compaction, use NVMe SSDs, increase memory for block cache, use incremental checkpoints to reduce snapshot time.
• External enrichment DB latency: use local caches, pre-warm, use replicated read-only stores, or move enrichment data into Flink state for ultra-low latency.
• Checkpoint overhead: incremental checkpoints, increase checkpoint interval, asynchronous snapshots, tune checkpointing threads and RocksDB memory.
• Kafka consumer throughput: increase partitions, tune producer acks and batching, tune fetch sizes and consumer prefetch.
• Backpressure propagation: monitor and autoscale sources/sinks; add buffering layers (Kafka topic TTL) to smooth spikes.

Observability & testing:

• End-to-end metrics (ingest rate, processing latency percentiles, checkpoint duration, backpressure), distributed tracing for latency breakdown, chaos tests and failover drills.

Trade-offs:

• Exactly-once adds complexity and latency vs at-least-once with idempotent sinks; choose based on correctness needs. NVMe + more partitions reduce latency but raise operational cost.

This design yields horizontal scale to 1M eps with <500ms E2E if resources, partitioning, caching, and checkpoint tuning are applied and external lookups are bounded by cache or replicated stores.

Requirements & constraints:

• Real-time analytics for recommendation feature: low-latency (sub-second to seconds) user-level metrics, aggregated trends, anomaly detection, ability to backfill historical data, high throughput (millions/day), strong data quality and observability.

Architecture choice:

• Streaming-first approach (Kafka + stream-processing) because we need near-real-time metrics, alerting, and feedback loops for model/product tuning. Micro-batch (e.g., Spark structured streaming with larger windows) only for heavy historical backfills and nightly aggregates.

High-level components:

1. Ingestion

• Event producers (app servers, CDN edge) emit structured events (JSON/Avro) to Kafka topics per domain (impressions, clicks, recommendations_shown, user_actions). Use schema registry (Avro/Protobuf) for contracts and evolution.

2. Stream processing

• Real-time processing in Flink (or Kafka Streams) for enrichment (join with user/session metadata from Redis/feature store), deduplication, windowed aggregations, and feature counters. Output both aggregated metrics and per-event enriched streams.

3. Storage

• Hot: time-series store (ClickHouse/ClickHouse Cloud or Druid) for fast analytical queries and dashboards.
• Cold: object store (S3) with parquet partitions for historical backfill and ML training.
• Feature store for model inputs (Feast or internal).

4. Monitoring & Observability

• Metrics: Prometheus + Grafana for pipeline lag, throughput, error rates.
• Tracing: OpenTelemetry for end-to-end latency.
• Logging: centralized ELK/Datadog for errors.
• Data quality dashboards: Great Expectations / custom validators publishing metrics.

5. Data Validation

• Inline (stream) validators: schema conformance, rate/sanity checks, spike detection; sink bad rows to dead-letter Kafka topic and S3.
• Downstream sampling: periodic row-level checks comparing counts vs authoritative sources; automated alerts on threshold breaches.

6. Ownership model

• Platform team owns Kafka, processing infra, monitoring. Product/feature team owns event schemas, enrichment logic, SLAs for correctness. Data engineering owns storage schemas, backfill tooling, and data quality policies. Clear SLAs and runbook for incidents.

Rollout strategy

• Start with shadow mode: send events to a mirror topic and run processing without affecting production. Validate results vs batch baseline (daily aggregates).
• Phased rollout: 1) internal dashboards, 2) beta cohort (1% users), 3) progressive ramp (10%, 50%, 100%) with automated rollback on quality/latency alerts.

Backfill approach

• Use batch job (Spark) to reprocess historical events from raw S3 or archival Kafka into the same downstream stores with idempotency keys. Maintain a "processing window" and mark datasets with versions. For large backfills, throttle to avoid cluster starvation and run during low-traffic windows.

Measuring data quality & business impact

• Data quality metrics:
  • Event completeness (% events received vs expected per user/session)
  • Schema conformance rate
  • Duplicate rate
  • End-to-end processing lag (P99)
  • Drift detection (feature distribution changes)
  • Alert thresholds and SLOs (e.g., <1% invalid events; P95 lag <2s)
• Business impact metrics:
  • Recommendation CTR, conversion, retention lift (A/B test)
  • Time-to-first-action after recommendation
  • Revenue per user segment
  • Model feedback loop metrics: model input freshness, feature availability rate
• Tie analytics to experiments: run A/B tests and attribute KPI deltas to ensure the analytics reflect product impact.

Trade-offs & rationale

• Streaming adds complexity but meets latency and alerting needs. Using schema registry and dead-letter topics balances strictness with robustness. ClickHouse provides low-latency analytics; S3/parquet supports cost-effective long-term storage and reproducible backfills.

Operational considerations

• Implement automated canaries that compare streaming vs batch aggregates.
• Enforce cost controls and retention policies.
• Document runbooks and run quarterly DR/backfill rehearsals.

This design ensures reliable, observable real-time analytics with clear ownership, safe rollout, and measurable business outcomes.

Requirements & constraints

• 100M DAU, peaks 10k messages/sec, global multi-region, <150ms P95 message delivery within region, <300ms cross-region target, strong per-conversation ordering, offline delivery, abuse detection, compliance.

High-level architecture

• Clients ↔ Edge Gateways (regionally deployed, TLS, WebSocket/quic) ↔ Frontend services (auth, presence) ↔ Message Router / Sequencer ↔ Durable Storage & Replication ↔ Delivery Workers ↔ Moderation Pipeline

Core components

1. Edge Gateways & Protocol

• Maintain long-lived connections (WebSocket/QUIC) and do auth, rate-limiting, lightweight anti-DDoS.
• Sticky edge routing using Anycast + geo-DNS to minimize latency.

2. Presence

• Use an in-memory, region-local presence store (Redis Cluster / Aerospike) with TTL heartbeats from clients.
• Publish presence changes via regional pub/sub (Redis Streams / Kafka) and selectively replicate presence metadata cross-region (eventual) for friends in other regions.
• For scale: sharded by user ID; each instance handles ~100k concurrent connections.

3. Message routing & partitioning

• Partition by conversation ID (hash) into N partitions. Each partition has a primary sequencer (leader) colocated in a region owning that shard.
• Client sends message to nearest edge → frontends forward to partition leader via gRPC. Leaders assign monotonic increasing sequence numbers per conversation.

4. Ordering guarantees

• Strong ordering per conversation via single-writer sequencer (leader). Sequencer writes ordered message to an append-only log (partition-level Kafka topic or custom log) before ack.
• Consumers (delivery workers) read in order; clients apply sequence numbers to ensure ordering and idempotency.

5. Offline delivery & reconciliation

• Messages persisted in cold/nearline storage (S3 + index DB) and in hot read-store (Cassandra or Dynamo-style) for recent messages.
• On reconnect, client provides last-received sequence; server streams missing messages. Also support dedup by client message-id.
• Background reconciliation: client periodically verifies with server for gaps.

6. Multi-region replication

• Cross-region replication uses async replication of partition logs. For low-latency local writes, leader is regional; replicate to other regions with causal metadata (vector clocks or per-partition offsets) to preserve ordering.
• For geo-failover: use consensus (Raft) per partition to elect leader among replicas; prefer local leader for latency unless failover occurs.

7. Moderation & abuse controls

• Inline lightweight checks at frontends (rate, banned words hashes, ML model quick checks).
• All messages stream into moderation pipeline (Kafka → real-time ML scoring + human review queue).
• For severe violations, sequencer can mark message as tombstoned; workers can redact and send delete/update deltas to clients.
• Audit logs retained in write-once storage for compliance.

8. Latency & scalability targets

• P95 local: <150ms from client send to deliver to recipients in same region (edge + sequencer + write + pub/sub + delivery fanout).
• Cross-region: additional replication + delivery within <300ms for typical paths.
• Use connection fanout via hierarchical delivery workers to avoid N-to-N push explosion.

9. Capacity planning (back-of-envelope)

• Peak 10k msg/s → allocate partitions: choose partition count so each partition handles ≤200 msg/s → 50 partitions for logical growth per region; multiply by redundancy factor (3 replicas) => 150 partition replicas.
• Storage: avg message size 1KB. 10k msg/s → ~864M messages/day ≈ 0.86 TB/day raw. Plan for 30-day hot storage → 25.8 TB plus replication factor 3 → ~78 TB. Archive older to cold storage.
• Connections: 100M DAU, assume 10% concurrent → 10M concurrent connections. Gateways scale such that each instance handles ~20k connections → 500 gateway instances globally.

Trade-offs & alternatives

• Single-writer sequencer gives per-conversation strong ordering; trade-off is leader hotspot for very active rooms — mitigate with sharding conversation or partitioning large groups into sub-shards.
• Async cross-region replication favors availability & latency; for stricter global ordering use global consensus (higher latency).

Operational concerns

• Monitoring: per-partition lag, replication lag, connection metrics, moderation queue size.
• Autoscaling: gateway pools, delivery workers, sequencer pods.
• Chaos testing for region failover and reconciliation correctness.

This design provides strong per-conversation ordering, low-latency regional delivery, robust offline reconciliation, and scalable multi-region replication while giving clear levers for capacity planning and moderation.

Requirements & assumptions

• Functional: threaded comments anchored to document ranges (character offsets / DOM ranges), create/edit/delete replies, resolve/unresolve, mention users, notifications.
• Non-functional: support real-time collaboration for multiple editors (sub-100ms UX), offline edits with sync/merge, per-comment access controls (read/write/owner), audit/history of edits, retention policy.
• Assumptions: documents are mostly plain-text or rich-text with deterministic model; clients have stable identities; we can add lightweight metadata to document model.

Design A — Real-time with CRDTs (client-driven)

• High-level: represent document and comment-anchor positions using CRDTs (e.g., RGA/WOOT for sequence). Comments stored as separate CRDT objects with references to anchor IDs; anchors move with concurrent edits.
• Pros: strong eventual consistency, no central OT server, easier offline-first support.
• Cons: larger metadata overhead, complex tombstone/compaction, anchoring semantics tricky for rich ranges.
• Implementation notes: store CRDT state-deltas in append-only log; use vector clocks for causal merging; server acts as relay + persistent store.

Design B — Real-time with OT (server-mediated)

• High-level: use OT on the document server to transform operations; comment anchors maintained as ranges transformed by same OT transforms. Server is authoritative, clients send ops, server broadcasts transformed ops.
• Pros: lower metadata, mature for text editors, predictable anchor behavior if transforms applied consistently.
• Cons: server becomes coordination bottleneck, more challenging offline support, complex transform correctness with rich-objects.
• Implementation notes: persist ops for replay, version vectors to detect divergence.

Offline-first comments

• Local queue of comment operations (create/edit/delete/resolve) with timestamps and client IDs. On reconnect:
  • For CRDT: merge ops naturally.
  • For OT: replay ops against latest server state, resolve conflicts with last-writer-wins or merge UI for complex edits.
• Anchors: store both anchor-ID and fallback (text-snippet/context) to re-resolve if original anchor lost.

Consistency / latency / usability trade-offs

• CRDT: better offline UX and availability; eventual consistency may show temporary divergent views; higher bandwidth/storage.
• OT: lower divergence, lower metadata, lower latency when connected; weaker offline UX.
• UX choices: show inline "syncing" indicators, conflict-resolution UI for overlapping comment edits, preserve user intent (who authored and timestamps).

Access control & history

• Enforce RBAC at server: comment-level ACLs stored with comment objects; server filters responses accordingly.
• History: immutable audit log of operations (opstore) enabling undo, version diffing, and legal export.

Communication to Product & QA

• Product: present wireframes showing anchor behaviors (stickiness, re-anchor when text moves), user flows for offline creation and conflict resolution, SLAs (latency goals, retention). Discuss trade-offs and recommend CRDT if offline-first is priority, OT if minimal metadata and strict ordering matters.
• QA: provide test matrix: concurrency (N editors making simultaneous edits/comments), offline/rehydration scenarios, ACL enforcement, anchor resilience (inserts/deletes around anchors), performance tests at scale. Supply reproducible harness with deterministic event replay and fuzzing scripts to simulate race conditions.

Metrics & rollout

• Track conflict rate, merge latency, comment placement accuracy, storage overhead. Start with gated rollout (alpha) on a subset of users, collect telemetry, iterate.