Real-Time and Streaming System Design Questions

Requirements clarification:

• Real-time booking event stream with at-least-once delivery guarantee.
• Ability for partners (or internal teams) to replay events up to 90 days.
• Scalable to high throughput, secure, observable, and easy to integrate.

High-level architecture (components)

1. Messaging backbone

• Kafka (self-managed or cloud: Confluent Cloud / AWS MSK) as the source-of-truth event log.
• Topic per logical stream (e.g., bookings), partitioned by booking_id (or tenant+booking_id) to preserve ordering at booking-level.
• Configure retention.ms = 90 days (or use tiered storage to offload older segments to S3 while preserving replay capability).

2. Schema & compatibility

• Schema Registry (Avro/Protobuf) for schema evolution and compatibility guarantees. Embed schema id in messages.

3. Producers

• Application services produce events to Kafka with idempotent producers and retries. Each event includes event_id, timestamp, version, and booking_id.

4. Consumers / Partner delivery

• Two delivery options:
  a) Direct Kafka access: provide partners with read-only Kafka credentials and topic ACLs + recommended consumer group pattern; partners handle their own replay via offsets/time.
  b) Managed delivery (recommended for non-Kafka partners): Delivery service that consumes Kafka and forwards to partners via Webhooks or HTTP streaming (SSE/gRPC). The delivery service:
  • Maintains per-partner offsets committed to durable store (e.g., Kafka consumer group offsets OR a metadata DB) to support replay/seek.
  • Implements at-least-once semantics by redelivery until explicit 2xx ACK. Use exponential backoff, retry policy, and DLQ for persistent failures.
  • Exposes a replay API for partners to request replay for a time range or offset range; service seeks consumer to requested offset/time and replays events.

5. Storage & long-term replay

• Primary: Kafka retention 90 days satisfies replay requirement.
• Cost optimization: enable Kafka tiered storage (broker -> S3) or export topics to S3 via Kafka Connect periodically for audit/long-term archiving; keep metadata to map offsets -> S3 segments for replay if needed.

6. Reliability & delivery guarantees

• At-least-once: producer retries + idempotent producers to avoid dupes at produce time; consumers/processors must be idempotent OR include dedupe logic using event_id (in-memory cache + Redis or datastore for longer dedupe windows).
• Delivery service tracks delivered offsets and only advance after successful ACK to avoid data loss.
• DLQ for events failing after N retries; provide partner-level monitoring and manual replay from DLQ.

7. Scalability

• Scale Kafka by increasing partitions and brokers.
• Delivery service horizontally scalable; partition assignment based on Kafka partitions, use sticky partition consumer groups to maintain ordering.
• Use autoscaling for webhook workers, bounded concurrency per partner, rate-limiting & backpressure (429/Retry-After).

8. Security & multi-tenancy

• TLS encryption in transit, ACLs per topic, SASL/OAuth for Kafka auth.
• Per-partner credentials for webhook endpoints, HMAC signatures for authenticity, rate limits.

9. Observability & operations

• Metrics: producer/consumer lag, end-to-end latency, delivery success rate, retries, DLQ rates.
• Logging/tracing: include event_id, booking_id in traces (OpenTelemetry).
• Alerting: consumer lag thresholds, increased DLQ rates.
• Runbook for replay: APIs to request replay, validate offsets/time ranges, and monitor progress.

Trade-offs and notes

• Direct Kafka access is simplest for partners with Kafka expertise; managed webhook delivery simplifies integration at cost of operational overhead.
• Retention = 90 days is straightforward but consumes broker storage; use tiered storage or S3 export for cost efficiency.
• At-least-once means consumers must be idempotent; if partner requires exactly-once, add dedupe TTL or transactional end-to-end patterns (more complexity).

APIs to offer to partners

• Subscription API: create subscription (delivery method, filter, starting position: earliest, time, or offset).
• Replay API: request replay by time range or offsets; returns job id and progress.
• Health & metrics endpoint: delivery status, last-offset, lag.
• Management web UI for onboarding, keys, and replay controls.

This architecture meets at-least-once semantics, provides 90-day replay through Kafka retention/tiered storage, supports scalable delivery patterns (direct Kafka or managed webhooks), and includes operational controls (monitoring, DLQ, replay APIs) required for production-grade partner integrations.

Requirements clarification:

• Ingest 1M events/sec, real-time (sub-second) processing, ad-hoc queries P95 < 200ms, multi-tenant isolation, cost-conscious.

High-level architecture (components & data flow):

1. Edge/SDKs → Ingestion

• Lightweight SDKs / HTTP/gRPC collectors fronted by CDN + regional ingress (Autoscaling proxy + rate-limits).
• Kafka (or cloud Kinesis) as durable, partitioned event bus (topic per customer or sharded by tenant-key) for backpressure and replay.

2. Stream Processing

• Stream layer (Flink/Beam/ksqlDB) consumes Kafka, performs enrichment, dedup, windowed aggregations, and computes pre-aggregates/metrics. Outputs:
  • Low-latency materialized views for recent data
  • Cold-tier batch outputs

3. Storage tiers

• Hot tier (seconds → minutes): In-memory distributed OLAP like Druid/ClickHouse or cloud managed materialized view service (e.g., BigQuery BI Engine + Materialized Views) for ad-hoc queries. Per-tenant logical isolation via namespaces; physically multiplexed to save cost.
• Warm tier (hours → days): Columnar store (ClickHouse/Parquet on S3 with query engine like Trino) for slightly older data.
• Cold tier (weeks → years): Compressed Parquet on object storage for compliance/analytics.

4. Query API & Ad-hoc engine

• Stateless query API routes queries to hot or warm tier using cost/latency-aware planner. Pre-warmed nodes and cached materialized views for common query patterns to keep P95 < 200ms.
• Query sandboxing enforces per-tenant resource limits and uses query timeout/slots to preserve SLAs.

5. Multi-tenant isolation & security

• Logical isolation: tenant-id in metadata, RBAC, encryption at rest/in transit.
• Resource isolation: QoS via Kafka quotas, stream processing per-tenant pools for high-value customers; burst credits and fair-share scheduling.
• Billing: meter ingestion, query compute, storage.

6. Observability & ops

• End-to-end tracing, metrics, and SLO dashboards. Autoscaling policies, cold-start mitigation, cost alerts.
• Cost optimizations: tiered storage lifecycle, shared compute for low-activity tenants, spot instances for non-critical batch processing, pre-aggregate heavy queries.

Trade-offs and rationale:

• Using a shared hot OLAP reduces cost vs fully dedicated clusters while per-tenant logical isolation retains security and predictable latency via QoS.
• Materialized views and pre-aggregates shift compute earlier (streaming) to keep ad-hoc query latency low and cost-efficient.
• Kafka + stream processing provides resilience/replay and lets costlier heavy queries run against warm/cold tiers asynchronously.

How this maps to product strategy & engineering values:

• Customer-focused: low-latency, ad-hoc exploration for analysts.
• Scalable & reliable: partitioned ingestion and replay for correctness at 1M/s.
• Cost-conscious: tiered storage, shared compute, and spot/managed services to minimize ops burden.
• Secure & fair: multi-tenant QoS, per-tenant billing and isolation supports enterprise adoption.
• Iterative: start with managed components (cloud pub/sub + managed OLAP), measure costs, and optimize hotspots into specialized infra as product scales.

Requirements & constraints:

• Offline-create/edit contacts on mobile; low-latency UX (optimistic local commits).
• Efficient sync over variable bandwidth; support conflict detection + resolution, auditability.
• Scalable server-side reconciliation, idempotency, and security.

High-level architecture:

• Client: local DB (SQLite/Realm), change queue (operation log), sync engine, UI merge component.
• Server: API + sync service, persistent store (contacts + metadata), reconciliation worker, audit log.

Data model (contact + metadata):

• Contact { id (UUID), fields: {name, phone, email, ...}, tombstone:false }
• Metadata per contact: versionVector: {clientId:counter}, lastModifiedAt (ISO), lastModifiedBy (clientId), opLog: [opIDs...]
• Operation: {opID(UUID), contactID, type: create/update/delete, fieldChanges:{field: newValue}, clientId, counter, timestamp}

Client change queue:

• Append-only operation log persisted locally (ops immutable, includes opID).
• Optimistic apply: update local DB immediately; mark contact syncState: pending.
• Queue supports replay, dedupe by opID, and compaction (squash sequential updates to same field before upload).

Batching & sync strategy:

• Adaptive batching: send sync when (a) ops >= 25, (b) total payload >= 64KB, or (c) idle/network restored/time window (every 30s background) or app foreground.
• Prefer incremental sync: client sends last-known server versionVector & batched ops.
• Compress payload (gzip) and use HTTP/2 or gRPC streaming where available.
• Prioritize small essential fields first (list sync) then full payload for details.

Retry / backoff:

• Use idempotent POST with opIDs. On transient failures: exponential backoff with jitter: base=1s, multiplier=2, max=64s, jitter= +/-20%.
• Persist retries across app restarts; limit attempts (e.g., 10) then mark offline-failed and surface to user.

Server reconciliation algorithm:

1. Receive client batch with ops and client's versionVector.
2. Validate idempotency (skip known opIDs).
3. Apply ops in causal order using client's counters; update server versionVector.
4. For concurrent conflicting writes (server versionVector and client vector are concurrent — neither dominates):
   • Per-field reconciliation: attempt automatic merge where safe (e.g., merge lists, union phone numbers).
   • For scalar fields, use hybrid policy: prefer higher lastModifiedAt across authoritative source if within allowed skew AND same user; otherwise mark field as conflicted.
5. Produce server response: appliedOps, conflicts[] (list of fields with both values, metadata), newServerVersionVector, and authoritative contact snapshot.
6. Persist audit trail for each op and conflict resolution.

Conflict detection/resolution strategy:

• Use version vectors for causality detection; per-field LWW only as fallback.
• Auto-resolve trivial merges (non-overlapping fields, additive lists).
• Flag true conflicts (concurrent scalar edits to same field) for manual resolution.

Client-side handling & presenting merges to users:

• Sync response returns authoritative snapshot + conflict metadata (both values, timestamps, clientIds).
• UX: Show merge screen per-contact listing only conflicted fields:
  • Side-by-side values with provenance (device/user, timestamp).
  • Quick actions: select A, select B, edit combined value, or accept server.
  • Show "auto-merged" indicator for fields auto-resolved, and an “undo” option.
• Resolve locally creates a new operation with opID and is enqueued; optimistic apply updates UI.
• Offer bulk resolve for multiple contacts with same conflict policy, and "always prefer my edits" toggle per-user.

Edge cases & extras:

• Deletes: tombstones with versionVectors to avoid resurrecting deleted contacts.
• Offline new contact UUIDs ensure uniqueness.
• Conflict escalation: if user ignores conflicts > X days, use policy to auto-resolve and notify.
• Security: encrypt local DB, TLS for transport, auth tokens + replay protection.
• Monitoring: metrics for conflict rate, sync latency, failed ops to tune thresholds.

Why this works:

• Version vectors provide correct causality detection; op-based model enables idempotency, efficient incremental sync, and compact batching. Adaptive batching balances latency and bandwidth; server-side conservative merging plus clear UX for manual conflicts keeps data correct and user trust high.

Requirements & goals:

• Prevent unauthorized playback, restrict content to entitled users/devices, enable traceability for leaks, minimize UX friction, scale globally.

High-level architecture:

• Client apps (web/Android/iOS/OTT) ← CDN (signed URLs, WAF) ← Origin DRM license server + token validator + forensic watermarking service + analytics + KMS/CA for signing.

DRM integration:

• Support Widevine (CENC), PlayReady, FairPlay depending on platform. Use a unified license abstraction layer (LaaS) that maps content-policy → vendor-specific license response.
• Use per-play licenses tied to user/device attributes. License payloads include key IDs, usage rules (HD/SD, output protection level), expiration and device binding (device ID or hardware-backed key attestation like Android Keystore/TEE, iOS Secure Enclave).

Signed tokens & authentication:

• Issue short-lived JWT-like playback tokens from auth service after entitlement check. Token claims: user_id, content_id, session_id, device_fingerprint, nonce, exp. Tokens signed with KMS private key and validated by license server and CDN edge.
• Token flow: client requests manifest/URL with token → CDN validates signature & expiry → license server re-validates token before issuing key.

Per-device licensing & attestation:

• For high-risk content, require device attestation (SafetyNet/Play Integrity, Apple DeviceCheck) and embed device public key into license binding.
• Enforce hardware-backed key usage: license only usable by TEE/secure decoder (Widevine L1/PlayReady HW).

Forensic watermarking:

• Integrate dynamic forensic watermarking at packaging/origin or player SDK. Watermark payload = session_id, user_id hash, content_id, timestamp, device_id. Use imperceptible audio/video watermarks (frame-level spread-spectrum or audio phase modulation).
• Trigger watermarking per-play or per-chunk for traceability. Store watermark mapping in secure DB for investigations.

CDN-level protections:

• Signed URLs for manifests/segments with short TTL and path-restrictions (IP, token). Use edge token validation (Fastly/Varnish/Lambda@Edge).
• WAF rules: block known scraping patterns, rate-limit manifest/segment requests, geo/IP reputation checks, behavior-based anomaly rules (excessive parallel streams).
• Edge cache key hardening: include token/session in cache key only if safe; otherwise route to origin for validation.

Anti-piracy & enforcement:

• Rate-limit per-account concurrent streams; detect abnormal patterns (many unique IPs for same account) → auto-throttle or revoke tokens.
• Suspicious leak detection: apply automated monitoring for public URLs, social media scanning (third-party services) and correlate watermark data to identify source.
• Automated takedown pipeline: when watermark identifies leaker, generate legal report + CDN takedown request + revoke device licenses and tokens.

Operational touchpoints:

• Provisioning: KMS for signing tokens/licenses, cert rotation policy, HSM usage.
• Monitoring: real-time metrics (license requests, failed attestation, replay attempts), SIEM ingest, dashboarding (DRM errors by region/device), alerting for anomaly thresholds.
• Incident response: playbook for detected leaks (isolate account, revoke tokens, collect forensic watermark evidence, engage legal).
• Testing & QA: regular pentests, DRM interoperability tests, watermark robustness tests (re-encoding/resampling).
• Compliance & auditing: logging of license issuance, token validation, watermark mappings; retention policies and audit trails.

Trade-offs:

• UX vs security: shorter tokens/attestation increase friction; use risk-based escalation (transparent defaults, stricter on premium content).
• Cost vs detection fidelity: per-chunk watermarking and advanced monitoring are costlier but improve traceability.

This design scales across platforms, enforces per-device rights, integrates industry DRM, embeds traceability via watermarking, and provides CDN-layer hardening plus operational controls for enforcement and monitoring.

Requirements & constraints:

• Minimize viewer disruption (low error/latency), support multiple clients/regions, rollback fast, validate correctness (playback, DRM, captions), measure QoE (startup time, rebuffering, bitrate), and scale to peak traffic.

High-level rollout strategy (staged, automated):

1. Pre-release: unit/integration tests + compatibility matrix (clients, CDNs, DRM).
2. Synthetic load & correctness testing:
   • Run CI jobs that exercise new codec/manifest generation at scale using traffic replay and emulated clients (vary bandwidth, latency, device profiles). Measure encoding CPU, manifest correctness, segment timing, and end-to-end playback success.
   • Inject faults (packet loss, stalled encoders) to validate resiliency.
3. Shadow / traffic mirroring:
   • Mirror a fraction of real live traffic to a non-production environment that serves responses using the new format without impacting users. Compare outputs (bit-exact or checksum for segments; parsed manifests) and QoE signals from synthetic clients.
   • Use deterministic hashing (user-id modulo) to mirror representative demographics.
4. Canary release:
   • Progressive traffic shift: 0.01% → 0.1% → 1% → 5% → 20%. Use feature flags and routing controls at CDN/edge and origin.
   • For each step hold for observation window (e.g., 30–60 min for transient issues; longer for gradual metrics).
5. Monitoring gates & observability:
   • Real-time dashboards and automated gates for:
     • Error rate (4xx/5xx, manifest parse errors) — e.g., fail if >0.1% increase vs baseline (stat sig).
     • QoE: startup time, rebuffer ratio, average bitrate — e.g., >10% degradation or KS-test significant change.
     • Client crashes or telemetry (SDK exceptions).
     • Backend metrics: origin CPU, encoding latency, segment generation time.
     • Business KPIs: playback completion, ad impressions.
   • Use anomaly detection (rolling baseline, seasonality-aware) and statistical tests (A/B significance) per region/client.
6. Rollback & safety:
   • Automated rollback triggers: any gate breach triggers immediate traffic rollback to previous version and alerting. Implement instant kill-switch to route new requests away at the CDN edge.
   • Graceful fallback: ensure manifests include backward-compatible fields; support serving legacy manifests if clients fail to parse.
   • Post-rollback: capture debug artifacts (failing manifests, client traces) and run deeper forensics.
7. Post-deploy validation & ramp-down:
   • After successful 20% stable period, continue ramp to 100% with longer windows and sampling.
   • Run periodic A/B tests comparing QoE and cost (bandwidth/encoding) to validate benefits.

Implementation & automation:

• CI/CD pipeline with staged environments, feature-flag service, traffic router integration with CDNs, synthetic testing harness, and runbooks.
• Define SLAs and SLOs; encode thresholds into deployment orchestration so gating is automated but allow manual override with approvals.
• Logging/Tracing: attach request IDs for manifest/segment flows, collect client-side SDK telemetry (sampled) and correlate with backend logs.

Trade-offs:

• Shadow testing gives realism without risk but is costlier. Canary reduces blast radius but takes time. Threshold strictness balances safety vs speed.

This strategy provides layered validation (synthetic→shadow→canary), clear monitoring gates, and fast rollback paths to safely roll out codec or manifest changes at scale.