Real-Time and Streaming System Design Questions

High-level approach
Design a horizontally scalable, partitioned streaming layer (managed or self-managed Kafka/PubSub), a processing tier (stream processors), and a tiered storage backend (hot store for sub-second analytics + cold store for long-term). Prioritize partitioning, buffering/backpressure, autoscaling, and correctness (EOS or idempotent at-least-once).

Ingestion: managed vs self-managed

• Managed (AWS Kinesis, GCP Pub/Sub, Confluent Cloud, AWS MSK Serverless): faster ops, autoscaling, integrated IAM/monitoring. Good for 1M eps to reduce SRE burden.
• Self-managed Kafka (MSK/own clusters): more tunable (broker count, disk, network), lower cost at scale but requires ops expertise (ZK/RAFT, replicas, rolling upgrades).
  Recommendation: start managed (Confluent Cloud or MSK with Provisioned/Serverless) for reliability, move to self-managed only if cost/latency demands justify ops.

Partitioning & keying

• Target high partition parallelism: estimate partitions >= peak throughput / per-partition throughput. If 1M eps and safe per-partition ~10-20k eps, need 50–100 partitions per topic; for headroom aim 200–500.
• Keying: pick business key when ordering matters (userId, sessionId). For even load, use composite/hashing: hash(userId) % N_partitions. Avoid hot keys; use salted keys for extreme skews.
• Use topic-per-tenancy or per-event-class when retention/consumers differ.

Buffering & backpressure

• Client-side batching, compression, async acks. Tune linger.ms, batch.size.
• Broker-level quotas and producer backpressure: producers should handle backpressure (retry/backoff, circuit-breaker), rate-limit at ingress (API GW or load balancer) to avoid cascade failures.
• Use intermediate durable buffer (Kafka / Kinesis) with retention > processing SLAs to absorb spikes; if bursts exceed capacity, spill to S3 as durable overflow.

Storage tiers

• Hot tier: Kafka/KV store (DynamoDB/Redis/Bigtable) for sub-second lookups and materialized views (windowed aggregates). Keep recent X hours/days.
• Warm/Cold: Batch-append raw events to object storage (S3/GS) partitioned by time and shard for analytics and reprocessing; use Iceberg/Parquet and partition pruning for queries.
• Lambda/Glue or flink jobs to compact events into OLAP tables.

Consumer scaling & processing

• Use scalable stream processors: Apache Flink (stateful, low latency), Kafka Streams, or managed stream analytics. Map partitions -> parallel operator instances (one subtask per partition).
• Prefer stateless horizontal scaling where possible; for stateful operators use keyed state with RocksDB and checkpointing.
• Autoscale consumer groups by partition count; implement rebalance-stable strategies (sticky) to limit downtime.

Exactly-once vs at-least-once + idempotency

• Exactly-once: Kafka EOS + transactional producers + consumers with read-process-write transactions; or Flink’s two-phase commit sink connectors + checkpointing. Requires broker/support and careful sink semantics.
• At-least-once + idempotency (practical): enable producer idempotence, store event-id-based dedupe in sinks (unique key constraint in DynamoDB/SQL, upserts in OLAP) or use idempotent write tokens. This scales easier with external sinks (S3/batch sinks) and is simpler operationally.
  Recommendation: use EOS for end-to-end when sinks support transactions; otherwise implement at-least-once + dedupe and consumer-side dedup windows.

Operational & monitoring

• Track throughput, consumer lag, partition skew, GC, network I/O. Use alerting on throttling, high lag.
• Load test with realistic key distributions and spike scenarios; tune partition count and retention.

Trade-offs: EOS simplifies correctness but increases latency and complexity. Managed services reduce ops but can cost more. Prioritize predictable partitioning and durable buffering; combine autoscale + idempotent sinks for robust, cost-effective 1M eps.

Choice & summary
I would choose Confluent Cloud (SaaS) for a real-time streaming pipeline because it minimizes operational burden while providing enterprise features (multi-tenant scaling, connectors, schema registry) that speed time-to-value for a cloud engineering org.

High-level architecture

• Producers (microservices, IoT, DB CDC) -> Confluent Cloud Kafka topics -> Stream processors (ksqlDB / consumer apps / Flink) -> Sinks (data lake, OLAP, search, alerting).
• VPC peering / PrivateLink between cloud VPC and Confluent for private network traffic.
• Schema Registry, REST Proxy, Connectors managed by Confluent; client apps in our accounts.

Throughput & latency

• Provision throughput via Confluent capacity tiers (dedicated throughput units). For >1M msgs/sec choose provisioned/dedicated cluster; for bursty loads use auto-scaling tiers.
• Expect 10s–100s ms end-to-end for basic producer->consumer patterns; tune partition count, client batching, linger.ms, and use dedicated clusters for lower jitter.

Operational burden

• Provider responsibilities: broker lifecycle, patching, replication, Zookeeper/KRaft, scaling, backups, basic monitoring of cluster health.
• Our responsibilities: client configuration, schema evolution, connector configuration, consumer lag management, alerting, cost management, IAM mapping.
• Benefit: fewer on-call nights and no OS/VM maintenance compared to self-hosted.

Data retention

• Use topic-level retention policies: short retention for hot streams, long retention or tiered storage (Confluent Cloud’s storage) for audit/replay. For regulatory needs export to S3/GCS via sink connectors and enforce immutability/versioning there.

Security responsibilities

• Encryption: Provider handles encryption at-rest and in-transit by default. We enforce TLS, restrict listener endpoints to PrivateLink or VPC peering.
• ACLs & auth: Use SASL/SCRAM or mTLS + RBAC (Confluent Cloud RBAC). We define least-privilege roles, manage service principals, rotate keys via automation.
• Network: Restrict IPs, use VPC endpoints, and private networking.
• Monitoring & logging: Provider supplies cluster metrics; we integrate with Datadog/Prometheus via Confluent metrics export and enable audit logs to our logging account (S3/CloudWatch). We monitor consumer lag, throughput, error rates, and security events; alert on threshold breaches.
• Compliance: Use Confluent features + our cloud controls (KMS for key management, IAM policies, bucket encryption) to meet PCI/HIPAA/GDPR.

Trade-offs

• Pros: Lower ops, faster features, built-in security and support.
• Cons: Higher cost at scale and less deep control than self-hosted; for ultra-low latency or custom broker tuning, IaaS might be preferred.

Operational checklist

• Define SLOs (latency, availability), capacity plan, runbook for failover, automated secret rotation, and CI for connector/config changes.