Cloud Architecture Design Principles and Trade-offs Questions

Approach summary
Design for 1M writes/sec by combining horizontal sharding, geo-partitioning, lightweight consensus for metadata, and a write-path optimized for append-only operations with caching and batching.

Partitioning / sharding

• Shard by write-key hash to evenly distribute load; use consistent hashing to allow elastic resharding.
• For geo-sensitive data, use geo-partitioning: primary shard per region and asynchronous cross-region replication for global reads.
• Support range shards for ordered workloads (time-series) to minimize hotspotting; use adaptive split/merge.

Consensus / coordination

• Use Raft/Paxos for small control-plane metadata (shard map, leader election). Keep data path leaderless or multi-leader:
  • Leaderless (e.g., Cassandra-style) for high write throughput and low coordination: quorum writes with vector clocks or lightweight timestamp conflict resolution.
  • Multi-leader with conflict resolution for geo-local writes where stronger local availability is required.
• Avoid consensus on each write; rely on durable local commit and background anti-entropy.

Write amplification & storage

• Prefer LSM-tree stores (rocksdb) for high ingest; tune compaction (size-tiered, leveled) to balance amplification vs. read latency.
• Mitigate amplification with larger memtables, batching, and short-lived write buffers; use SSDs with provisioned write endurance.

Caching layers

• Client-side SDK batching and write buffers.
• Regional write frontends that ACK on durable local write (WAL + local memtable) and replicate asynchronously.
• Hot-key caches (in-memory) and a CDN for read-heavy traffic.

Latency vs throughput trade-offs

• Lower latency: sync replication, single-leader per shard — increases coordination cost and limits throughput.
• Higher throughput: leaderless/multi-leader, batched async replication — increases eventual consistency and tail-latency variance.
• Reasoning: set SLOs per operation (p99 latency, durability window). Use backpressure, adaptive batching, and priority lanes (small sync writes vs bulk async) to meet mixed SLAs.

Operational considerations

• Autoscaling shards, observability (per-shard metrics), chaos testing for consistency, and clear SLA-driven choices between consistency vs availability.

Overview & goals
Design an ingest-to-query observability pipeline for 100k events/sec (mixed logs/metrics/traces) that is scalable, cost-controlled, and SLO-safe.

Ingestion architecture

• Fronted by an autoscaling Ingress fleet (API Gateway + NLB) and regional Kafka (or MSK / Confluent) topics partitioned by tenant/service.
• Lightweight schema validation (Protobuf/Avro) and rate-limiting at edge; ACK after durable write to the stream.

Stream processing

• Use stream processors (Kafka Streams / Flink) for:
  • Real-time enrichment, dedup, normalization
  • Aggregation windows for metrics (1s/10s/1m)
  • Trace tail-sampling decisions
• Emit three streams: hot (recent), warm (24–90d aggregated), cold (archival).

Storage tiers

• Hot: Time-series DB (Cortex/Prometheus Remote Write+Cortex or M3) + fast object store for logs/traces (Elasticsearch replacement like ClickHouse or Loki for logs; Tempo/Jaeger for traces).
• Warm: Downsampled TSDB + compressed logs in columnar store (Parquet on S3 with partitioning).
• Cold: Immutable, compressed archive in object storage with indexed manifests.

Indexing & query patterns

• Index only high-cardinality keys in hot; use inverted index for logs and minimal trace indexes (traceID, service, span name).
• Support two query classes: real-time SLO queries hitting hot; historical/forensic queries that hit warm/cold with async retrieval.
• Precompute common aggregates (per-service latency histograms) and synthetic metrics for quick SLO evaluation.

Sampling & aggregation strategies

• Metrics: full retention of counters; histogram bucketing + downsampling.
• Logs: structured indexing for errors; adaptive log sampling (keep all error-level, tail-sample noisy debug).
• Traces: tail-based sampling using stream processor that scores traces by error/latency and preserves all high-value traces; probabilistic sampling for baseline traffic.
• Use deterministic sampling keys for consistent reductions across services.

SLO-driven alerts under sampling

• Maintain a separate, high-fidelity SLO pipeline: keep full-resolution metrics and SLO-relevant traces for SLO windows (e.g., keep 100% of error/latency samples that affect SLO thresholds).
• When applying sampling, attach sampling weights and expose them in queries; alerting engine applies inverse-weighted estimators and computes confidence intervals.
• Implement adaptive sampling: increase retention automatically when estimated error/confidence crosses thresholds to preserve alert accuracy.

Observability & reliability

• End-to-end SLAs for ingest, processing lag, and query latencies. Instrument pipeline health, backpressure, and sampling effectiveness. Continuous validation: synthetic traffic with golden signals to verify SLO computations.

Clarify goals & constraints

• Prevent noisy-neighbor and DoS, preserve low latency, support global regions, allow developer self-service and smooth upgrades, cost-bounded.

Tiered quotas & per-key limits

• Define tiers: Free, Developer, Business, Enterprise. Each tier includes:
  • Monthly quota (requests/month, data egress)
  • Steady-state RPS (requests/sec) and burst capacity
  • Priority level (for graceful degradation)
• Per-key limits = (steady RPS, burst tokens, concurrent requests). Keys map to tenant + application.

Burst handling & throttling semantics

• Use token-bucket per key for bursts: bucket size = burst_capacity, refill rate = steady RPS.
• When tokens exhausted: return HTTP 429 with Retry-After header (seconds until next token) and descriptive body (current usage, tier, link to dashboard).
• Soft vs hard limits: soft throttles (429) for normal enforcement; hard caps for monthly quota (403 when exceeded) with billing/upgrade link.

Enforcement architecture

• Enforce at the edge (API Gateway / CDN) for low latency and reduced attack surface. Implement local per-key caches and token buckets in gateway instances.
• Central quota service for accounting, long-term quotas, billing, policy management, and global decisions.
• Edge-first; central authoritative. Edges operate independently using lease-based local allowances and periodic reconciliation.

Distributed quota sync

• Use hierarchical allowance model:
  • Central allocator gives regional leases (e.g., N tokens/min) per key via consistent hashing or tenant sharding.
  • Edge consumes from local bucket; when local allowance low, edge requests refill from central (non-blocking fallback).
• For cross-region failover, use optimistic local borrowing with background reconciliation and CRDT-based counters for monthly usage to tolerate partitions.
• Central store: strongly consistent DB for policy (e.g., Spanner/Cockroach) + distributed cache (Redis Cluster) for fast lookups. Use rate-limiter metadata with short TTLs.

Operational & security considerations

• DDoS protection: integrate WAF and anomaly detection (rate spikes per IP, geolocation), auto-block or escalate to manual review.
• Monitoring & alerting: per-tenant RPS, token consumption, rejects, error budgets; SLOs for 429 rates.
• Cost control: circuit breakers for extreme spikes and emergency kill-switch per tenant.

Developer self-service

• Usage dashboard showing live RPS, burst consumption, monthly quota, predicted exhaustion time.
• Self-serve upgrade: tier change via UI/API with immediate policy push and pro-rated billing.
• SDKs add client-side exponential backoff respecting Retry-After; sample code + best-practice docs.
• Webhooks/alerts for approaching quota thresholds.

Trade-offs

• Edge enforcement = low latency, complexity in consistency. Central-only = simpler but higher latency and single point of failure. Lease-based hybrid balances both.

This design provides low-latency protection at the edge, strong central accounting and global fairness, predictable burst behavior, and a developer-friendly upgrade and visibility experience.

Brief comparison (high level)

• Serverless Functions (FaaS)

  • Cold-start latency: can be high for infrequent, heavy runtimes (ms–s).

  • Concurrency limits: platform-managed limits; scale quickly but may hit account throttles.

  • Statefulness: inherently stateless; require external state (DynamoDB, Redis).

  • Operational burden: minimal infra ops; focus on code and permissions.

  • Observability: platform provides traces/metrics, but instrumentation across invocations is needed.

  • Portability / vendor lock-in: higher when using proprietary triggers or bindings.

  • Cost model: pay-per-invocation and execution-time; excellent for spiky, low-utilization workloads.

  • Example: API endpoint for user-uploaded image thumbnails — spiky traffic, short execution, low ops.

• Containerized workloads on managed Kubernetes

  • Cold-start latency: moderate (seconds) depending on pod startup and image size.

  • Concurrency limits: configurable (pods, HPA/VPA); can handle complex scaling policies.

  • Statefulness: supports stateful sets and sidecars; better for complex stateful apps.

  • Operational burden: higher than FaaS (cluster upgrades, networking, Helm charts) but reduced on managed services.

  • Observability: rich ecosystem (Prometheus, Grafana, Jaeger); requires integration work.

  • Portability / vendor lock-in: high portability if using standard images and manifests.

  • Cost model: pay for nodes (or serverless nodes); efficient at steady, moderate-to-high utilization.

  • Example: Microservices platform with multiple services, custom runtime, and complex networking.

• Long-running VMs / Instances

  • Cold-start latency: low for warm instances; provisioning can be slow.

  • Concurrency limits: determined by instance sizing; high single-instance throughput possible.

  • Statefulness: native support for local state and high-performance disks.

  • Operational burden: highest (OS patches, capacity planning) unless using managed instances.

  • Observability: full control; integrate standard monitoring agents.

  • Portability / vendor lock-in: moderate; images more portable but infra differences exist.

  • Cost model: pay-for-provisioned capacity; good for predictable, high-utilization workloads (reserved instances reduce cost).

  • Example: High-throughput database or legacy application requiring local disk and predictable low-latency.

Trade-offs summary: choose FaaS for event-driven, short-lived, spiky tasks with low ops; choose containers for complex microservices and moderate-to-high steady load with portability needs; choose VMs for latency-sensitive, stateful, or legacy workloads where full control and local resources matter.

Clarify requirements

• RPO/RTO targets, expected TPS/IOPS, dataset size, cross‑AZ latency tolerance, maintenance windows, compliance (encryption, retention), ops team skill level.

High‑level architecture

• Multi‑AZ Kubernetes cluster (control plane HA + worker nodes across AZs) hosting Postgres StatefulSets with Patroni for leader election and automatic failover.
• External high‑performance storage via Cloud provider replicated block volumes (e.g., AWS EBS Multi‑AZ or Azure Ultra Disk zone‑redundant) where available; otherwise use networked filesystems only for read replicas.

Storage choices & CSI

• Preferred: Replicated block storage (PV per pod) for single‑node primary with synchronous or semi‑sync WAL shipping. Use CSI drivers from cloud vendor for volume provisioning, encryption, snapshots, and topology awareness.
• Alternative: Distributed filesystem (Ceph/Rook, Longhorn) if you need PV mobility and cross‑AZ mounts — tradeoff increased complexity and potential latency.

Pod/node affinity & topology

• Use podAntiAffinity to avoid colocating primary and replicas in same AZ/node.
• Use volumeBindingMode: WaitForFirstConsumer + topologyKeys to ensure PV provisioned in same AZ as pod.

Replication & HA

• Patroni + etcd/consul/DCS for cluster state; async replicas in other AZs for reads; quorum kept across AZs for safe failover. Use synchronous replication to an AZ‑local replica for zero data loss when latency allows.

Backup/Restore

• Regular base backups (pg_basebackup / WAL‑archiving) + continuous WAL shipping to object storage (S3), complemented by automated CSI snapshots for fast restores.
• Test restores frequently; stage point‑in‑time recovery (PITR) playbooks and runbooks.

Performance isolation

• Use CPU/memory requests & limits, Pod QoS, storage class with IOPS guarantees, separate node pools for DB with tuned kernel/CPU settings, and dedicated network paths.

Operational complexity vs managed services

• Self‑managed on K8s gives flexibility, control over tuning/compliance, and potential cost savings for large scale, but increases operational burden: upgrades, backups, failover testing, tuning, and operator expertise.
• Managed DB (RDS/Azure Database) reduces operational risk, offers automated backups, cross‑AZ replicas, and SLA but less control and higher cost.

I would recommend self‑managed Kubernetes Postgres only when business needs require custom tuning, extensions, or data locality; otherwise prefer managed DB for lower operational overhead.