Infrastructure Strategy and Technology Selection Questions

Clarify goals & targets

• RPO: 1 minute for critical DBs, 1 hour for non-critical.
• RTO: 15 minutes for failover to standby/replica, 2 hours for full regional recovery, 24 hours for cross-region restore.

High-level design

• Multi-AZ primary with synchronous replicas in AZs and an async replica in another region.
• WAL archiving + PITR for fine-grained recovery.
• Periodic full backups (daily) + incremental backups (hourly).

Implementation details

• Tooling: standardize on pgBackRest for full/incremental + WAL streaming/archiving; keep wal-e as alternative for simple object-store pushes.
• WAL archiving to S3-compatible storage with lifecycle policies; encrypt-at-rest and versioned buckets.
• Daily full backups retained 30 days; incremental chain retention 7 days; archived WALs retained to cover 7–30 days depending on SLA.

Cross-region & testing

• Copy backups to a secondary region asynchronously (immutable snapshots).
• Automated restore tests weekly in a sandbox cluster: restore latest full + WAL apply to a target timestamp, run smoke tests and integrity checks, and report SLA metrics to dashboard.

Operational & team practices

• Runbooks for failover, PITR, and cross-region restores; drill quarterly with on-call rotation.
• Monitoring: alert on backup failures, WAL lag, storage quotas; expose RPO/RTO SLIs to leadership.
• Ownership: designate backup owner, SRE + DB engineer collaboration, postmortems on failures.

Trade-offs

• Synchronous cross-AZ replicas increase write latency; mitigate by local sync and async cross-region for disaster recovery.

Approach summary (EM perspective)
I’d choose managed services and a clear policy-first architecture so teams implement compliance without heavy bespoke work. My role: set standards, pick stack, remove blockers, ensure auditing and runbooks.

Stack choices

• Databases: region-locked managed DBs (e.g., Cloud SQL/Aurora, Azure SQL, Cloud Spanner where regional) with read-replicas only in approved regions. Tag instances with data_class and residency attributes.
• Cloud regions: map customer/data classification to permitted regions; use organization policies (GCP IAM Org Policy / AWS Organizations SCPs) to prevent resource creation outside allowed regions.
• KMS: multi-tenant KMS per jurisdiction (AWS KMS multi-Region keys or separate key rings per region in GCP), with CMKs owned by security team and strict IAM policies. Use HSM-backed keys for regulated data.
• Backups: enforce backup-location policies that pin snapshots to allowed regions; use immutable backups and lifecycle rules. Encrypt backups with region-specific CMKs.
• Access controls: least-privilege IAM, attribute-based access (ABAC) for residency tags, VPC Service Controls / PrivateLink to restrict data egress, and logged sudo/privileged access with Just-In-Time workflows.

Operational controls & validation

• Automation: terraform modules embedding region and KMS constraints; CI checks to prevent infra drift.
• Logging & audit: centralized immutable audit logs (CloudTrail, Audit Logs) forwarded to SIEM; periodic automated checks (Custodian/Conftest) and daily compliance dashboards.
• Validation: quarterly pen tests, yearly third-party compliance audit, monthly automated residency reports that compare data locations vs policy.
• Documentation: living runbooks (confluence) with architecture diagrams, key ownership, rotation schedules, backup retention, incident playbooks, and audit evidence repository with signed attestations.

Trade-offs

• Slightly higher cost for separate regional keys and replicated infra vs reduced legal risk and simpler audits.

This plan lets teams move fast while the org enforces residency and encryption through policy, automation, and clear operational ownership.

High-level approach
Estimate monthly delta = (compute + control-plane + LB + storage + data-transfer + logging + monitoring) for EKS minus current EC2 baseline. Break each into unit metrics (hours, GB, vCPU-hours, GB transferred, log GB) then multiply by cloud pricing.

Factors and how to size them

• Compute: translate 1,000 RPS → required concurrency → pods → vCPU & memory. Example: assume one pod (0.5 vCPU,1GB) handles 200 RPS → 5 pods baseline; with HA and autoscaling buffer use 3× → 15 pods. Map pods to node sizes (e.g., m5.large, 2 vCPU). Calculate node count and vCPU-hours × price.
• EKS control plane: managed EKS fee (~$0.10/hr per cluster) and cost of managed node groups or Fargate (use whichever you plan).
• Load balancers: NLB/ALB hourly charges + per-GB processed. Estimate active connections and egress.
• Storage: EBS for nodes (size × IOPS if provisioned) and persistent volumes (EFS if used).
• Data transfer: egress to Internet and cross-AZ; multiply GB/month by per-GB rates.
• Logging: CloudWatch ingestion and retention (GB/day → monthly cost) and S3 archival.
• Monitoring: Prometheus/Thanos on EKS (additional nodes/storage) or managed solutions (CloudWatch Container Insights) — include their compute and storage.

Concrete quick example (illustrative)

• Compute: 15 pods → ~5 m5.large nodes @ $0.096/hr → 5 * 24 * 30 * 0.096 ≈ $345/month
• EKS control plane: 24300.10 ≈ $72/month
• ALB + data: ALB ~$20 + 10TB egress @ $0.09/GB ≈ $900
• CloudWatch logs: 100 GB/month ingest ≈ $30
  Total ≈ $1.4k/month (values illustrative; compute precise with real metrics)

Validation strategy (tests / pilot)

• Metric collection: instrument current EC2 service to capture RPS, latency P95/P99, CPU/memory per process, connection counts, network egress, and daily log volume.
• Small-scale pilot: deploy identical service on EKS with autoscaling; run synthetic load tests (k6, Gatling) ramping to 1,000 RPS while measuring pod density, node packing, LB behavior, and egress.
• Cost pilot: run pilot for 1–2 weeks with representative traffic (or replay production traces) and gather billing tags to extrapolate monthly spend.
• Safety checks: run chaos tests (node drain, scaling) to ensure HA and validate headroom assumptions.
• Iterate: adjust pod sizing, node types (spot vs on-demand), use savings plans/reserved instances, and re-run pilot to refine TCO.

As an engineering manager I’d present the modeled estimate, validation plan, and risks (control-plane + LB and logging often drive surprises), then run the short pilot with clear success metrics before wider migration.

Overview (Engineering Manager perspective)
Below I describe each stack component, its primary responsibilities, one common technology, and when a team should adopt it.

Relational database

• Responsibilities: durable transactional storage, strong schema, ACID guarantees, complex joins and reporting.
• Example: PostgreSQL.
• When to adopt: use when data integrity, multi-row transactions, and predictable schemas are required.

Non-relational store (NoSQL)

• Responsibilities: flexible schema or key-value access, horizontal scale, specialized models (document, wide-column, graph).
• Example: MongoDB (document) or Cassandra (wide-column).
• When to adopt: use when you need flexible models or very high write/read scale without strict joins.

Caching layer

• Responsibilities: low-latency reads, reduce DB load, store ephemeral derived data.
• Example: Redis.
• When to adopt: use when read latency matters and data can be eventually consistent or recomputed.

Message queue

• Responsibilities: decouple services, buffer load, support async processing and retries.
• Example: RabbitMQ or Kafka.
• When to adopt: use when you need resilience, backpressure handling, or async workflows.

Load balancer

• Responsibilities: distribute traffic, health check instances, enable zero-downtime deploys.
• Example: HAProxy or AWS ELB.
• When to adopt: use immediately when running >1 instance or requiring fault tolerance.

Service runtime

• Responsibilities: execute business logic, manage threads/containers, expose APIs, handle scaling and observability.
• Example: containerized app on Kubernetes.
• When to adopt: use from day one to enforce deployment, scaling, and operational standards.

Overview & requirements
I’d design a horizontally scalable streaming pipeline to ingest 100k events/sec with <1s E2E latency, strong durability, and failure isolation. As manager I’d outline architecture, pick technologies, and surface trade-offs for exec and engineers.

Architecture (high level)

• Ingress: HTTP/gRPC collectors behind autoscaling LB (Nginx/Envoy) + regional edge for geo.
• Durable buffer: Partitioned Kafka cluster (topic replication factor 3); producers use async acks=1 for latency, consumer lags monitored.
• Stream processing: Apache Flink (low latency, event-time, exactly-once with state snapshots).
• State store: Flink RocksDB local state with incremental checkpoints to S3; standby replicas via Kafka changelog for quick recovery.
• Caching/fast read: Redis Cluster for read-heavy lookups (TTL, hot-key mitigation).
• Sinks: Downstream microservices or OLAP (ClickHouse) via Kafka Connect.

Partitioning & throughput

• Partition by event key (userId or shard key) to ensure affinity; set partitions >= number of consumer slots (nodes * task slots). Aim ~300-500 partitions to distribute 100k/s.
• Key hashing with range balancing; add consistent hashing to re-shard with minimal movement.

Durability & failure isolation

• Kafka replication + ISR, Flink checkpointing to durable storage, consumer groups isolated per pipeline to avoid noisy-neighbor effect. Use circuit-breakers and DLQ topics per stage.

Backpressure & flow control

• Configure producers to handle 429s, client-side batching, and Kafka broker quotas. Flink’s backpressure metrics trigger autoscaling or shedding policies (sample, prioritize).

Trade-offs

• Kafka vs Kinesis: Kafka gives more control, higher throughput and ecosystem (Connect, MirrorMaker) but higher ops cost; Kinesis reduces ops but has shard limits and higher per-unit latency/cost at scale.
• Flink vs Spark Streaming: Flink for sub-second latency and strong event-time semantics; Spark Structured Streaming simpler for micro-batch but higher latency.
• In-memory state vs RocksDB: Pure in-memory (Redis/local heap) gives fastest access but limited by memory and risk on failover; RocksDB + incremental checkpoints offers larger state, lower memory pressure, and faster recovery with acceptable latency.
• Caching layer: Redis reduces load for hot keys; consistency trade-off handled via TTLs and pub/sub invalidation.

Operational concerns

• SLOs, observability (Prometheus, Jaeger), chaos testing, playbooks for broker/controller failures, capacity planning, and runbooks for rebalancing partitions.