Cloud Architecture Design Principles and Trade-offs Questions

Clarify requirements

• Read-heavy, customer-facing, multi-AZ HA, horizontal read scaling, minimal downtime, acceptable replica lag thresholds (define SLA e.g., <2s), RTO/RPO targets.

High-level architecture

• Primary: Amazon RDS Multi-AZ (synchronous standby in another AZ) for HA of writes.
• Read scale: Amazon RDS Read Replicas (in same region, across multiple AZs). For extreme scale, consider Aurora with cluster endpoints.

Placement & replication

• Primary in AZ A with synchronous Multi-AZ standby in AZ B (automatic failover target).
• N read-replicas spread across AZs (C, D…). Use cross-AZ placement to reduce correlated failures.
• Use engine-native async replication (or Aurora’s distributed storage for lower lag).

Automated failover behavior

• Multi-AZ failover: RDS promotes standby to primary automatically; DNS endpoint (writer) remains same CNAME; connections briefly drop.
• Read-replica promotion: If primary fails and standby not available, promote a read-replica manually/automated via failover plan (Aurora supports faster failover).

Connection routing / read-write separation

• Maintain two endpoints: writer (primary) and reader(s).
• Use application-side logic or proxy (AWS RDS Proxy, HAProxy, PgBouncer) to route writes to writer endpoint and distribute reads across reader endpoints.
• For Aurora: use cluster writer endpoint + reader endpoint (automated load balancing).

Handling replica lag

• Monitor replica_lag metrics, CloudWatch alarms.
• Prefer replicas for non-critical, eventual-consistent reads.
• Implement client-side fallbacks: on read-after-write paths, read from writer or use session affinity token until replica catch-up.
• Circuit breaker: if replica lag > threshold, remove replica from pool via proxy/auto-scaling policy.

Failover testing

• Regular automated DR tests: simulate AZ failure (stop primary), promote standby, verify writer endpoint, check replication and app behavior.
• Chaos exercises for replica lag and mass failover.
• Measure RTO, failover time, connection drop count; automate rollback playbooks and runbooks.

Trade-offs

• Multi-AZ increases write availability but not read scale; read replicas add scale but add eventual consistency and management. Aurora reduces lag and simplifies endpoints at higher cost.

Short answer / recommendation
Use containerized workers (ECS with Fargate or EC2-backed ECS) for this pipeline. Containers give more predictable <50ms per-record latency at 10k/sec burst with lower operational risk; serverless (Lambda) can be used for low-throughput or sporadic workloads.

Comparison

• Cold starts

  • Lambda: variable cold starts (tens-to-hundreds ms) depending on language; provisioned concurrency reduces cold starts but costs scale with provisioned instances.
  • Containers: near-zero cold start once tasks are warmed; use pre-warmed pool or autoscaling groups.

• Concurrency limits

  • Lambda: account-level concurrency and per-region quotas; bursts of 10k/s require large provisioned concurrency and careful quota increases.
  • Containers: scale pods/tasks horizontally; can support high concurrency with cluster autoscaling and tuned resource limits.

• Batching & throughput

  • Lambda: good for per-event processing; small batch sizes possible but billing per-invocation; coordinating large bursts needs parallelism.
  • Containers: easier to implement batching within workers to amortize overhead and maintain <50ms if batches small; better for custom protocols and connection reuse.

• Cost

  • Lambda: cost-effective at low to moderate steady usage; high sustained throughput with heavy provisioned concurrency can be more expensive.
  • Containers: better cost predictability at high sustained throughput; spot instances or reserved capacity reduce cost.

• Observability

  • Lambda: CloudWatch logs/traces per invocation; high-cardinality and correlation can be noisy; X-Ray integrates but sampling may be needed.
  • Containers: richer observability via Prometheus, Fluentd, tracing with sidecars; easier to inspect long-running processes and connection metrics.

• Operational complexity

  • Lambda: lower infra ops, simpler deployment; complexity in cold-start tuning, concurrency quotas, and deployment package size.
  • Containers: higher ops (cluster management, autoscaling, deployment pipelines) but more control for performance tuning.

Trade-offs & justification
To guarantee <50ms tail latency at 10k/s, predictable resource control and ability to reuse warm connections matter most. Containers let you size CPU/memory, maintain warm pools, and implement tight autoscaling—trading more operational overhead for predictable latency and lower cost at scale. Choose ECS Fargate for reduced infra management; use cluster autoscaling, pre-warmed task pool, circuit breakers, and robust observability (Prometheus/Grafana + distributed tracing). Use Lambda only if bursts are short-lived, throughput is lower, or you prioritize minimal ops.

Requirements & constraints

• Support internal teams across 50+ AWS accounts, partner access (SaaS and external contractors), central audit/compliance, rapid revocation, least-privilege, and short-lived credentials.

High-level architecture

• Central AWS Organization with delegated OU structure (prod, non-prod, shared-services, partners).
• Central Identity: AWS SSO (IAM Identity Center) backed by corporate IdP (SAML/OIDC, e.g., Azure AD) for employees; external partners use separate external IdP tenants or SAML/OIDC federation.
• Cross-account access via IAM Roles with external IDs for partners and condition keys for session tags.

Core components & responsibilities

• Organization Management Account: SCPs to enforce guardrails (deny iam:CreateUser, restrict regions, require MFA).
• Shared-services account: central logging (CloudWatch Logs, Kinesis, S3), AWS Config, Security Hub, GuardDuty, and AWS CloudTrail (multi-region, encrypted).
• Account-level roles: assume-role patterns — “AuditRole”, “AdminRole” (break-glass), and scoped “AppX-Role-ReadOnly”.
• Permission boundaries on role principals to prevent privilege escalation (deny actions outside boundary).
• Use STS AssumeRoleWithSAML / OIDC for federated sessions; require MFA and device attributes via condition keys.

Least privilege & role design

• Role-per-purpose: small permission sets, resource-level ARNs, and attribute-based access (session tags used in policies to filter resources).
• Use IAM Access Analyzer and automated policy generation (grant least privilege from recorded sessions).
• Enforce permission boundaries and SCPs so even compromised principals cannot expand privileges.

Short-lived creds & session management

• Always use STS tokens with limited duration (15–60 minutes). For machine-to-machine, use IAM Roles for Service Accounts (IRSA) or OIDC with token exchange.
• Rotate and revoke refresh tokens via IdP; enforce short-lived sessions on federation endpoints.

Auditability & detection

• Centralized CloudTrail + S3 lake and Athena for queries; send events to SIEM.
• Real-time detection: CloudWatch metric filters + EventBridge rules for suspicious activities (policy changes, new inline policies, AssumeRole outside business hours) triggering automated workflows.
• Use AWS Config rules for detecting privilege expansions and resource changes; GuardDuty for anomalous STS usage.

Detecting privilege escalation

• Monitor creation/attachment of policies, updates to permission boundaries, and role trust policy changes.
• Alert on STS:AssumeRole with privilege escalation patterns (assume of AdminRole after lateral moves).
• Periodic automated tests: simulate-principal (with least privilege) to check for unexpected permissions.

Fast revocation & incident response

• Immediate steps: disable federated user in IdP (cuts SAML/OIDC logins), remove role trust for the compromised principal (deny AssumeRole), attach deny-all SCP to the member account if needed.
• For session revocation: use STS token revocation via rotating role trust provider or change role’s trust policy to include a condition with a timestamp; revoke refresh tokens at IdP.
• Orchestrate via automated runbooks (Lambda/SSM Automation) to quarantine account, rotate keys, and notify stakeholders.

Trade-offs & alternatives

• Pure IAM users vs federation: federation preferred for centralized lifecycle and fast revocation.
• Short token durations increase friction; mitigate with transparent SSO UX and session renewal automation.

Metrics & compliance

• Track Mean Time to Revoke (MTTR), number of over-privileged roles, drift from permission boundaries, and frequency of AssumeRole events. Regular access reviews and attestation for partner roles.

This design balances operational usability with strict guardrails, enforces least privilege through role scoping and permission boundaries, detects escalation with centralized telemetry, and enables rapid revocation via IdP controls and automated account-level lockdowns.

Compute

• AWS: EC2 (VMs)
• GCP: Compute Engine / Azure: Virtual Machines
• Use case: Host containerized or legacy applications requiring full OS control and custom networking.

Storage

• AWS: S3 (object)
• GCP: Cloud Storage / Azure: Blob Storage
• Use case: Durable, cost-effective object storage for backups, static website assets, and data lakes.

Networking

• AWS: VPC + Transit Gateway
• GCP: VPC + Cloud VPN / Azure: Virtual Network + Azure VPN Gateway
• Use case: Securely isolate workloads, connect hybrid networks, and manage traffic flows across regions.

Databases

• AWS: RDS (managed relational)
• GCP: Cloud SQL / Azure: Azure Database for PostgreSQL
• Use case: Managed OLTP databases with automated backups, patching, and high availability.

Serverless

• AWS: Lambda
• GCP: Cloud Functions / Azure: Functions
• Use case: Event-driven microservices or short-lived compute for webhooks, ETL triggers, and API backends without managing servers.

High-level approach (why):
Defend-in-depth: use network-level segmentation to limit blast radius, then enforce strong identity and encryption at the application layer so only authenticated services communicate.

Network-level controls

• VPC design: isolate workloads into multiple VPCs/subnets (public for ingress, private for services), use Transit Gateway / VPC Peering for controlled connectivity.
• Security groups & NACLs: allow least-privilege traffic (service ports only), use stateful SGs for instance-level, stateless NACLs for subnet boundaries.
• Network policies: in Kubernetes use Calico/NetworkPolicy to restrict pod-to-pod traffic by label; enforce egress controls and DNS filtering.
• Private connectivity: use AWS PrivateLink, Azure Private Endpoint, or GCP Private Service Connect to keep traffic off the public internet.

Application-level controls

• mTLS: require mutual TLS between services for strong mutual authentication and encryption in transit. Use sidecar proxies (Envoy) or platform support to terminate and validate certs.
• JWTs & OAuth2: issue short-lived service tokens via an identity provider (e.g., AWS Cognito, Azure AD, Google IAM) and validate scopes/audiences in services.
• RBAC: apply fine-grained RBAC in K8s and at application APIs to restrict actions per service/principal.
• Secrets & cert management: store keys/certs in AWS Secrets Manager / ACM, Azure Key Vault, or GCP Secret Manager; automate rotation.

Cloud-managed features to use

• Service mesh: AWS App Mesh, Istio/Anthos, or Linkerd for mTLS, traffic policy, observability.
• Managed certificates: AWS Certificate Manager, Google Managed SSL, Azure App Service certificates.
• Private endpoints & load balancers: PrivateLink, Private Service Connect, Azure Private Endpoint; use internal NLB/ALB for internal traffic.
• Identity & IAM: AWS IAM roles for service accounts (IRSA), Azure AD workload identities, GKE Workload Identity.

Example implementation (Kubernetes on AWS)

• Put apps in private subnets, use Security Groups per service tier.
• Deploy Istio or App Mesh with mTLS enforced.
• Use IRSA to grant pods short-lived IAM creds, validate JWTs at API ingress, and store TLS certs in ACM with automatic rotation.

Trade-offs

• Service mesh adds complexity and overhead but centralizes mTLS, routing, and observability.
• Strict network policies can complicate deployment pipelines — automate policy as code.

This layered design ensures network isolation, authenticated encrypted traffic, and centralized management of keys and policies.