Multi-Cloud and Hybrid Cloud Architecture Questions
Designing systems that span multiple cloud providers or bridge cloud and on-premises. Covers cloud-agnostic abstraction, workload placement, cross-cloud networking and identity, data gravity, resilience across regions and providers, and the operational cost of avoiding lock-in versus the risk of accepting it. Includes multi-region resilience patterns.
HardSystem Design
54 practiced
Design a hybrid multi-cloud data platform: primary workloads in GCP, disaster recovery in AWS, and specialized compute in Azure. Discuss networking (interconnects, VPNs), identity federation, cross-cloud data replication, deployment automation, and how to test failover end-to-end.
Sample Answer
Requirements & constraints:
- Primary workloads and data lake in GCP (low-latency analytics, BigQuery/Dataproc)
- Cold/DR replica in AWS (S3, EMR, RDS replica)
- Specialized GPU/FPGA compute in Azure (Azure ML, NC-series VMs)
- RPO ~5–15 minutes for critical tables, RTO <1 hour for core pipelines
- Secure cross-cloud networking, consistent identity, automated deployments, testable failover
High-level architecture:
- GCP primary: Ingest -> Cloud Storage / Pub/Sub -> Dataflow/Dataproc -> BigQuery
- Cross-cloud replication plane: CDC streams + object replication -> AWS S3 (DR) and Azure Blob (specialized compute staging)
- Azure compute: Pull processed datasets from GCS/S3 via high-bandwidth links or use imported snapshots
- Control plane: CI/CD (Terraform + Terragrunt + GitOps), orchestration (Airflow/Composer or Cloud Composer with cross-cloud operators)
- Observability: Prometheus/Grafana, Stackdriver (GCP), CloudWatch (AWS), Azure Monitor, centralized logs in GCS and S3
Networking:
- Primary high-bandwidth links:
- GCP Dedicated Interconnect (or Partner Interconnect) to on-prem/partners
- AWS Direct Connect to same colocation and set up AWS Direct Connect + Transit Gateway
- Azure ExpressRoute to same colo
- Use the colo as an N-point fabric for private cross-cloud connectivity (lowest latency, highest throughput).
- If colo not available, set up IPsec VPNs as fallback between clouds but restrict for DR due to bandwidth/latency.
- Network topology:
- Each cloud uses its VPC/VNet with strict subnetting, private IP ranges, and Transit Gateways / VPC-NAT for controlled egress.
- Use route tables and firewall rules to permit only necessary CIDR ranges and ports.
- Use BGP for dynamic routes between Direct Connect / Interconnect / ExpressRoute for failover and path selection.
- Security: TLS, mTLS for service-to-service, private endpoints for storage (VPC Service Controls in GCP, S3 VPC endpoints, Azure Private Link).
Identity & access:
- Central IdP (recommend Okta or Azure AD as primary) with SAML/OIDC federation to GCP IAM, AWS IAM Identity Center (SSO), and Azure AD tenants.
- Use short-lived workload credentials:
- GCP Workload Identity Federation to allow services to assume roles in AWS and Azure without long-lived keys.
- AWS STS AssumeRole for service principals from GCP/Azure.
- Azure Managed Identities for accessing resources.
- Principle of least privilege via IaC-managed IAM policies and automated access reviews. Use ABAC or tags to scope access.
Cross-cloud data replication:
- For event/streaming data:
- Pub/Sub in GCP as primary; use a durable connector (Kafka Connect with Pub/Sub source or Managed Dataflow) to stream to Kafka clusters (Confluent or MSK) or directly to AWS Kinesis via connectors.
- For guaranteed semantics, use exactly-once sinks where possible or idempotent writes.
- For databases:
- Use CDC (Debezium) capturing from primary databases (Cloud SQL/Spanner / Postgres) into Kafka, then mirror topics to AWS consumers (Kafka MirrorMaker or Confluent Replicator).
- For object data:
- Use asynchronous object replication: GCS -> S3 replication via tools (rclone, custom Dataflow job, or Storage Transfer Service to S3).
- For snapshots/backups, push versioned Parquet/AVRO snapshots to both GCS and S3.
- For analytic tables:
- Use BigQuery scheduled exports to GCS in AVRO/Parquet; Data Transfer Service or Glue jobs to catalog into AWS Glue / Athena.
- Maintain a shared schema registry (Confluent Schema Registry or open-source) accessible across clouds.
Deployment automation:
- IaC: Terraform modules for each cloud, with a root-level orchestration using Terragrunt or CI pipelines.
- GitOps: Repos per environment (dev/stage/prod) with automated pipeline (GitHub Actions / Cloud Build / Jenkins) that runs plan/apply with policy checks (OPA/Gatekeeper).
- Secrets: Central secrets manager (HashiCorp Vault with auto-unseal) with cloud-specific secret engines; ensure ephemeral secrets for cross-cloud ops.
- Pipelines deploy:
- Infra (Terraform) -> Platform services -> Data pipelines (Airflow DAGs, Dataflow templates) -> Application code.
- Testing: Use automated integration tests in CI that spin up ephemeral infra in sandbox projects.
Failover & end-to-end testing strategy:
- Define clear failover plan (partial vs full): Data-only failover (serve reads from AWS) vs compute failover (run pipelines in AWS or Azure).
- Regular exercises:
- Canary replication tests: Verify CDC topics replicate within RPO SLA — monitor lag metrics and end-to-end digest checksums.
- Automated DR drills (monthly/quarterly):
- Stage 1: Failover dry-run (non-prod) — promote S3 copies and run test pipelines in AWS to validate transformations produce same hashes as GCP.
- Stage 2: Partial failover: Route specific analytics queries to AWS Athena using replicated datasets; validate correctness and latency.
- Stage 3: Full failover (scheduled): Redirect DNS (internal via Route53 / Cloud DNS cross-cloud) and promote IAM roles; execute smoke DAGs to rebuild materialized datasets and run data validations.
- Blue-green approach for failover: Keep DR infra warm (autoscaling min >0 for critical services) and continuously run parity jobs to ensure readiness.
- Verification steps post-failover:
- Record-level checksums and row counts vs baseline, schema validation, downstream consumer health checks, and SLA alerting thresholds.
- Run integration tests that validate downstream ML models and BI dashboards.
- Rollback plan: Automated scripts to revert DNS/routes, re-point consumers back to GCP, and reconcile resumed deltas using CDC offsets.
Observability & operations:
- Centralized metrics and alerting: export core metrics (replication lag, transfer throughput, error rates) to a central monitoring layer (Prometheus + Grafana, or push metrics to a single cloud monitoring ingestor).
- Tracing: Distributed tracing propagated across cross-cloud calls (OpenTelemetry) with sampled traces for critical flows.
- Playbooks & runbooks: Documented runbooks for failover, including pseudo-commands and escalation contacts.
Trade-offs & considerations:
- Using colo + direct connects increases cost but reduces latency and gives predictable bandwidth for large transfers vs VPNs.
- Replication consistency: Real-time CDC gives low RPO but adds complexity; snapshots simpler but slower.
- Keeping DR warm costs more but reduces RTO; cold backups are cheaper with higher RTO.
- Security complexity increases with multi-cloud; invest in automation for IAM and secret lifecycle.
Why this approach:
- Provides robust, testable replication paths (CDC + object replication), secure and performant network fabric (Direct Connect / Interconnect / ExpressRoute), unified identity, and repeatable automation for deployments and DR tests. It balances cost, complexity, and SLA requirements while allowing specialized compute on Azure without compromising primary GCP workloads.
That is every published Multi-Cloud and Hybrid Cloud Architecture question for Data Engineer so far. Browse the other topics in this category, or practice this one interactively.