Multi-Cloud and Hybrid Cloud Architecture Questions

Role of BGP in hybrid cloud architectures
BGP is the control-plane protocol that exchanges IP prefixes and reachability between your on‑prem network and cloud providers. It enables dynamic failover, scalable multi‑path connectivity, and consistent routing policies across Direct Connect / ExpressRoute and site‑to‑site VPNs.

BGP with Dedicated Interconnects (Direct Connect / ExpressRoute)

• Typically uses eBGP sessions between your edge ASN and the cloud provider’s ASN over the dedicated link.
• Advertise your on‑prem prefixes and accept provider prefixes (VPC/VNET routes).
• Supports route policies, prefix limits, and communities for traffic engineering (e.g., prefer specific locations or failover paths).

BGP with Site‑to‑Site VPNs

• BGP runs over IPsec tunnels (dynamic routing). Commonly eBGP for different ASNs; some designs use iBGP if you aggregate behind a router.
• Enables automatic route updates when tunnels flap and supports active/standby or active/active designs.

ASNs, eBGP vs iBGP

• ASN = Autonomous System Number identifying an administrative domain (public or private ASNs available).
• eBGP: peer between different ASNs (used with cloud/provider links). Next‑hop typically set to peer.
• iBGP: peers within same ASN (used to distribute learned routes inside an AS). Requires full mesh or route‑reflectors to avoid split‑horizon/hidden route issues.

Route advertisement & selection behavior (practical points)

• Advertise only intended prefixes; apply filters and max‑prefix.
• Selection order (most relevant): Weight (vendor local), Local Preference, Local‑AS path length (shorter AS_PATH), Origin type (IGP < EGP < Incomplete), MED, eBGP over iBGP, IGP cost to next‑hop, oldest route, router ID.
• eBGP routes preferred over iBGP — important when you have both VPN and Direct Connect: you can prefer Direct Connect by setting higher local‑pref or weight.
• Next‑hop reachability: ensure IGP or static routes inside AS to BGP next‑hops, or use next‑hop-self on eBGP/iBGP as appropriate.
• Use communities, MEDs, and local‑pref for traffic engineering and failover between multiple cloud regions or links.

Example: Use eBGP on Direct Connect with ASN 65000 ↔ provider ASN; set local‑pref higher on Direct Connect routes and lower on VPN routes so traffic prefers the private interconnect, falling back to VPN if DC fails.

Learning focus: secure peering, correct ASN choices, route filters, next‑hop handling, and explicit route policies to control hybrid path preference and avoid routing loops.

Clarify requirements & constraints

• Global production clusters in AWS and GCP; central auth service accessible cross-cloud; low latency where feasible; no single-cloud dependency; minimize cross-cloud egress; strong service discovery/authentication; Kubernetes-native.

High-level topology

• Each cloud: VPC/VPC-Native (GCP) per environment with private GKE/EKS clusters.
• Private inter-region backbone: use cloud-to-cloud VPNs + one redundant high-throughput transit (AWS Transit Gateway <-> GCP Cloud Router via VPN/Partner interconnect). Implement two independent paths (different providers) to avoid single-cloud dependency.
• Deploy a small “control plane” of auth service replicated in both clouds (active-active) behind private load balancers; use geo-aware DNS or internal DNS with consistent service records.

Latency & egress cost strategy

• Replicate stateful auth caches in both clouds; prefer in-cloud calls for most traffic. Route local cluster calls to local replica via service discovery. Cross-cloud only for cache-miss or admin sync.
• Use Cloud Interconnect or Partner Exchanges where high-volume cross-cloud traffic is unavoidable to reduce egress pricing.
• Implement traffic steering: annotate services/pods to prefer local endpoints; use service mesh locality routing.

Service mesh considerations

• Use a mesh that supports multi-cluster/multi-cloud federation (e.g., Istio with multicluster multi-network or Linkerd with federation).
• Features required: mTLS, identity federation (SPIFFE/SPIRE), traffic splitting by locality, circuit breaking, and observability.
• Mesh control plane: federated controllers per cloud with a lightweight global control plane for policy sync; avoid single control plane placed in one cloud.

Service discovery & authentication

• Use SPIFFE IDs issued by a cross-cloud trust domain (central SPIRE cluster with failover replicas in each cloud) for workload identities.
• Service discovery: combine Kubernetes EndpointSlices for intra-cloud and a global service registry (e.g., Consul or ExternalDNS + internal DNS zones replicated) for cross-cloud; mesh sidecars enforce discovery validation.

Network policy & security posture

• Kubernetes NetworkPolicies (Cilium or Calico) enforcing least-privilege pod-to-pod flows; L7 policies at mesh sidecars for service intent.
• Perimeter: restrict ingress via private LB, deny public access; use egress/NAT gateways with logging and flow controls.
• Mutual TLS everywhere, RBAC for control planes, and centralized secrets via HashiCorp Vault replicated cross-cloud.

Resiliency & trade-offs

• Pros: low-latency local-first, cryptographically strong auth, no single-cloud control plane.
• Cons: complexity in mesh federation and state replication; operational overhead and potential eventual consistency. Balance by limiting cross-cloud synchronous calls and using async replication for non-critical state.

Implementation checklist

• Establish transit connectivity and redundancy, deploy SPIRE replicas, mesh per-cluster with federated policies, replicate auth service, configure locality-aware routing, enforce CNI network policies, monitor egress and optimize interconnect capacity.

Overview — role perspective
As a Network Engineer I collect telemetry that gives visibility into performance, capacity, health and security across on‑prem, hybrid links and multiple cloud providers. Below are essential metrics and the typical data sources.

Performance

• Latency: RTT per hop / path (probe/SLA, ICMP/tcp/HTTP synthetic tests). Source: active probes, cloud monitoring (CloudWatch/Stackdriver), SD-WAN telemetry.
• Jitter: variation in inter‑packet delay for real‑time traffic. Source: RTP/voice probes, SD‑WAN, cloud VPN metrics.
• Packet loss: per path and per interface. Source: active tests, SNMP counters delta, VPC Flow Logs for cloud flows.

Capacity / Utilization

• Link/interface utilization (bps, %) and errors. Source: SNMP ifOut/ifIn, sFlow, NetFlow/IPFIX, cloud NIC metrics.
• Queue depths, buffer drops. Source: device telemetry, telemetry streaming (gNMI), cloud NIC stats.

Health / Availability

• BGP session state, route churn, prefix flaps. Source: router/switch telemetry (SNMP, gNMI), BGP monitoring tools, cloud route table metrics.
• VPN/Direct Connect status and throughput. Source: cloud vendor APIs, SD‑WAN controllers.

Security signals

• Unusual flows, top talkers, geo/IP anomalies, port scans, DDoS spikes. Source: NetFlow/IPFIX, VPC Flow Logs, IDS/Firewall logs, cloud WAF/DDoS metrics.

Data-source mapping (quick)

• NetFlow/IPFIX / sFlow: rich flow context, volume/top talkers, protocol distribution.
• VPC Flow Logs / Cloud Flow Logs: east‑west cloud flows, source/dest, bytes, action.
• SNMP / gNMI / Telemetry streaming: interface counters, errors, BGP state, device health.
• Cloud provider metrics (CloudWatch/Azure Monitor/GCP): VPN, load balancer, NIC latency, throughput, health checks.
• Active probes / synthetic monitoring: end‑to‑end latency, jitter, loss from user perspective.

Why these matter
Combining flows + device counters + cloud metrics + active probes lets you correlate symptoms (high latency) to causes (congestion, route changes, security events) and prioritize remediation and capacity planning.

Approach — requirements & scope

• Identify regulated workloads (CDE for PCI, personal data for GDPR) and map where they live (on‑prem, AWS/Azure/GCP regions). Capture data residency constraints and retention rules up front.

Network segmentation design

• Perimeter: NGFWs + IDS/IPS between Internet and DMZ; WAF for app tiers.
• Macro segmentation: separate physical/virtual networks per trust zone — CDE, non‑CDE, management, partner/VPN.
  • On‑prem: VRFs + VLANs per zone; edge ACLs + stateful firewall.
  • Cloud: separate VPCs/Virtual Networks per zone + route tables, security groups, service endpoints.
• Microsegmentation: host/instance-level controls (cloud security groups, host firewall, or east‑west firewall like VMware NSX, Calico, Guardrails).
• Zero Trust: least privilege routing, explicit allow rules, MFA for admin flows.

Data residency & encryption

• Enforce region restrictions with IAM and org policies (AWS Organizations SCPs, Azure Policies).
• Encrypt data in transit (TLS 1.2+/mTLS for service-to-service) and at rest (cloud KMS/HSM, LUKS/BitLocker on‑prem).
• Use customer-managed keys stored in HSM for regulated data; rotate keys and log key ops.

Logging, monitoring & evidence

• Centralize logs: VPC Flow Logs, NetFlow, firewall logs, host logs → SIEM (Splunk/ELK/Sumo) with immutable storage and retention per regulation.
• Generate artifacts for auditors: flow logs demonstrating no cross‑zone traffic, firewall rule sets, route tables, VRF configs, ACLs, IAM policies, KMS audit trails, certificates of key origin (HSM), and change-control tickets.
• Periodic segmentation validation: automated flow tests, internal/external segmentation scans, and reportable test results (PCI segmentation testing).

Separation of environments

• Strict dev/test/prod separation via separate VPCs, subnets, and accounts/projects; CI/CD pipelines enforce promos across boundaries; no shared credentials or network paths.

Operational controls & evidence

• Network device configs under version control; signed change requests, scheduled backups, and config hashes.
• Audit evidence pack: network diagrams, config extracts, flow log queries showing deny results, SIEM alerts, segmentation test reports, key rotation logs, and DPA/SLA documents.
• Continuous compliance: automated policy checks (Terraform Sentinel, Cloud Custodian), periodic pen tests, and compliance scans.

This design satisfies PCI/GDPR by isolating regulated zones, enforcing residency, strong crypto, exhaustive logging, and producing concrete, queryable evidence for auditors.

Active-active means multiple regions serve traffic at the same time. It improves availability and can reduce failover time, but it usually requires more complex routing, data replication, and session handling.

Active-passive means one region serves traffic while another stays on standby. It is simpler to operate and easier to reason about, but failover is slower and the standby may be less well exercised.

How I decide

• Choose active-active when the service needs very high availability, can tolerate the complexity, and the data layer can support replication or regional sharding.
• Choose active-passive when the workload is more stateful, the business can accept a short recovery window, or simplicity is more important than maximum utilization.

Rule of thumb
For a business-critical customer-facing service, I look at RTO, RPO, session state, and operational maturity. If the team can test failover regularly and handle distributed traffic safely, active-active is attractive. If not, a well-tested active-passive design is often the safer first step.