Multi-Cloud and Hybrid Cloud Architecture Questions
Designing systems that span multiple cloud providers or bridge cloud and on-premises. Covers cloud-agnostic abstraction, workload placement across providers or environments, cross-cloud networking and identity federation, data gravity, infrastructure-as-code and centralized observability that span providers, and the operational cost of avoiding vendor lock-in versus the risk of accepting it. Also covers keeping a system correct once it spans providers: leader election, distributed transactions, rate limiting, and service discovery across cloud or cluster boundaries. Resilience patterns here are scoped to crossing a provider or on-prem/cloud boundary (for example failover from one provider to another, or from on-prem to cloud). Resilience across regions of a single provider, with no second provider or on-prem leg involved, is a different topic (multi-region architecture) and is out of scope here.
Design an end-to-end service mesh architecture that spans on-prem and multiple cloud providers to provide mTLS, consistent policy enforcement, traffic shifting for canary releases, and unified observability. Discuss control-plane placement (single vs federated), certificate distribution and rotation, latency considerations, and integration with existing security controls and firewalls.
Sample Answer
Direct answer
For a mesh spanning on-prem and multiple clouds, use a federated control-plane model, meaning one control-plane instance per cluster, each trusting a shared root certificate authority, rather than a single control plane governing every cluster remotely. A single remote control plane makes every cluster's mesh functionality depend on network reachability to wherever that control plane lives, which is exactly the kind of cross-boundary single point of failure a multi-cloud or hybrid design should avoid. Layer mutual TLS, policy enforcement, canary traffic shifting, and observability on top of that federated foundation, and treat the existing firewalls and security controls as something the mesh's cross-cluster gateways must integrate with explicitly, not bypass.
Structured elaboration
flowchart TB
subgraph OnPrem["On-prem cluster"]
DPO["Envoy sidecars"]
end
subgraph CloudA["Cloud A cluster"]
DPA["Envoy sidecars"]
end
subgraph CloudB["Cloud B cluster"]
DPB["Envoy sidecars"]
end
CP["Federated control plane (per-cluster istiod)"] -- "mTLS certs + policy" --> DPO
CP -- "mTLS certs + policy" --> DPA
CP -- "mTLS certs + policy" --> DPB
RootCA["Shared root CA"] --> CP
DPO <-- "cross-cluster mTLS" --> DPA
DPA <-- "cross-cluster mTLS" --> DPB
Control-plane placement: single versus federated
- Single control plane: one instance manages every cluster remotely. Simpler to operate day to day, one thing to upgrade and one place to look, but every cluster's mesh configuration and certificate issuance depends on network reachability to that one control plane. A network partition to the on-prem site, a common event in hybrid setups since the on-prem-to-cloud link is usually the least reliable path in the whole system, degrades mesh functionality everywhere, not just at the partitioned site.
- Federated control plane: each cluster runs its own control-plane instance, all trusting a shared root certificate authority so cross-cluster mutual TLS still works, with configuration synced rather than remotely enforced between them. More moving parts to operate, but each cluster keeps functioning locally, including issuing and rotating its own workload certificates, even if cross-cluster connectivity or another cluster's control plane is down. For a hybrid or multi-cloud spread specifically, where the on-prem link is the most fragile connection in the topology, federated is the safer default; a single control plane is reasonable only when every cluster sits within one provider's low-latency backbone, which is not the case being designed for here.
Certificate distribution and rotation
Issue a shared root certificate authority once, offline, and have each cluster's local control plane act as an intermediate authority that automatically issues short-lived workload certificates, often on the order of hours rather than the long-lived certificates typical of traditional public key infrastructure, to each sidecar. Automatic, short-lived rotation matters more in a hybrid spread than in a single cluster, because a manual renewal process that depends on a human noticing an expiry across three separate clusters and three separate calendars is a guaranteed eventual outage. Distribute the root authority's public certificate, never its private key, to every cluster at bootstrap, and store the root's private key offline or in a hardware security module, issuing intermediates rarely and deliberately, since a compromised root compromises trust across every cluster in the mesh at once.
Cross-cluster service discovery
Each cluster's control plane needs visibility into the other clusters' service registries so a service in the on-prem cluster can resolve and call a service in a cloud cluster by name. This is typically done either by each control plane watching the others' Kubernetes (the container-orchestration platform each cluster runs) API servers, which requires network reachability and credentials scoped read-only to service and endpoint resources, or via a dedicated east-west gateway that other clusters address instead of reaching into the cluster's internals directly. The east-west gateway approach is preferable across a hybrid or multi-cloud boundary specifically because it avoids exposing each cluster's full Kubernetes API server to the others, narrowing the integration surface to one well-defined gateway per cluster.
Mutual TLS and consistent policy enforcement
With the shared root authority in place, enforce mesh-wide policy that all cross-cluster traffic must use mutual TLS and reject plaintext, and define authorization policy, meaning which service identities may call which, centrally, synced to every cluster's local control plane so a service's access rules do not depend on which cluster evaluates the request.
Traffic shifting for canary releases
Canary a new version of a service by weighting traffic, for example 95 percent to the stable version and 5 percent to the canary, at the mesh layer, which works identically whether the two versions are in the same cluster or split across clusters, letting a team canary a cloud-hosted new version against an on-prem-hosted stable baseline using the same traffic-splitting configuration used within one cluster.
Unified observability
Aggregate telemetry, such as latency, error rate, and request volume, from every cluster's sidecars into one observability backend, tagged with cluster and cloud or on-prem origin as a label, so a latency regression can be attributed to a specific cluster or to the cross-cluster hop itself, rather than lost in an average across the whole mesh.
Latency considerations
Every cross-cluster call adds the network path's own latency, typically single-digit to tens of milliseconds for cloud-to-cloud traffic over a dedicated interconnect and more for on-prem depending on the link, on top of the mesh's own per-hop sidecar overhead, roughly single-digit milliseconds per proxy hop, so two hops for a cross-cluster call. Design service placement so latency-sensitive call chains stay within one cluster where possible, reserving cross-cluster calls for interactions that can tolerate the added round trip.
Integration with existing security controls and firewalls
The mesh's east-west gateways and control-plane synchronization traffic need explicit firewall rules, specific ports and specific source or destination cluster ranges, rather than a blanket allow between environments. Treat the mesh as a new, named flow through the existing security architecture, reviewed the same way any other cross-boundary traffic would be, not as something that operates below or outside the existing firewall and network-security posture.
Worked example
A retailer runs an on-prem cluster for a legacy inventory system that cannot move, a primary cloud cluster for customer-facing checkout, and a secondary cloud cluster in a second provider for fraud scoring, chosen for that provider's specific machine-learning tooling. They deploy a federated mesh: each cluster runs its own control plane trusting a shared root authority generated once and stored offline, and each cluster exposes one east-west gateway, with the two cloud clusters' gateways reachable over a dedicated interconnect and the on-prem gateway reachable over the existing site-to-site VPN. When rolling out a new checkout service version, they canary 10 percent of traffic to the new version within the primary cloud cluster first, avoiding a cross-cluster hop for the fastest feedback loop, then expand to 50 percent, reserving cross-cluster canarying for testing the checkout service against a cloud-hosted upgrade path from the legacy on-prem system. Observability dashboards tag every trace with its origin cluster, so when p99 latency on checkout rises, the team can immediately see whether it is the fraud-scoring cross-cluster call through the second provider's gateway or something local to the primary cluster.
Trade-offs and pitfalls
Choosing a single remote control plane for operational simplicity is the most common mistake in a hybrid spread specifically, because it looks fine in a demo and fails exactly when the on-prem link degrades, which is also when the team most needs the mesh's policy and observability to keep working locally. A second pitfall is exposing each cluster's Kubernetes API server directly to the others for service discovery instead of using a dedicated gateway, widening the security surface unnecessarily. A third is under-budgeting the per-hop sidecar latency and the cross-cluster network latency together; teams that benchmark only the mesh's overhead in a single, low-latency cluster and extrapolate to the hybrid topology are routinely surprised by the compounded number in production.
What is a service mesh and how can it be applied in multi-cloud or hybrid-cloud Kubernetes deployments? Describe two benefits and two operational challenges of deploying a service mesh across clusters in different providers.
Sample Answer
Direct answer
A service mesh is a dedicated infrastructure layer, usually implemented as lightweight proxies deployed alongside each service instance, that handles service-to-service communication concerns, such as encryption, retries, traffic routing, and observability, outside the application code, coordinated by a control plane. In multi-cloud or hybrid Kubernetes deployments, extending a service mesh across clusters gives every service the same secure, observable communication layer regardless of which provider or on-prem cluster it happens to run in, which is exactly the uniformity multi-cloud otherwise lacks.
Structured elaboration
What it does, concretely
Each service gets a sidecar proxy, meaning a small proxy process deployed alongside that service (often in the same pod), for example an Envoy proxy in Istio, that intercepts its network traffic. The proxies enforce policy, such as which services may talk to which and whether traffic must be encrypted, and report telemetry, such as latency, error rate, and request volume, to a control plane, without the application code needing to implement any of it itself.
Two benefits in a multi-cloud or hybrid context
- Uniform mutual TLS across cluster boundaries: mutual TLS, or mTLS, means both sides of a connection cryptographically verify each other, not just the client verifying the server. Services in an on-prem cluster and services in a cloud cluster get the same encrypted, authenticated communication without each provider's own, differently configured network security tooling being the thing holding it together.
- Consistent observability and traffic policy regardless of location: the same dashboards and the same retry and traffic-splitting rules apply whether a service call stays inside one cluster or crosses to another provider, so on-call engineers are not reasoning about three different networking models mid-incident.
Two operational challenges
- Control-plane placement and cross-cluster trust: deciding whether one cluster's control plane governs all clusters, or each cluster runs its own, federated via a shared root of trust with certificates that chain to a common certificate authority, is a real design decision with latency and blast-radius trade-offs, not a default that works out of the box across providers.
- Added latency and resource overhead: every service call now passes through two additional proxy hops, the sender's sidecar and the receiver's sidecar, which adds measurable latency and CPU and memory overhead per pod. Across a cross-cluster hop, that overhead compounds with the network path's own latency, and teams that do not budget for it are surprised when p99 latency, meaning the 99th-percentile response time, rises after mesh adoption.
Worked example
A company runs a customer-facing service on an on-prem Kubernetes cluster and a supporting fraud-scoring service on a cloud provider's Kubernetes cluster. Without a mesh, securing that cross-cluster call requires manually configuring mTLS certificates and a firewall rule for each new service pair, an approach that does not scale past a handful of services. With a mesh spanning both clusters and a shared certificate authority, every new service automatically gets mTLS to any other service in either cluster, and the platform team sees one latency dashboard showing the on-prem-to-cloud hop's contribution separately from each cluster's internal call time, which is exactly what is needed to tell whether a slowdown is the cross-cluster network path or the fraud-scoring service itself.
Trade-offs and pitfalls
Teams sometimes adopt a service mesh to solve a security or observability problem that a simpler point solution, such as a shared certificate for the two services that need to talk or an existing application performance monitoring tool, would have solved more cheaply; a mesh's value grows with the number of services and the number of cluster boundaries, and is proportionally more overhead for a small number of services in one cluster. The added-latency challenge above is also frequently under-measured before rollout; benchmark the actual per-hop overhead in a representative environment rather than assuming it is negligible.
IaC strategy for multi-cloud: Describe a practical Infrastructure as Code strategy that uses Terraform across AWS, Azure, and GCP. Cover module design, state backend architecture (including locking), secrets in IaC, and how to structure environments (dev/stage/prod) to enable safe changes and testing.
Sample Answer
Approach
Structure Terraform (a declarative infrastructure-as-code tool: you describe the desired end state in HCL,
HashiCorp Configuration Language, and it computes and applies the difference) around three separate
concerns that are easy to tangle together if you do not separate them from day one: reusable modules that
are cloud-specific underneath but expose a consistent interface, a state backend with real locking so two
engineers can never corrupt the same state file, and a directory-per-environment layout so a change to dev
cannot accidentally reach prod through a shared workspace.
Module design
One module per logical piece of infrastructure (a network, a database, a compute cluster), parameterized so
the same module works across environments by varying inputs, not by copying the module. For a genuinely
multi-cloud module, accept a cloud variable and use count or for_each to conditionally create the
AWS, Azure, or GCP resource, exposing one common set of outputs regardless of which cloud instantiated it,
so callers of the module do not need cloud-specific logic:
variable "cloud" {
type = string
validation {
condition = contains(["aws", "gcp", "azure"], var.cloud)
error_message = "cloud must be aws, gcp, or azure."
}
}
variable "cidr_block" { type = string }
variable "name" { type = string }
resource "aws_vpc" "this" {
count = var.cloud == "aws" ? 1 : 0
cidr_block = var.cidr_block
tags = { Name = var.name }
}
resource "google_compute_network" "this" {
count = var.cloud == "gcp" ? 1 : 0
name = var.name
auto_create_subnetworks = false
}
resource "azurerm_virtual_network" "this" {
count = var.cloud == "azure" ? 1 : 0
name = var.name
address_space = [var.cidr_block]
location = "eastus"
resource_group_name = "rg-${var.name}"
}
output "network_id" {
value = coalesce(
try(aws_vpc.this[0].id, null),
try(google_compute_network.this[0].id, null),
try(azurerm_virtual_network.this[0].id, null),
)
}
State backend architecture
Use a remote backend with native locking so concurrent applies fail loudly instead of corrupting state: an
S3 bucket (Amazon S3, AWS's object storage service) with versioning enabled plus a DynamoDB table (AWS's
managed NoSQL database, used here only to hold a single lock record) for locking is the common AWS-hosted
pattern (Azure Storage with a lease-based lock, or Terraform Cloud's built-in locking, work the same way
conceptually).
Split state per environment, never one shared state file for dev, stage, and prod, using a partial backend
configuration filled in at init time:
# envs/prod/main.tf
terraform {
backend "s3" {} # values supplied via: terraform init -backend-config=backend.hcl
}
# envs/prod/backend.hcl
bucket = "acme-tfstate-prod"
key = "envs/prod/network/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "acme-tf-locks"
encrypt = true
Secrets in IaC
Never write a secret literal into a .tf or .tfvars file that reaches version control. Reference secrets
by their non-secret identifier (an ARN, Amazon Resource Name, or a resource path) and pull the value at
plan/apply time through a data source, so the secret itself never touches the Terraform source tree:
data "aws_secretsmanager_secret_version" "tunnel_psk" {
secret_id = "arn:aws:secretsmanager:us-east-1:111111111111:secret:hybrid/tunnel-psk"
}
Terraform state can still contain the resolved secret value in plaintext, so the state backend's own access
control and encryption at rest matter as much as keeping secrets out of source files.
Environment structure
A directory per environment (envs/dev, envs/stage, envs/prod), each with its own backend
configuration and its own .tfvars, calling the same shared modules from modules/, gives every
environment an independent state file and an independent blast radius: a broken plan in dev cannot lock or
corrupt prod's state, and promoting a change from stage to prod is a deliberate terraform apply in a
different directory, not an implicit consequence of a shared workspace switch.
# envs/prod/main.tf
module "prod_network" {
source = "../../modules/network"
cloud = "aws"
cidr_block = "10.50.0.0/16"
name = "prod-hub-vpc"
}
Key points
Module interfaces stay stable even as the underlying cloud resource changes; state locking prevents the
single most common cause of a corrupted apply, two engineers running terraform apply at once; secrets
referenced by identifier, not embedded, keep the source tree safe to commit even though the state file still
needs its own protection; and directory-per-environment makes "which environment does this change affect"
an answerable question by looking at which directory you ran the command in.
Complexity and edge cases
Terraform's plan/apply cost scales with the size of the state file and the depth of the dependency graph in
that state, not with lines of HCL, so keeping state split per environment also keeps plan times bounded as
the estate grows. Edge cases worth naming: a state lock left stranded by a crashed CI job needs a documented
force-unlock procedure with an approval gate, since force-unlocking blind risks exactly the corruption
locking exists to prevent; a module renamed or refactored can orphan existing resources in state unless
moved blocks are used to tell Terraform the resource's new address; and a count-based conditional
resource, as in the module above, shifts every subsequent resource's index if an earlier one is removed,
which for_each avoids by keying resources on a stable name instead of a position.
terraform init, terraform fmt -check, and terraform validate all pass cleanly against the module and
environment configuration shown above; there are no cloud credentials in this environment, so plan/apply
output is not something this answer can execute or claim.
Design a global service discovery mechanism for ephemeral Kubernetes workloads running in clusters across multiple clouds and regions. Consider how services find each other, handle latency, deal with stale entries, and maintain security (authentication and authorization) across cluster boundaries.
Sample Answer
Direct answer
I would build global service discovery as a service mesh control plane that federates per-cluster local registries into one global view, rather than trying to run a single flat registry across three clouds: each cluster keeps a fast local registry for its own pods, a lightweight sync layer propagates short-lived leases (not permanent records) between clusters, and every cross-cluster call is authenticated with mutual TLS (mTLS) so cluster boundaries are a trust boundary, not just a network hop.
Structured elaboration
How services find each other: each pod registers itself with a local agent on join and deregisters (or lets its lease expire) on termination; this is the layer that must be fast because it is on every pod's lifecycle path. A separate, slower propagation layer syncs a summarized view (which services exist, their health, and their cluster) to the other clusters' control planes, so a caller in Cluster A's mesh sidecar first checks its local view before querying the global layer, keeping the common case (same-cluster call) cheap and the cross-cluster case (uncommon, but must work) correct.
Latency handling: rank candidate endpoints by measured round-trip latency, not by a static preference list, and default to same-region/same-cluster targets whenever a healthy local instance exists. Ephemeral workloads mean the set of valid endpoints changes constantly, so latency-aware routing has to be a continuous measurement (piggybacked on existing health checks or mesh telemetry), not a one-time configuration.
Handling stale entries: use short TTL (time-to-live) leases rather than permanent registration records; an entry that isn't refreshed within its TTL window is dropped automatically rather than requiring an explicit deregistration message that an abruptly killed pod will never send. Pair this with active health checking at the mesh sidecar level as a second line of defense: even a technically-not-yet-expired entry gets marked unhealthy and skipped if its last few health probes failed, which matters because a TTL long enough to tolerate normal network jitter is also long enough to route a few requests to a pod that already died.
Security across cluster boundaries: every cross-cluster call goes through mTLS with certificates issued by a shared root of trust (a central certificate authority the mesh control plane manages, with per-cluster intermediate certificates so a compromised cluster doesn't require rotating the global root). Authorization is enforced per-service, not per-cluster: a service in Cluster A being allowed to call a service in Cluster B does not imply every service in A can call every service in B, which is the natural but wrong assumption once you have federated the registries.
Worked example
A checkout service running in the AWS cluster needs to call a pricing service that exists in both the GCP and Azure clusters for redundancy. Its local mesh sidecar first checks whether a healthy pricing-service instance exists in the AWS cluster; if not, it queries the federated registry, gets back a latency-ranked list (GCP instance at 45ms measured p50, Azure instance at 130ms measured p50 from this cluster's vantage point), and routes to the GCP instance. Each pricing-service registration carries a 15-second TTL lease refreshed every 5 seconds; if three consecutive refreshes are missed (an abrupt pod kill during a node eviction, for instance), the entry disappears from the federated view within 15 to 20 seconds without any explicit deregistration message. The mTLS certificate presented by the GCP pricing-service instance is validated against the shared mesh root of trust, and a per-service authorization policy explicitly allows checkout.aws-cluster to call pricing.* but does not implicitly allow every other AWS-cluster service to do the same.
flowchart TB
subgraph AWS_Cluster["Cluster A (AWS, us-east-1)"]
SvcA["Service X pods"]
AgentA["Registry agent"]
end
subgraph GCP_Cluster["Cluster B (GCP, us-central1)"]
SvcB["Service X pods"]
AgentB["Registry agent"]
end
subgraph Azure_Cluster["Cluster C (Azure, eastus)"]
SvcC["Service X pods"]
AgentC["Registry agent"]
end
SvcA --> AgentA
SvcB --> AgentB
SvcC --> AgentC
AgentA -->|mTLS heartbeat, TTL lease| GlobalRegistry["Global service registry (multi-cluster mesh control plane)"]
AgentB -->|mTLS heartbeat, TTL lease| GlobalRegistry
AgentC -->|mTLS heartbeat, TTL lease| GlobalRegistry
Client["Caller in Cluster A"] -->|1 resolve Service X| LocalProxy["Local mesh sidecar"]
LocalProxy -->|2 query, prefer local| GlobalRegistry
GlobalRegistry -->|3 endpoint list ranked by latency| LocalProxy
LocalProxy -->|4 mTLS call| SvcB
Trade-offs and pitfalls
The core tension is propagation latency versus registry load: propagating every registration change to every cluster instantly keeps the global view fresh but means a busy cluster's churn floods the sync layer; batching or debouncing propagation reduces load but widens the window where a caller in a remote cluster sees a stale view. A TTL that's too short causes needless re-registration traffic and flapping under normal network jitter; a TTL that's too long means dead endpoints stay discoverable longer than the failure they're meant to route around. The security pitfall that's easy to miss is treating "the clusters trust each other" as equivalent to "every service trusts every other service": once a global registry exists, the temptation is to skip per-service authorization because the network-level trust is already established, which turns a single compromised low-value service into a lateral-movement path across all three clouds.
Cross-cloud deployment strategy: Describe a blue/green or canary deployment pattern that operates across three Kubernetes clusters in different providers. Explain how you would manage traffic shifting, schema migrations, and rollback while avoiding data corruption and maintaining customer experience.
Sample Answer
Direct answer
I would run a canary rollout coordinated by a global traffic-management layer sitting above the three clusters, gate schema migrations to be backward-compatible with both the old and new application version for the entire rollout window (never a hard cutover), and treat the three clusters as independent rollout targets that are promoted one at a time rather than in lockstep, so a bad canary in one cluster never becomes a bad canary in all three simultaneously.
Structured elaboration
Traffic shifting: a global load balancer or service mesh with cross-cluster awareness shifts a small percentage (commonly starting at 1 to 5%) of traffic to the new version in one cluster first, monitors error rate and latency against the baseline for a defined bake time, and only then increases the percentage or promotes to the next cluster. Canary (gradual percentage-based) is generally preferable to blue/green (instant full cutover) for a cross-cluster rollout specifically because it limits blast radius to a small slice of traffic in one cluster at a time, rather than blue/green's all-or-nothing cutover per cluster, which is higher-stakes precisely when you have three independent clusters that could each go wrong differently.
Schema migrations: the single hardest part of this pattern, and the part most rollout designs get wrong. A schema migration must be expand and contract: first deploy a migration that adds the new schema element while the application still works correctly against the old schema (expand phase), deploy the new application version that can read/write both old and new schema shapes, let it bake and roll out fully, then only after the old application version is completely retired everywhere, run a second migration that removes the old schema element (contract phase). Never deploy a migration that breaks the currently-running old version, because during a canary rollout, the old version and new version are both live simultaneously by design, often for hours, and across three separately-deployed clusters that may not all be at exactly the same migration state at the same moment.
Rollback: because schema changes are expand/contract, rolling back the application to the old version is always safe as long as the contract migration hasn't run yet, the old version simply ignores the new schema elements it doesn't know about. This is the actual payoff of the expand/contract discipline: rollback becomes a traffic-shift operation, not a data-recovery operation.
Avoiding data corruption: the risk is a new-version instance and an old-version instance both writing to the same row/document concurrently in incompatible ways. Guard against this with additive-only schema changes during the bake window (new nullable columns, new optional fields, never renaming or repurposing existing ones) and, where the two versions genuinely cannot coexist safely (a field whose meaning is changing, not just its presence), consider dual-writing or a versioned-record approach until the old version is fully retired.
Maintaining customer experience: bake time and automated rollback criteria (error rate or latency crossing a threshold triggers automatic traffic-weight reduction back to zero for the canary, without waiting for a human to notice) matter more here than in a single-cluster canary, because three clusters means three chances for something to go subtly wrong, and a human watching three dashboards simultaneously is a weaker safety net than an automated guardrail per cluster.
Worked example
Rolling out v2 of an API across EKS, GKE, and AKS clusters: the pipeline first deploys v2 to 5% of pods in the EKS cluster only, with the global load balancer weighting 5% of EKS-bound traffic to those pods; a monitoring gate watches error rate and p99 latency (the 99th-percentile response time: the value 99% of requests are faster than, a common way to track worst-case-leaning tail latency instead of just the average) against the v1 baseline for a 15-minute bake window. If the gate passes, weight increases to 25%, then 100% within EKS, at which point the pipeline moves to GKE and repeats the same staged rollout independently, then finally AKS. Throughout this entire process (which might span hours across all three clusters), the database schema is in its expand phase: a new column added in a prior migration is nullable and ignored by v1, populated by v2, so v1 pods still running in GKE and AKS while v2 is fully live in EKS continue working correctly against the same shared database. Only after v2 reaches 100% in all three clusters does a separate, later migration drop the old column that v1 used and v2 has fully superseded.
Trade-offs and pitfalls
The most common and most damaging mistake is a migration that assumes the rollout is instantaneous and fully synchronized across clusters, dropping or renaming a column the moment the "new" version is deployed anywhere, which breaks every old-version pod still serving traffic in the clusters that haven't been promoted yet, exactly the multi-cluster version of the classic single-cluster canary schema mistake, just with a longer exposure window because cross-cluster rollouts take longer end to end. A second pitfall is promoting all three clusters in lockstep to save time; this feels efficient but means a canary issue specific to one cluster (a resource limit misconfigured only in that cluster's manifests, for instance) surfaces simultaneously everywhere instead of being caught and stopped in the first cluster before it ever reaches the other two. The trade-off of doing this safely is time: a fully staged, one-cluster-at-a-time canary with real bake windows is slower than a simultaneous rollout, and that slowness is the actual safety mechanism, not an inefficiency to optimize away.
Unlock Full Question Bank
Get access to all 13 Multi-Cloud and Hybrid Cloud Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.