Cloud Security Architecture Questions
Designing and reasoning about the security posture of cloud and hybrid infrastructure: the shared responsibility model, network segmentation and boundary design, multi-account and multi-region security architecture, workload identity as an architectural choice, threat modeling a cloud architecture, cloud-specific attack vectors and mitigations, defense-in-depth control selection, secure cloud deployment patterns, and continuous cloud risk assessment and posture. IAM policy authoring, role/trust-policy mechanics, and secrets/credential lifecycle belong to identity-and-access-management; logging-pipeline design and SIEM/detection-rule engineering belong to security-monitoring-and-detection; encryption-key-management mechanics (KMS/CMK/BYOK) belong to data-protection-and-encryption; compliance-framework mapping (SOC2, PCI-DSS, HIPAA, GDPR) belongs to compliance-frameworks-and-certification-standards. This topic keeps identity, logging, or encryption content only when it is one ingredient inside a genuinely multi-control cloud-hardening question, not as a standalone ask.
You receive a penetration test report noting: (a) publicly accessible object storage buckets with sensitive files, (b) overly permissive CORS policies on an API gateway, and (c) a Lambda function with a wide IAM policy. Prioritize remediation actions, justify trade-offs between speed and production impact, and propose controls to prevent recurrence and to validate fixes across environments.
Sample Answer
Direct answer
Fix the public bucket first: it requires no attacker skill and data is exposed the moment it exists. The over-broad Lambda IAM (Identity and Access Management) policy is second, because it is the finding with the largest blast radius once any foothold exists. The permissive Cross-Origin Resource Sharing (CORS) policy on the API gateway is third: on its own it needs a victim's browser to be useful to an attacker, and its real danger is usually in combination with the other two, not in isolation.
Structured elaboration
| Finding | Exploitability | Impact if left unaddressed | Immediate low-risk action | Full remediation | Production risk of the fix |
|---|---|---|---|---|---|
| (a) Public buckets | Trivial: any unauthenticated actor with the bucket name or a scanner | Direct data exposure right now, no further steps needed | Enable Block Public Access at the bucket and account level, after checking access logs for legitimate public-read traffic | Private bucket, serve any genuinely public content through a content delivery network (CDN) with Origin Access Control, or issue short-lived pre-signed URLs for one-off access | Low if a log check confirms nothing legitimate depends on public reads; otherwise a CDN migration is needed first |
| (c) Wide Lambda IAM policy | Requires a foothold (code execution or event injection into the function) | Turns one function compromise into an account-wide privilege-escalation path | Generate a policy from the function's actual CloudTrail activity (IAM Access Analyzer policy generation) as a comparison baseline, do not flip yet | Replace the wildcard policy with the generated least-privilege policy, scoped by resource ARN, deployed behind a canary period | Medium: a low-frequency legitimate call path can be missed by activity-based analysis; needs a shadow/monitoring window before full cutover |
| (b) Permissive CORS on the API gateway | Requires a victim to visit an attacker-controlled page while authenticated | Lets a malicious site make privileged cross-origin requests using the victim's session, amplified by whatever the wide IAM policy or public data already exposes | Restrict Access-Control-Allow-Origin from a wildcard to an explicit allow-list of the known frontend origins | Same allow-list enforced in the API gateway configuration itself (not just application code) plus a check that Access-Control-Allow-Credentials: true is never paired with a wildcard origin | Low: allow-listing known origins rarely breaks a legitimate frontend, but a missed origin (a staging domain, a partner integration) causes a visible break, so an inventory pass first avoids a second incident |
Worked example
A realistic 5-day remediation sequence for this exact report:
- Day 0, first hour: confirm via S3 server access logs (or CloudTrail data events) that no legitimate service depends on the bucket's public read, then flip Block Public Access. This is non-disruptive because it is reversible in seconds if something breaks, and the finding's exploitability was the highest of the three.
- Day 0, same day: capture the current CORS configuration, replace the wildcard origin with the known production and staging frontend origins, and deploy behind a feature flag so it can be reverted without a full redeploy if a missed origin surfaces.
- Day 1: run IAM Access Analyzer's policy generation against 90 days of the Lambda function's CloudTrail activity to produce a scoped candidate policy; diff it against the current wildcard policy and flag every action the function legitimately used.
- Day 2 to 4: deploy the scoped policy to a canary alias or a staging copy of the function, replay representative traffic (including any known rare code paths, such as a monthly batch job) against it, and watch for
AccessDeniederrors. - Day 5: cut the production alias over to the scoped policy once the canary period shows no denied calls, and archive the wildcard policy version rather than deleting it, so a fast rollback exists if something in production diverges from the sampled traffic.
Trade-offs and pitfalls
- Speed versus production impact is not a straight line. The public bucket fix is both the highest priority and the lowest risk to flip immediately; the IAM fix is the opposite (real but lower immediate exploitability, real risk of breaking a legitimate rare call path if scoped from an incomplete activity sample). Sequencing by exploitability first and reversibility second, rather than by "IAM is scary so do it last," is what keeps the team from either leaving the bucket open too long or breaking production by rushing the IAM change.
- Wide IAM policies exist because scoping is tedious, not because anyone chose them deliberately. Preventing recurrence means making the scoped path the path of least resistance: a CI (continuous integration) gate that runs policy-as-code checks (Open Policy Agent/Conftest, or a managed rule set) against every Terraform or CloudFormation change, blocking wildcard actions or resources before merge.
- CORS misconfiguration is easy to fix wrong. Allow-listing origins from memory instead of from an inventory of every legitimate caller (including a partner integration or an internal tool) causes the second incident: a real caller breaks silently, and the fastest fix under pressure is often to widen the origin back to a wildcard, undoing the remediation.
- Validate fixes the same way across every environment, not just production. A Config rule or CSPM (Cloud Security Posture Management) check that only runs against the production account will let the same misconfiguration ship again from a developer copying dev or staging Terraform into a new module; the detection gate belongs in the pipeline that produces the IaC (Infrastructure as Code), applied identically to every environment's plan.
- Preventing recurrence needs an organization-level guardrail, not just a per-account fix. A Service Control Policy (SCP) denying changes to S3 Block Public Access settings, paired with a scheduled drift-detection rule, catches the case where a well-meaning engineer reverses today's fix six months from now through the console.
Perform a threat modeling exercise for a large-scale streaming pipeline (e.g., Kafka or managed equivalent). Identify the highest-risk attack vectors across the producer, broker, and consumer layers, and propose mitigations and detection controls for each.
Sample Answer
Direct answer
A large-scale streaming pipeline (Kafka or a managed equivalent) has three layers with genuinely different threat profiles, producers (where data enters), brokers (where it is held and distributed), and consumers (where it is read and acted on), and the highest-risk vectors at each layer are different in kind, not just in severity: producer risk centers on what gets written, broker risk centers on who can reach and control the cluster itself, and consumer risk centers on what a compromised or malicious reader can do with what it consumes.
Structured elaboration
Producer layer: highest-risk vectors.
- Data poisoning. A compromised or malicious producer writes malformed, false, or adversarially-crafted records into a topic; because downstream consumers and stream-processing jobs generally trust that a message came from a legitimate producer once it is in the topic, poisoned data can propagate through every downstream system before anyone notices the source was compromised. Mitigation: schema validation enforced at write time (a schema registry rejecting a record that does not conform), and per-producer identity so a specific compromised producer's writes can be traced and, if needed, the topic partition it wrote to can be examined for the exact time range of compromise.
- Producer credential compromise. A leaked producer credential (an API key, a client certificate) lets an attacker write directly to the cluster with the legitimate producer's own authorization. Mitigation: short-lived, frequently-rotated producer credentials rather than long-lived static ones, and per-producer authorization scoped to only the specific topics that producer legitimately writes to, so a compromised producer credential cannot write to an unrelated, more sensitive topic.
Broker layer: highest-risk vectors.
- Unauthorized administrative access to the broker cluster itself. Broker administrative access (creating or deleting topics, modifying retention or replication configuration, or the underlying host access to a self-managed cluster) is the highest-leverage compromise in the entire pipeline, since it can affect every topic and every producer/consumer relationship at once, not just one data flow. Mitigation: the narrowest possible administrative access, scoped by role and audited continuously, following the same least-privilege discipline used throughout this domain, with a managed broker service (reducing the host-level attack surface entirely) preferred over self-managed brokers where the operational trade-off allows it.
- Inter-broker and client-broker traffic left unencrypted or unauthenticated. Traffic between brokers, and between clients and brokers, that does not enforce Transport Layer Security (TLS) and mutual authentication is interceptable or spoofable on the underlying network. Mitigation: TLS for all broker-to-broker and client-to-broker traffic, with mutual TLS (mTLS) or an equivalent strong authentication mechanism (Simple Authentication and Security Layer (SASL) with a strong mechanism) required for every client connection, not an optional configuration.
Consumer layer: highest-risk vectors.
- Over-broad consumer authorization. A consumer granted read access to more topics than its actual function requires can read data (including sensitive data flowing through an unrelated topic) it has no legitimate need to see; this is the consumer-side mirror of the producer-side scoping issue, and it matters specifically at scale, where consumer group sprawl over time tends to accumulate broader access than any individual consumer was originally provisioned with. Mitigation: per-consumer, per-topic least-privilege authorization, reviewed periodically rather than granted once and left unexamined.
- Replay and offset manipulation. A consumer (or an attacker who has compromised a consumer's credentials) can manipulate its own committed offset to re-read historical data it should only have consumed once, or, in a system that treats message consumption as a trigger for a side effect (a payment being processed, for instance), replay old messages to trigger that side effect again. Mitigation: idempotent consumer-side processing (designing the downstream action to be safe even if the same message is processed twice), and monitoring for anomalous offset resets or backward-jumping consumer positions as a detection signal independent of the idempotency safeguard.
Cross-layer detection controls
Beyond the per-layer mitigations above, two detection controls span all three layers: continuous audit logging of every administrative action (topic creation/deletion, access-control changes, offset resets) shipped to a centralized, separate log destination, consistent with the centralized-logging pattern used throughout this domain; and per-identity behavioral baselining (a specific producer's typical write volume and topic set, a specific consumer's typical read volume and topic set), flagging a deviation, a producer suddenly writing to a topic it has never written to before, or a consumer's read volume spiking well beyond its established baseline, as an anomaly worth investigating regardless of which specific layer or vector caused it.
Worked example
A financial services streaming pipeline processes transaction events. A compromised producer credential (a leaked API key from a misconfigured logging pipeline) is used to write malformed transaction records directly into the transactions topic. Schema validation at write time rejects most of the malformed records outright, but a subset that happens to satisfy the schema's structural requirements while carrying adversarially-incorrect values passes through; per-producer behavioral baselining flags the anomaly within minutes, since this producer's typical write volume is a small fraction of the burst the compromised credential generated. The security team isolates the compromised credential, and because producer authorization was scoped to only the transactions topic specifically (not broader), the attacker's reach never extended to the pipeline's other topics even during the window before detection.
Trade-offs and pitfalls
- Schema validation catches structurally malformed data but not adversarially valid data (values that pass every schema check while being substantively false or malicious), which is exactly the residual risk the worked example's "subset that passes through" represents. Schema validation and behavioral baselining are complementary, not redundant, precisely because they catch different halves of the same producer-layer risk.
- Broker administrative access is the single highest-leverage compromise in this entire threat model, and it is also the layer most often under-scrutinized relative to producer and consumer access, since day-to-day attention tends to focus on data flowing through the system rather than on who can reconfigure the system itself. The mitigation here deserves proportionally more rigor than either the producer or consumer layer alone, given its blast radius.
- Idempotent consumer-side processing is the correct architectural response to replay risk, but it requires deliberate design at the point every downstream side effect is implemented, not a bolt-on fix; a system built without idempotency in mind from the start is a materially larger retrofit than one designed for it from the beginning, which is why this needs to be a day-one architectural decision, not a response to a discovered replay incident.
- Per-consumer and per-producer authorization scoping tends to erode gradually over time as new consumers and producers are added under delivery pressure, each individually granted "just this one topic, temporarily broader than ideal"; periodic access review, not a one-time provisioning decision, is what keeps the least-privilege posture this threat model depends on actually real over the pipeline's operational lifetime.
How would you architect network segmentation and access control in a high-regulation environment where data residency and regional compliance requirements differ per customer? Explain account design, VPC placement, routing, and how to enforce region-specific controls.
Sample Answer
Direct answer
Network segmentation and access control for a high-regulation environment with per-customer data residency means the account boundary, not just the network boundary, has to carry the residency guarantee: each customer's data lives in an account (or a set of accounts) scoped to their specific regulatory region, and region-specific controls are enforced through both account placement and Service Control Policies (SCPs) that make the residency rule structurally difficult to violate, not just documented as a design intent.
Structured elaboration
Account design. One account (or a small set of accounts, following the base multi-account pattern of separating production, security, and shared services) per customer per region, rather than a shared multi-tenant account serving customers with different residency requirements. This is a stronger commitment than the base multi-account design used for a single-region product, since here the account boundary is doing double duty: blast-radius isolation between customers, and legal/regulatory boundary enforcement between regions.
VPC placement. Each customer's account provisions its virtual private cloud (VPC) only in the region their contractual and regulatory requirement specifies; an SCP attached at the organizational-unit level for that customer's regulatory grouping denies the creation of any resource, not just a VPC, but any regional resource generally, outside the approved region list, so a resource cannot be accidentally provisioned in the wrong region even by an authorized administrator acting in good faith but working from a stale runbook.
Routing. No default cross-region, cross-customer routing exists; if a shared service genuinely needs to reach multiple customers' regions (a centralized billing system, for instance, which does not itself hold regulated customer data), that service's own account has narrowly-scoped, individually-reviewed connectivity to each customer account, rather than customer accounts being able to reach each other or a broad shared network directly.
Enforcing region-specific controls. Beyond the resource-creation SCP, use the organization's account-vending process (an automated account factory) to bake the correct region restriction into every new customer account at creation time, rather than relying on a human to correctly configure it manually for each new customer; a per-region-grouped organizational unit structure, with SCPs attached at the organizational-unit level rather than per-account, means the restriction is inherited automatically as new customer accounts are added to the correct group, and a misclassification (a new customer account accidentally placed in the wrong organizational unit) is itself a detectable, auditable event rather than a silent gap.
Worked example
A SaaS platform onboards a new customer in the European Union (EU) whose data must remain in the EU under a General Data Protection Regulation (GDPR)-driven contractual requirement, alongside an existing customer in Asia-Pacific (APAC) with a similar but regionally distinct requirement. The account-vending process creates the EU customer's account inside an "EU-regulated-customers" organizational unit, which carries an SCP denying resource creation outside EU regions; the APAC customer's account sits in a separate "APAC-regulated-customers" organizational unit with its own region-restricting SCP. A platform engineer, following a runbook written for the APAC customer, accidentally attempts to apply the same Terraform module to the EU customer's account, specifying an APAC region; the SCP denies the resource creation outright, regardless of the engineer's own IAM permissions, which would otherwise have been sufficient to create the resource had the account not carried this specific organizational guardrail. The mistake is caught structurally, at the moment of the attempted action, rather than discovered later during an audit.
Trade-offs and pitfalls
- Per-customer, per-region account provisioning does not scale for free; it multiplies the number of accounts that need consistent baseline security configuration (logging, guardrails, patching) proportional to the number of customers, not just the number of regions. This is exactly why the account-vending automation matters as much as the account-design decision itself: manually configuring dozens or hundreds of per-customer accounts consistently is not realistic at any meaningful customer count, and inconsistency in that baseline is where a residency guarantee actually breaks down in practice.
- A shared cross-customer service (billing, in the worked example) is a deliberate, narrow exception to the isolation model, and its own access needs more scrutiny, not less, precisely because it is the one component with legitimate reach into multiple regulatory boundaries at once. Treating it as "just infrastructure" and under-scoping its access undermines the isolation the rest of the design carefully enforces.
- SCPs enforce where a resource CAN be created; they do not, on their own, guarantee that data already correctly placed never gets copied or replicated somewhere else by an application-level process (a cross-region backup misconfigured to target the wrong destination, for instance). The account and SCP design is the structural foundation, but application-level data-flow review is still needed to confirm nothing inside the correctly-placed account itself creates an unintended cross-region copy.
- Organizational-unit-level SCP inheritance is powerful specifically because it removes per-account configuration as a point of failure, but it depends entirely on new accounts being correctly classified into the right organizational unit at creation time; a misclassification at that one step (unlike a subsequent attempted region violation, which the SCP catches) is not itself caught by this design and needs a separate validation step in the account-vending process.
System design: Architect a secure, multi-account VPC/virtual-network architecture for a global e-commerce company. Requirements: public web tier, private app and DB tiers across 3 regions; separate prod/stage accounts; a shared-services account for NAT, logging, and patching; integration with CDN/WAF and DDoS protection; and centralized monitoring. Provide a high-level diagram and justify segmentation, routing, transit architecture (transit gateway or hub), IAM boundaries, and flow-log placement.
Sample Answer
Direct answer
A secure, multi-account VPC (Virtual Private Cloud) architecture for a global e-commerce company spanning three regions has to compose two ideas, per-tier network segmentation (public web, private app, private database) and multi-account isolation (separate production and staging, a dedicated shared-services account, a dedicated security account), into one design where a transit gateway is the connective tissue holding it together, not an afterthought bolted on once each region's network was already built independently.
Structured elaboration
flowchart TB
CDN["CDN + WAF + DDoS protection (global edge)"]
CDN --> R1["Region 1: prod account VPC"]
CDN --> R2["Region 2: prod account VPC"]
CDN --> R3["Region 3: prod account VPC"]
Shared["Shared-services account: transit gateway hub, NAT, logging, patching"]
R1 <--> Shared
R2 <--> Shared
R3 <--> Shared
Stage["Stage account (isolated, separate transit attachment)"]
Shared <--> Stage
Shared --> LogArchive[("Log-archive account: flow logs, CloudTrail")]
Security["Security account: centralized findings + monitoring"]
R1 -.->|"read-only findings"| Security
R2 -.->|"read-only findings"| Security
R3 -.->|"read-only findings"| Security
Account structure. A production account per region (or, depending on scale, one production account spanning all three regions with per-region VPCs; the diagram above shows the per-region-account variant for the strongest blast-radius isolation), a separate stage account entirely isolated from production with its own transit-gateway attachment, a shared-services account hosting the transit gateway hub, NAT gateways, and centralized patching tooling, a security account serving as the delegated administrator for organization-wide threat detection, and a log-archive account receiving centralized, write-only logs from every other account.
Public web tier: CDN, WAF, and DDoS protection. Global edge infrastructure (a content delivery network (CDN) integrated with a WAF and a Distributed Denial of Service (DDoS) protection service) fronts every region, terminating the majority of read-heavy and static traffic at the edge, closer to the customer, before it ever reaches a specific region's VPC; this both improves latency for a global customer base and reduces the volume of traffic each region's own infrastructure needs to absorb directly.
Private app and database tiers, per region. Within each region's production VPC, the same three-tier subnet pattern established for a single-region design applies: public subnets hosting only the regional load balancer, private application subnets, and private database subnets with no default internet route, replicated across Availability Zones (AZs) within that region.
Transit architecture: transit gateway (hub) versus direct peering. A transit gateway hosted in the shared-services account connects every production region's VPC, the stage account, and the shared-services account's own resources through a single, centralized routing construct, rather than a mesh of direct VPC peering connections between every pair of accounts and regions, which would grow combinatorially unmanageable as the number of regions and accounts increases (a full mesh across even 5 accounts is already 10 separate peering relationships to track and secure individually; a hub scales linearly instead, one attachment per account, regardless of how many other accounts exist).
IAM boundaries. Each production region's account has its own scoped identity and access management (IAM) roles; no standing role grants access across regions or across the production/stage boundary by default. Cross-account access that genuinely needs to exist (the shared-services account's NAT and patching operations reaching into each production account, the security account's read-only findings access) is implemented as the same narrow, purpose-specific IAM roles used in the base multi-account pattern, not broadened for this larger, multi-region version of the design.
Flow-log placement. VPC flow logs are enabled in every VPC, in every account, in every region, and all of them ship continuously to the single, centralized log-archive account; this gives a security investigator one place to query traffic patterns across the entire global footprint, rather than needing to separately query three regions' worth of logs stored in three different locations.
Centralized monitoring. The security account aggregates findings (from a cloud-native threat-detection service, or an equivalent) from every production account and region into one dashboard, with read-only cross-account access into each production account for exactly that purpose, following the same delegated-administrator pattern used in the base multi-account design.
Justification for the design choices
Why segmentation by both tier and account, not just one. Tier-level segmentation (public/app/database) limits what a compromised component within one region can reach; account-level segmentation limits what a compromise in one region, or in staging, can reach in another region or in production. The two are complementary, not redundant: a compromised application-tier instance in Region 1's production account is stopped from reaching Region 1's database tier by the tier-level network controls, and separately stopped from reaching Region 2's or Region 3's resources at all by the account-level IAM and transit-gateway routing boundary, two independent containment mechanisms addressing two different lateral-movement paths.
Why a transit gateway over full mesh peering. Beyond the combinatorial scaling problem named above, a transit gateway gives a single, centralized point where routing policy and, where the provider's transit gateway offering supports it, inspection can be enforced consistently across every attachment, the same centralized-enforcement benefit hub-and-spoke topology provides generally, now applied across regions as well as accounts.
Why the stage account has its own separate transit-gateway attachment rather than sharing production's. A staging environment, by design, runs less-reviewed code and configuration than production; giving it its own attachment (rather than routing it through the same attachment production regions use) means a routing-table-level policy in the transit gateway can explicitly restrict what staging can reach, without that restriction needing to also account for every production region's legitimate cross-region traffic pattern.
Trade-offs and pitfalls
- A transit gateway hub becomes the design's own single point of failure and single point of concentrated trust, the same tension inherent to any hub-and-spoke topology, now at a larger, business-critical scale. Redundant, highly-available transit gateway configuration and unusually tight administrative-access control on the shared-services account are not optional hardening steps for a design at this scale, they are load-bearing parts of the architecture.
- Per-region production accounts (the strongest isolation variant shown in the diagram) multiply the operational overhead of maintaining consistent security baselines across every account, compared to a single production account with per-region VPCs; the stronger blast-radius isolation needs to be weighed against the real, ongoing cost of keeping three (or more) accounts' guardrails, patching, and configuration consistently correct, via an account-vending process and organization-wide Service Control Policies (SCPs), not manual per-account discipline.
- Terminating traffic at a global CDN edge before it reaches a specific region means the WAF's rule set needs to be consistent across the entire global footprint, not configured separately per region, since an attacker will simply route their attempt through whichever regional edge has the weakest currently-deployed rule set if the WAF configuration is allowed to drift out of consistency between regions.
- A common wrong turn at this scale is under-specifying the stage account's isolation because "it's just staging," treating it as a lower priority for the same rigor applied to production; given that staging often has direct code-deployment pathways feeding into production later, a compromise there is a realistic path toward a subsequent production compromise, not an isolated, lower-stakes environment.
Design a secure hybrid connectivity architecture between on-premises data centers and AWS for an enterprise with 10,000 VMs and latency-sensitive workloads. Requirements: per-environment isolation (dev/prod), end-to-end encryption, predictable failover, and least-privilege routing. Provide diagram-level components (for example: Direct Connect, transit gateway, VPN, BGP) and explain security controls at each hop.
Sample Answer
Direct answer
A secure hybrid connectivity design for 10,000 on-premises virtual machines (VMs) with latency-sensitive workloads needs a dedicated, encrypted primary path (Direct Connect) for the predictable low-latency traffic, a VPN as an independent failover path rather than the primary, and per-environment routing isolation enforced at the transit layer so a development-environment credential or misconfiguration structurally cannot reach production, not merely a convention that assumes it will not.
Structured elaboration
flowchart LR
subgraph OnPrem["On-premises datacenter"]
DC["10,000 VMs, per-env VRF (dev/prod)"]
end
DC -->|"Direct Connect + MACsec, primary"| DXGW["Direct Connect gateway"]
DC -->|"IPsec VPN, backup path"| VPNGW["VPN gateway"]
DXGW --> TGW["Transit gateway (BGP)"]
VPNGW --> TGW
TGW --> ProdVPC["Prod VPC (isolated route table)"]
TGW --> DevVPC["Dev VPC (isolated route table)"]
ProdVPC -.->|"no route"| DevVPC
Component roles. Direct Connect provides the dedicated, predictable-latency primary path between the on-premises datacenters and AWS, terminating at a Direct Connect gateway; a site-to-site VPN provides an independent backup path over the public internet, terminating at a VPN gateway, active in the routing topology but only preferred by Border Gateway Protocol (BGP) path-selection when Direct Connect is unavailable; a transit gateway connects both paths to the cloud-side environment, and BGP handles dynamic route advertisement and the actual failover decision between the two paths, rather than a manual cutover process.
Security controls at each hop.
- On-premises to Direct Connect: MACsec (Media Access Control Security) encryption at the physical link layer, since Direct Connect's underlying connection is not encrypted by default the way an internet-routed VPN is; MACsec closes that gap for the primary path specifically, giving link-layer encryption on a connection that is otherwise private (not traversing the public internet) but not inherently encrypted.
- On-premises to VPN gateway (backup path): IPsec encryption, which is encrypted by construction as part of the VPN protocol itself, requiring no additional link-layer encryption step the way Direct Connect does.
- Direct Connect gateway and VPN gateway to transit gateway: both paths terminate into the same transit gateway, but with per-environment route-table isolation applied at the transit gateway itself (a distinct route table for production traffic and for development traffic), so encryption in transit is necessary but not sufficient, the routing-layer isolation is the control that actually enforces the "per-environment isolation" requirement, not the encryption.
- Transit gateway to VPCs: each environment's virtual private cloud (VPC) has its own transit gateway attachment associated with its own route table, with no route between the production and development route tables, making cross-environment reachability structurally absent rather than merely blocked by a security group that could be misconfigured later.
Predictable failover. BGP route advertisement from on-premises includes both the Direct Connect and VPN paths, with local preference or AS-path prepending configured so Direct Connect is always preferred when available; failover to the VPN path happens automatically at the BGP layer within the routing protocol's own convergence time, without requiring a manual intervention, and the VPN path's own capacity needs to be provisioned to genuinely sustain the latency-sensitive workloads' traffic during a failover event, not just "enough to keep things technically connected," since a failover path that cannot actually carry production load defeats the predictability goal even though it technically exists.
Least-privilege routing. Beyond the production/development route-table separation, route advertisement itself is scoped: on-premises only advertises the specific prefixes each cloud-side environment legitimately needs to reach, and the cloud side only advertises back the specific prefixes on-premises needs, rather than a broad "advertise everything" default that would let either side discover and potentially reach more of the other's network than the actual workload requires.
Worked example
A latency-sensitive trading application's VMs, part of the production VRF (Virtual Routing and Forwarding) on-premises, communicate with a cloud-hosted risk-calculation service in the production VPC. Traffic flows over the MACsec-encrypted Direct Connect link as the preferred BGP path, through the Direct Connect gateway, into the transit gateway, routed via the production-specific route table to the production VPC, a path with no dependency on the development environment's routing at any hop. When the Direct Connect link experiences a maintenance-window outage, BGP detects the path withdrawal and reconverges onto the IPsec VPN backup path within the protocol's normal convergence window, and traffic continues flowing, now over the internet-routed but still-encrypted VPN path, without a human needing to intervene; the production route-table isolation remains in effect regardless of which physical path is currently active, since the isolation is a property of the transit gateway's routing configuration, not of which link happens to be carrying the traffic at a given moment.
Trade-offs and pitfalls
- The VPN backup path's capacity is the single most common gap in a design like this, because it is provisioned to satisfy "we have a failover path" as a checkbox rather than "we have a failover path that can genuinely sustain our latency-sensitive workload's actual traffic." A failover event that succeeds at the BGP layer but degrades application performance because the VPN path cannot carry the same throughput at the same latency has technically achieved failover while still failing the workload's actual requirement.
- MACsec on Direct Connect requires compatible hardware on both the on-premises and the provider-facing equipment, and retrofitting it onto an existing Direct Connect circuit that was not originally provisioned with MACsec support is a materially bigger project than enabling a software configuration flag. This needs to be planned at the time the Direct Connect circuit itself is provisioned, not added as an afterthought once encryption-at-the-link-layer is later flagged as a gap.
- Per-environment route-table isolation at the transit gateway is the control that actually matters for the stated isolation requirement, and it is easy to under-invest in relative to the more visible encryption controls, since encryption is what shows up prominently in an architecture diagram while route-table configuration is comparatively invisible; a design that gets MACsec and IPsec right but leaves production and development sharing one route table has satisfied the encryption half of the requirements while missing the isolation half entirely.
- A common wrong turn at 10,000-VM scale is treating BGP configuration as a one-time setup rather than an ongoing operational discipline; route advertisement scope tends to grow more permissive over time as new dependencies are added under time pressure, gradually eroding the least-privilege-routing goal unless route advertisements are periodically reviewed against what is actually still needed.
Unlock Full Question Bank
Get access to all Cloud Security Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.