DoorDash Cloud Architect (Junior Level) Interview Preparation Guide

High-level definition & goals

Network security guardrails are automated, organization-wide constraints that prevent insecure topology/configuration while allowing teams velocity. Goals: prevent public-exposed resources, enforce least-privilege flows, require segmentation and encrypted transit, and provide measurable compliance with fast remediation.

Guardrail layers & controls

• IaC linting (pre-commit / CI): run tflint, cfn-lint, checkov, and custom rulesets to reject insecure patterns (eg. aws_security_group allowing 0.0.0.0/0 on sensitive ports). Integrate as GitHub Actions/GitLab CI step.
• Policy-as-code: author policies in OPA (Rego) for CI gate, Sentinel for Terraform Enterprise, and Cloud Custodian for cloud resource audits. Examples: deny creation of Internet-facing load balancers without WAF; require NACLs/subnet tagging for segmentation.
• Provider-native: AWS Config managed/custom rules (e.g., restricted security group rules, VPC flow logs enabled, S3 block public access) with aggregated AWS Config aggregator for org-wide visibility.

Remediation vs Alerts

• Automated safe remediation for high-confidence fixes (eg. remove overly permissive SG rule, enable flow logs) using Lambda/Step Functions invoked by Config/CloudWatch Events or Cloud Custodian remediation actions.
• Alerts & human review for higher-risk changes (eg. removing NAT gateway) routed to Slack/PagerDuty with ticket auto-creation.

Onboarding & adoption

• Provide starter IaC templates, policy libraries, CI pipeline snippets, and runbooks. Run brown-bag sessions, pair-programming, and an exemption workflow (time-boxed approvals, audit trail).
• Start with a pilot team, iterate rules based on feedback, roll out org-wide.

Metrics & continual improvement

• Coverage metrics: % of repositories with linting & policy checks, % of accounts with AWS Config enabled.
• Compliance metrics: % resources compliant, mean time to remediate, number of exceptions.
• Regular reviews: quarterly policy sprints, incorporate incident findings into rules.

Direct answer

Before designing, find out how the 5,000 requests per second (RPS) actually splits by region and time of day, whether traffic is single-tenant or multi-tenant with isolation requirements, and what the payment provider's own latency and reliability really are, because each answer changes whether you build one active-active deployment or several regional ones with different capacity and failover needs.

Structured elaboration

Clarifying questions and what they change

Traffic distribution changes the architecture directly

Assume, once asked, the answer comes back as 40% North America, 30% Europe, 20% Asia-Pacific, 10% elsewhere:

Assume each service instance handles 50 RPS at the target P95, with 2x headroom for burst and failover:

Without the regional split, you would size one global pool for 5,000 RPS in one place, which is both the wrong shape (traffic is not colocated) and misses the actual failover unit, which is a region, not the global total.

Latency-budget arithmetic once the provider's numbers are known

Assume component P95s: 40 ms edge/network, 30 ms auth, 40 ms inventory reservation, 50 ms fraud check, 250 ms payment provider call, 20 ms response serialization:

Against a 500 ms budget, that leaves 70 ms (14%) of margin. That margin is the number that tells you how much slower the payment provider is allowed to get before the flow must switch to an async-confirm pattern (accept the order, confirm payment out of band) rather than blowing the SLO on every request.

Trade-offs & pitfalls

• Pitfall: sizing to the global RPS total instead of the regional split; you either over-provision the small regions or under-provision the busy one.
• Pitfall: assuming the payment provider's advertised latency holds under your own peak; treat its P95 as a variable you monitor, not a constant you designed around once.
• Multi-tenant isolation is easy to forget when the ask is phrased purely in terms of aggregate RPS; a single large tenant's flash sale can consume capacity meant for everyone else unless per-tenant quotas exist.
• Choosing an async-confirm path for payment buys latency headroom but costs the user a pending state, and costs you a reconciliation or webhook path instead of a single synchronous answer.

flowchart TB
    Client --> Router[Global traffic router]
    Router --> NA[NA region: checkout service]
    Router --> EU[EU region: checkout service]
    Router --> APAC[APAC region: checkout service]
    NA --> PayNA[Payment adapter + regional store]
    EU --> PayEU[Payment adapter + regional store]
    APAC --> PayAPAC[Payment adapter + regional store]

What a VPC provides

A Virtual Private Cloud (VPC) is an isolated virtual network in the cloud that gives you IP address space, subnets, route tables, Internet/NAT gateways, peering/VPN connectivity, and boundary controls. It enables tenancy isolation, network segmentation, private connectivity to on‑prem, and enforcement points for security and traffic flow.

Security Groups vs Network ACLs

• Stateful vs stateless

  • Security Groups: stateful — return traffic is automatically allowed for permitted inbound/outbound flows.
  • Network ACLs (NACLs): stateless — you must explicitly allow both directions.

• Typical use-cases

  • Security Groups: host-level microsegmentation, instance-level application allow-lists (SSH, app ports).
  • NACLs: subnet-level coarse controls, defense-in-depth, blocking known-bad IP ranges, cross-account protection.

• Rule ordering and evaluation

  • Security Groups: unordered; rules are aggregated — any match permits/denies (cloud SGs typically only allow; implicit deny otherwise).
  • NACLs: ordered numeric rules evaluated top-to-bottom; first match wins; explicit allow/deny entries plus a final implicit deny.

• Performance & operational implications

  • Both are high-performance and scalable; SGs are easier to manage for dynamic autoscaling (attach by tag/instance). NACLs are better for broad, static controls but can be error-prone when many ordered rules exist. Stateless NACLs can increase operational complexity (need mirrored rules).

Recommended pattern for multi-tier apps

• Use VPC subnets per tier (public for load balancers, private app, private data).
• Apply NACLs at subnet edge for broad, coarse-grained protections (block malicious CIDRs, rate-limit heuristics if supported).
• Use Security Groups for fine-grained, role-based access: LB SG allows 80/443 from Internet; App SG allows only from LB SG on app port; DB SG allows only from App SG on DB port.
• Leverage tags, centralized naming, and least-privilege rules. Use auditing and automation (IaC) to keep SGs and NACLs consistent.

This layered approach provides defense-in-depth, operational simplicity at scale, and minimal blast radius.

Direct answer

The two usual suspects are Availability Zone (AZ) crossing traffic between services that happen to land in different AZs, and Network Address Translation (NAT) Gateway data-processing charges when private-subnet resources reach the internet or other AWS services through a NAT Gateway rather than a direct path. Both are metered per gigabyte and both are easy to accumulate without noticing, because nothing about the traffic looks wrong, it's just routed the expensive way.

What's causing it

• Cross-AZ chattiness: a load balancer, service mesh, or just an Auto Scaling Group (ASG) spreading instances evenly across AZs means calls between two tiers of a service frequently cross AZ boundaries. Each direction of that hop is billed as inter-AZ data transfer, even though the two AZs are in the same region.
• NAT Gateway egress: resources in private subnets that reach S3, DynamoDB, or the public internet through a NAT Gateway pay a per-GB data processing charge on top of the underlying transfer. If the NAT Gateway itself sits in one AZ and the calling resources are spread across several, that traffic pays both the NAT processing charge and a cross-AZ hop to reach it.

Two straightforward fixes

1. Add Virtual Private Cloud (VPC) Gateway Endpoints for S3 and DynamoDB. These are free (no hourly or per-GB endpoint charge) and keep that traffic on the AWS backbone entirely, bypassing the NAT Gateway path altogether for two of the most common egress destinations. This is usually the highest-leverage, lowest-effort fix.
2. Give every AZ its own NAT Gateway instead of routing all private subnets through one shared NAT Gateway in a single AZ. This removes the extra cross-AZ hop to reach the NAT device; each AZ's traffic exits locally. It costs more in NAT Gateway hourly charges but usually still nets out cheaper than the cross-AZ transfer it eliminates, and it also removes an AZ-level single point of failure.

Worked example

A service handling 50 GB/day of outbound calls to an external API, routed through a single NAT Gateway in one AZ, with half of the calling instances in a different AZ: 25 GB/day of that traffic pays both the NAT Gateway data processing rate and a cross-AZ transfer rate, stacking two per-GB charges on the same bytes. Giving each AZ its own NAT Gateway removes the cross-AZ leg entirely for that half, leaving only the (unavoidable) NAT processing charge. This is a design-target illustration, not a measured bill; actual savings depend on current per-GB rates, which vary by region and change over time, and should be checked in AWS Cost Explorer rather than assumed.

Trade-offs and pitfalls

• Per-AZ NAT Gateways cost more in fixed hourly charges than one shared gateway; the fix only pays for itself once the eliminated cross-AZ transfer volume exceeds that fixed cost, which is worth checking with Cost Explorer before rolling it out everywhere.
• VPC Gateway Endpoints only cover S3 and DynamoDB. Other AWS services need Interface Endpoints (AWS PrivateLink-backed), which do carry an hourly and per-GB charge, so they're a narrower cost win, not a blanket replacement for NAT.
• Don't chase AZ-local placement so hard that it undermines the multi-AZ redundancy the architecture depends on for availability; the fix is about routing waste, not about collapsing back to a single AZ.

I would use count or for_each for the feature flag, but I would prefer for_each for the per-item objects. count is a simple on or off switch. for_each creates one instance per stable key, which is better when the list order changes.

Pattern

• For the feature-flagged singleton, create either one instance or none
• For the repeated objects, convert the caller’s list into a map keyed by a stable ID, such as name
• Avoid indexing directly into a list, because reordering ['api', 'worker'] can cause unnecessary replacement

Example
If the caller passes ['api', 'worker'] today and ['worker', 'api'] tomorrow, keys like api and worker still point to the same resources. That keeps Terraform from churning objects just because the order changed.

Rule of thumb
Use count for a single optional resource, and for_each for anything that should survive list reordering. That makes the module much easier to maintain as the list grows.

Situation & Task
I led cloud architecture for a 300‑engineer SaaS team migrating from monolith VMs to Kubernetes on AWS. Knowledge gaps around multi‑AZ networking, IAM least‑privilege, and cost‑aware autoscaling caused repeated design mistakes.

Action (Teaching format & preparation)
I designed a two‑hour hands‑on workshop + a living Confluence runbook with diagrams and reusable CloudFormation/Terraform snippets. Workshop included:

• short theory segments (VPC design, private endpoints)
• lab exercises (EKS cluster setup with cluster-autoscaler and OPA Gatekeeper policies)
• post‑workshop recorded walkthroughs and Q&A notes.

Preparing materials forced me to codify trade‑offs (e.g., CNI choices, cross‑AZ latency), write idempotent templates, and create failure scenarios—revealing gaps in my own assumptions about pod density and NAT gateway costs.

Result
Adoption of the runbook reduced misconfigurations by 70% (fewer architecture review rework items) and led to two concrete changes:

• standardized CNI (AWS VPC CNI with custom ENI limits) and node sizing guidelines,
• enforced policy guardrails via OPA/Gatekeeper integrated into CI.

I now include teaching artifacts as part of every design deliverable; it both scales knowledge and surfaces design flaws earlier.

Summary / goals
Design a cost‑optimized, global API at 100k RPS with ~50ms read and 200ms write targets by combining edge delivery, multi‑region read scaling, partitioned writes with region affinity, read caches, and tunable consistency.

High-level architecture

• Global API GW (CloudFront/Global Accelerator + regional API Gateways) for TLS termination, DDoS protection, & routing.
• Edge CDN for static/JSON cacheable responses.
• Regional API clusters (autoscaled compute) in N active regions.
• Regional in‑memory caches (Redis/Memcached) + global cache invalidation via pub/sub.
• Global datastore: geo‑aware DB (e.g., DynamoDB Global Tables, CockroachDB, or Spanner) with per‑entity partitioning.

Data partitioning

• Partition by customer/tenant or hash(key) to distribute keys evenly.
• Add region affinity: place primary write shard in nearest region for customer (reduces cross‑region writes).
• For hot keys, use deterministic sharding and moveable shards (rehash or split).

Caching strategy

• Edge CDN (TTL + stale‑while‑revalidate) for fully cacheable reads.
• Regional read‑through cache (Redis) for low latency ~<5ms. Populate caches on miss; use write‑through or invalidate via pub/sub.
• Client caching (ETag/Last‑Modified) and conditional requests to reduce origin load.
• Cache tiering: CDN -> regional cache -> DB.

Multi‑region replication & consistency

• Active‑active reads: replicate asynchronously to all regions for read scale.
• Writes: route to shard’s home region (primary) to avoid cross‑region consensus; asynchronously replicate to other regions.
• Consistency model:
  • Default: eventual consistency for global reads (fast, cheap).
  • Strong or read‑after‑write for critical endpoints: session tokens + sticky routing to primary region or use quorum reads (R + W > N) where supported.
  • Conflict resolution: last‑write‑wins for simple cases; application merge or CRDTs for complex state.

Latency mapping

• CDN edge hits: <20ms global.
• Regional cache hit: ~<5–10ms.
• Regional DB read (local replica): ~20–50ms.
• Writes (local primary + async replication): <200ms target; cross‑region writes avoided for most traffic.

Cost controls

• Autoscaling with CPU/memory/RPS metrics; scale down aggressively for idle regions.
• Use serverless (Lambda/FaaS) + API Gateway for highly variable workloads; provisioned capacity for steady baseline.
• Choose mixed instance purchasing: reserved/savings plans for baseline, spot for noncritical workers.
• Cache hit‑rate SLAs and cost per RPS monitoring; tune TTLs to meet hit targets.
• Rate limiting, API tiers and quotas to protect backend.
• Monitoring + budget alerts, cost allocation tags, periodic data lifecycle (cold storage, TTL on cold partitions).
• Evaluate per‑region write placement to trade latency vs storage/replication cost.

Tradeoffs

• Strong consistency everywhere increases cost and latency (consensus across regions) — use selective strong paths.
• More regions = lower latency but higher replication cost; choose regions by traffic/POPs.

This design meets 100k RPS by maximizing cache and edge hits, localizing writes via partitioning/affinity, and exposing tunable consistency to balance latency, correctness and cost.

Operational Readiness Checklist (Cloud Architect handoff)

1) Requirements & Service Summary

• Service purpose, SLA/ SLO targets, peak load, dependencies, topology diagram, SLA owners.

2) Monitoring & Observability

• Metrics: latency, error rate, throughput, resource utilization (CPU, memory, I/O), queue depth.
• Traces: distributed tracing enabled (e.g., OpenTelemetry) with sample traces for common flows.
• Logs: structured, correlated request-id, centralized retention and indexability.
• Alerts: alert burn-rate, severity, runbook links; alert thresholds tied to SLOs.

3) Runbooks / Playbooks

• For P0–P2 incidents: detection, mitigation steps, escalation matrix, command snippets, dashboards to inspect.
• Postmortem template and RCA expectations.

4) Access Controls

• IAM roles and least-privilege policies, emergency access procedure (just-in-time), audit log location, key/secret rotation policy.

5) Automated Tests & CI/CD

• Unit/integration tests, chaos tests (failure injection), canary deployment, synthetic tests and health checks.
• CI pipeline: promoted artifacts, deployment gating criteria.

6) Rollback & Recovery Criteria

• Clear rollback triggers, automated rollback script, warm standby or snapshot recovery steps, RTO/RPO targets.

7) Sign-off & Ownership

• Handoff sign-off form listing SRE/platform team owner, on-call rota, escalation contacts, knowledge-transfer sessions completed, agreed SLOs and review cadence.

Deliverables: checklist doc, diagrams, runbooks, test reports, access matrix, and a 1-hour walkthrough before sign-off.

Direct answer: The immediate priority is to stop the slowdown from consuming your own resources: set aggressive timeouts, open a circuit breaker so you stop calling the failing dependency, and isolate the connection/thread pool used for that call so it can't starve everything else. The longer-term fix is architectural: decouple the caller from the dependency's latency entirely, usually via an async queue or by making the call non-blocking, so a slow downstream degrades throughput instead of taking the whole service down with it.

Structured elaboration

Why this happens (the mechanism): by Little's Law, the number of requests in flight $L$ equals arrival rate $\lambda$ times the time each request spends in the system $W$: $L = \lambda W$. If a downstream call's latency goes from 50ms to 500ms while your request rate stays at, say, 200 requests/second, the in-flight count grows from $L = 200 \times 0.05 = 10$ to $L = 200 \times 0.5 = 100$, a 10x increase, purely from the latency change with no change in incoming traffic. If your thread or connection pool was sized for ~10-20 concurrent in-flight requests to that dependency, it's now exhausted, and requests start queueing on your side, which is exactly the symptom described.

Immediate mitigations (minutes, not a redesign):

Longer-term architectural fix:

• Decouple via an async queue: put a durable queue between the caller and the slow dependency so the caller can return quickly (accept-and-acknowledge) and the dependency is drained at its own sustainable pace, rather than the caller blocking on it synchronously. Trade-off: the caller can no longer return a synchronous success/failure for that operation; the interaction model has to change to something the client and product can tolerate (a "pending" state, a webhook, a poll).
• Idempotent retries with backoff and jitter: if retries are needed, they must be capped, exponential, and jittered so a fleet of callers doesn't retry in lockstep and re-create the exact overload it's recovering from. Trade-off: added complexity, and retries must be provably idempotent on the downstream side or they risk duplicate side effects.
• Capacity planning against the tail, not the average: provision the dependency (or the pool sized to call it) based on observed p99 latency, not p50, since it's the tail that determines when queues start building. Trade-off: costs more standing capacity for headroom that's idle most of the time.

Applying this to concrete variants of the same pattern: the reasoning above is the same whether the slow dependency is a payment-validation service (immediate: circuit breaker + fast-fail with a clear "try again" to the user rather than a silent hang; long-term: async payment confirmation via webhook), a message-queue consumer falling behind (immediate: shed or dead-letter the oldest low-priority messages, bulkhead the consumer pool by message type; long-term: scale consumers horizontally and partition by priority), a retry storm from a flood of client-side 503s (immediate: the client-side backoff-with-jitter above is the direct fix; long-term: make the shedding threshold adaptive so it doesn't itself become the trigger for a thundering herd), or a synchronous order-processing pipeline backing up (immediate: bulkhead the slow stage's pool; long-term: convert that stage to the async-queue pattern above).

Trade-offs & pitfalls

• Every immediate mitigation above trades some availability or correctness for stability: timeouts drop requests that might have succeeded, circuit breakers deny service during their open window, load shedding sacrifices some requests to save the rest. The point isn't to avoid the trade-off, it's to make it deliberately and visibly rather than let an unbounded queue make it for you via an eventual crash.
• A common wrong turn: adding retries as the first response to a slowdown. Naive retries without backoff amplify load on an already-struggling dependency and can turn a partial slowdown into a full outage (a retry storm).
• Circuit breakers and bulkheads need to be tuned against real traffic and latency distributions; thresholds copied from a different service's runbook are a common source of either false trips (unnecessary unavailability) or no protection at all (thresholds too loose to matter).

Direct answer

Safe retry semantics have to be designed per connector type, because each one offers a different native primitive for idempotency or atomicity: Kafka producers get exactly-once via the idempotent producer plus transactions; Kafka consumers get it via read_committed isolation reading only committed transactional output; database writes get it via native upserts or local transactions; REST calls to a third-party get it via an idempotency-key header when the API supports one, or an outbox-plus-proxy pattern when it does not; and object storage like S3 gets it via content-addressed keys or an atomic manifest commit. There is no single mechanism that covers all four; the architecture's job is to pick the right one per connector and make sure they compose correctly end to end.

Structured elaboration

Kafka producers. Enable the idempotent producer (enable.idempotence=true), which assigns each producer a unique ID and each message a sequence number, letting the broker deduplicate retried sends from the SAME producer session automatically. For cross-partition or cross-topic atomicity (writing to multiple topics as one unit), wrap the writes in a Kafka transaction (initTransactions, beginTransaction, commitTransaction), which the broker either fully commits or fully aborts.

Kafka consumers. Reading a transactional producer's output requires setting the consumer's isolation level to read_committed, so aborted or in-flight transactions are invisible; a consumer left at the default read_uncommitted would see uncommitted, possibly-aborted data, silently breaking the exactly-once guarantee the producer side worked to provide. Consumer offset commits should be tied to downstream processing completion (commit the offset only after the corresponding output is durably written), not committed eagerly on read.

Database writes. Use the database's native atomic primitives: an INSERT ... ON CONFLICT DO UPDATE (Postgres) or MERGE keyed by a business key plus version, for single-row idempotency; a local transaction for multi-row atomicity within that one database. If the write must be atomic with the Kafka consumer offset commit (a common payments pattern), the outbox pattern (write the outbox row in the SAME local database transaction as the business write) decouples that atomicity from needing Kafka and the database to share a distributed transaction, which they generally cannot.

REST calls. If the third-party API supports an idempotency-key parameter (Stripe-style), generate that key deterministically from the logical operation (not fresh per retry) and let the API's own deduplication handle it. If it does not, apply an idempotency-proxy pattern: put a proxy in front of the API (the strongest option, if worth building), or accept a compensating-transaction fallback for genuinely one-way, non-idempotent operations.

Object storage (S3). Native S3 operations are individually retry-safe (a PutObject retried with the same key and content just re-uploads the same bytes, no duplication), but a MULTI-OBJECT logical write (many files representing one dataset version) needs a manifest-based atomic commit: stage, then atomically swap a small manifest pointer, so a partial or duplicated multi-object write is never visible as "done."

Worked example

A pipeline reads Kafka, writes to a Postgres database (for a materialized view), calls a third-party fraud-check REST API, and archives raw events to S3, all per logical event, needing the whole chain to behave correctly under retries. Concrete wiring, in order:

1. Kafka consumer reads with read_committed, does not commit its offset yet.
2. Postgres write: INSERT ... ON CONFLICT (event_id) DO NOTHING (idempotent by event_id).
3. Fraud-check REST call: the API supports an idempotency-key header; the pipeline passes event_id as that key deterministically, so a retried call after a timeout is recognized and returns the original result.
4. S3 archive: PutObject keyed by event_id (content-addressed by logical identity), so a retried upload overwrites the identical object harmlessly.
5. Only after all three writes are confirmed does the Kafka consumer commit its offset.

If step 3 (the REST call) times out ambiguously and the whole event is retried from step 2: step 2's ON CONFLICT DO NOTHING is a safe no-op (already inserted), step 3's idempotency key correctly returns the cached fraud-check result rather than re-running it, and step 4's re-upload is harmless. The offset is committed only once all four steps are confirmed, so a crash before that point simply replays this exact same, now-fully-idempotent sequence, and a crash after commit never revisits this event again (correct, since it was already fully processed).

Trade-offs and pitfalls

• Common mistake: committing the Kafka offset before all downstream writes are confirmed. This is the single most common way to silently lose the "at-least-once" half of the guarantee: a crash between offset-commit and the last downstream write means that event is never retried, since the consumer believes it already handled it.
• Common mistake: assuming Kafka's idempotent producer alone gives end-to-end exactly-once. It only protects the Kafka WRITE from producer-side retries; it says nothing about the downstream database, REST call, or S3 write each independently needing their own idempotency discipline, exactly why this answer treats each connector type separately rather than claiming one mechanism covers the whole chain.
• Ordering the four connector writes matters for correctness, not just tidiness. Placing the offset commit last (as in the worked example) is deliberate: it is the one step in the chain that, if it happens too early, breaks the whole at-least-once guarantee; every other step being idempotent means their relative order among themselves is more flexible.
• Per-connector idempotency does not automatically give cross-connector atomicity. If the fraud-check call succeeds but the process crashes before the S3 archive, on retry the fraud-check idempotency key correctly avoids re-running (good), but there is a window where downstream state is partially applied; this is the same partial-failure-across-heterogeneous-sinks problem any multi-sink write faces, and the fix is the same: make every step both idempotent AND independently retriable, not build a fragile distributed transaction across all four.