Airbnb Entry-Level Systems Engineer Interview Preparation Guide

Direct answer

A cross-team initiative I was part of missed its goals because of how, not what, we coordinated: unclear ownership across the teams involved, and assumptions that stayed unstated until they caused real problems. The lasting change wasn't a one-time apology or a single retro action item; it was a concrete shift in how the teams handed work to each other afterward, and I could point to whether that same failure mode recurred as the real evidence it stuck.

Structured elaboration

What broke, specifically

Swap in whatever cross-team dependency applies in your own world (a shared data pipeline, an API contract, a joint launch). In this skeleton, a project spanning several teams missed its deadline and caused repeated problems during a pilot phase because of two gaps: an unstated assumption about how a downstream team's dependency actually worked, and no clear escalation path when a blocking issue crossed a team boundary, so problems sat for days before the right people even knew about them.

How I ran the postmortem

• Built a timeline from evidence (incident counts, missed dates, rollback frequency), not memory or opinion.
• Separated the technical root causes from the collaboration root causes, since they needed different fixes.
• Named my own part in the failure to the group first, rather than only pointing at others' misses.

What actually changed afterward, and how I know

Concrete artifacts, not intentions: a documented dependency map required before a cross-team project kicks off, a clear ownership assignment per milestone naming who is accountable for what, and a pre-cutover checklist signed off by every team with something at stake, not just the owning team.

When the real obstacle is culture, not process

Sometimes the harder problem isn't a missing checklist, it's shifting a broader culture away from punitive postmortems toward ones people are actually honest in, particularly when some teams still default to blame. Modeling that shift means naming your own contribution to the failure before asking anyone else to, keeping the review focused on the system and the decision points rather than individuals, and treating a later postmortem where someone from a still-blame-oriented team volunteers a candid mistake as the real signal that the culture is moving, not just a nice-to-have.

Worked example

A multi-team initiative to consolidate several systems onto a shared platform missed its timeline and caused a string of problems during a pilot rollout. The retro traced the root cause to two things: application teams weren't told about a change in how long access credentials would remain valid under the new platform, and there was no agreed escalation path when a blocking issue spanned two teams. The concrete changes that came out of it were a mandatory dependency map and sign-off checklist before any team's cutover, and a named escalation contact per team for the duration of the rollout. A better signal of real progress on culture came from a smaller moment: at the next postmortem, a team that had previously stayed quiet about its own mistakes volunteered, unprompted, that a missed step on their side had contributed to a separate incident, which said more about the blame reflex fading than anything written in a process document.

Trade-offs and pitfalls

• A postmortem that produces only reflections ('we should communicate better') without a concrete, checkable change is the most common failure of this kind of story; the interviewer is listening for what's different in the next project, not what was learned.
• Owning your own part in the failure has to be genuine, not a rhetorical move before pivoting to blame others; if it reads as performative, it undercuts the whole story.
• A culture shift away from blame doesn't happen from one retro; it shows up gradually, in whether people volunteer uncomfortable information without being asked, and that takes sustained modeling, not a single well-run session.
• Watch for a story that only describes what changed for the team that failed, rather than what changed structurally for how all the involved teams hand off work to each other, since the initiative broke because more than one team was involved.

Validating cross-region failover as a chaos experiment means proving, under controlled conditions, that isolating one region actually degrades gracefully instead of cascading, and doing that with a stated baseline, a bounded blast radius, and a predefined rollback trigger so the experiment itself never becomes the outage it's trying to prevent.

Experiment flow

flowchart TD
  A[Capture steady-state baseline] --> B[State hypothesis]
  B --> C[Pre-checks: runbook, on-call, dashboards ready]
  C --> D[Canary: partition 1% of inter-region traffic]
  D --> E{Within rollback thresholds?}
  E -- No --> R[Rollback immediately]
  E -- Yes --> F[Full partition: isolate target region]
  F --> G[Observe against baseline]
  G --> H{Within rollback thresholds?}
  H -- No --> R
  H -- Yes --> I[Restore connectivity]
  I --> J[Validate consistency + replication catch-up]
  R --> J

Steady state, with concrete SLIs

Pin illustrative baselines for a two-region service (established from a real 7-day traffic window in practice, stated here as inputs to the experiment plan): global p95 latency under 200 ms, regional availability at or above 99.95%, error rate at or below 0.1%, and cross-region replication lag with a median under 50 ms. These are the exact metrics the experiment will compare against during and after the fault injection, chosen because they're already the numbers the service's on-call would check during a real incident, not new ones invented for this experiment.

Hypothesis

"Isolating the target region from the rest of the network will not drop global availability below 99.8% or push p95 latency above 350 ms, because the load balancer's health checks detect the partition within N seconds and reroute traffic to the healthy region, and in-flight writes to the isolated region fail closed rather than silently succeeding." Stating it this way makes the experiment falsifiable: either the reroute happens fast enough and writes fail safely, or it doesn't, and either outcome is useful.

Fault injection design

Simulate the partition at the network layer (block inter-region traffic between the two regions while leaving each region's intra-region networking untouched) rather than manipulating DNS or client routing, since the goal is to test whether the system's own failure detection and reroute logic works, not to manually route around a fault that was never actually detected.

Blast-radius control

Run in two phases, never full-scale first:

• Canary phase: isolate a small fraction (for example 1%) of inter-region traffic, or run the test against synthetic/shadow traffic before any real user traffic, to catch an obviously broken assumption cheaply.
• Full phase: only after the canary phase stays within thresholds, isolate the target region fully, during a scheduled low-traffic window, with the on-call team aware and dashboards open.

Rollback criteria (defined before the experiment starts, not decided mid-experiment)

Immediate rollback if any of: global availability drops below 99.8% sustained for more than 2 minutes, p95 latency exceeds 350 ms sustained for more than 2 minutes, error rate exceeds 5x the steady-state baseline sustained for more than 2 minutes, or any write is observed to succeed on both sides of the partition (a split-brain signal, which is the most severe possible finding since it means the failure-detection assumption in the hypothesis is simply wrong).

Post-experiment validation

After restoring connectivity: confirm replication lag returns to its steady-state baseline (not just that it's decreasing), and run a consistency check on any writes accepted during the partition window, comparing record counts and using idempotency/version keys to detect and reconcile any conflicting writes rather than assuming the reconciliation logic worked silently. Only after both checks pass is the experiment considered closed; a rollback that "looked fine" on live dashboards but left an unreconciled write behind is a false negative that the consistency check exists specifically to catch.

Trade-offs and pitfalls

The most dangerous shortcut is skipping the canary phase and going straight to a full regional isolation "because staging already validated it": staging traffic patterns and real production traffic patterns diverge in exactly the ways that matter for a partition test (real geographic distribution, real retry storms from real clients), so the canary phase against a small slice of real traffic is not optional. A second pitfall is defining rollback thresholds around availability and latency alone and missing data-correctness signals: a partition that stays within latency and error-rate thresholds but produces a split-brain write is a worse outcome than one that trips the availability threshold and rolls back cleanly, so the rollback criteria need a correctness trigger, not just a performance one. This same experiment design, steady-state baseline, hypothesis, canary-then-full injection, predefined rollback, post-hoc consistency check, applies with the target system swapped: a checkout flow (does payment processing fail closed or silently double-charge during the partition), a payments-compliance path (does an audit trail stay complete), a GPU training job (does a worker resume cleanly from checkpoint or corrupt its gradient state), a primary database failover (does the promoted replica actually have every acknowledged write), or a task scheduler's dependency graph (do downstream jobs correctly wait rather than run against stale upstream data). The steady-state SLIs and the specific failure signal change per target; the shape of the experiment doesn't.

Direct answer
I'll use Amazon's "Customer Obsession" as the example: in plain terms it means starting from the customer's actual experience and working backward to the decision, rather than starting from what's easiest or cheapest for the team and working forward to how it will land on the customer. In day-to-day work that shows up as a specific, repeatable habit: before finalizing a decision, explicitly write down what the customer will experience as a result, not just what the team will ship.

Structured elaboration

1. State the value in plain language first, in one or two sentences, before layering on any nuance. A stated value is only useful if you can restate it without jargon; if you can't, you probably don't understand it well enough to apply it.
2. Trace two or three concrete decisions the value would actually change, not just decisions it would be compatible with. The test is not "does this decision fit the value" (almost any reasonable decision can be described as fitting almost any value after the fact); the test is "would I have decided differently without this value in mind."
3. Be specific about the mechanism, not just the outcome. It's not enough to say "I'd focus on the customer"; describe the actual practice (writing the customer-facing consequence down explicitly, reviewing a metric that measures customer impact rather than only internal effort, asking a specific question in a design review) that operationalizes the value day to day.
4. Acknowledge the value has a cost or a trade-off, because a value with no real cost usually is not being taken seriously. A genuinely operative value changes what you'd otherwise have done, which means it sometimes means doing the harder or slower thing.
5. Connect it back to your own role specifically, since the same value plays out differently for different functions; the mechanism for a backend engineer, a designer, and an analyst are all different concrete practices in service of the same underlying value.

Worked example
Say you're building a dashboard intended to help a seller reduce order defects. A team NOT applying customer obsession as a working discipline might ship the dashboard once the underlying data pipeline is stable and the metrics are technically correct, treating "the data is right" as the finish line. Applying the value changes the finish line: before shipping, you'd sit with two or three actual sellers using an early version and ask what decision they're trying to make when they open it, which might surface that they need same-day defect data to catch a bad batch before it ships further, not a metric that's accurate but a day stale. The concrete decision that changes: you invest in a same-day data refresh even though it's more engineering effort than the weekly batch job you'd planned, because the customer's real decision-making need, not the easier technical path, is what determines what "done" means. The cost is real (more pipeline complexity, tighter SLAs to maintain) which is exactly why it's evidence the value is actually operative rather than decorative.

Trade-offs & pitfalls
The most common failure is reciting the value's definition fluently and then giving an example so generic it would apply to any company with any stated value ("I always think about the user"), which demonstrates you've read the careers page rather than that you understand the mechanism. A second pitfall is picking an example where the value cost nothing: if every example you give was also simply the obviously correct engineering or business call regardless of the stated value, you haven't actually shown the value did any independent work in your reasoning. A third is over-indexing on one company's specific phrasing so heavily that the answer would sound out of place at any other employer; the goal is to show you can genuinely reason from a stated principle to a concrete decision, a transferable skill, not that you've memorized one company's vocabulary.

Keep every region fully functional in isolation (write-local, read-local) and treat cross-region replication as an asynchronous durability mechanism, not a synchronous dependency for local dashboards. That's what lets a region stay observable during a full outage of any other region: nothing on the local read or write path ever blocks on a remote call.

Architecture

flowchart LR
  A[Region 1: Local Agents] --> B[Region 1: Hot Store]
  B -->|async replicate compressed blocks| C[Region 2: Durable Copy]
  B -->|async replicate compressed blocks| D[Region 3: Durable Copy]
  E[Region 2: Local Agents] --> F[Region 2: Hot Store]
  F --> C
  G[Query Router] --> B
  G --> F
  G --> H[Cross-Region Federation: nearest healthy]

• Each region runs a complete local observability stack (ingestion, hot storage, dashboards) so local telemetry never has to leave the region to be queried.
• Long-term/durable copies are replicated asynchronously to the other regions' object storage, batched and compressed, so a region that goes down entirely still has its recent history durable elsewhere.
• A query router directs dashboard reads to the local hot store first; if that region is down, it falls back to the nearest healthy region's replicated copy, and a federation layer merges results for genuinely global queries (e.g., "error rate across all regions").
• Clients (application agents) that can't reach their local ingest endpoint fail over to the nearest healthy region via DNS/client-side retry, so telemetry keeps flowing even during a full regional outage of the ingest path itself.

Why async replication of compressed data, not synchronous cross-region writes

Synchronous cross-region writes would mean every sample write waits on a round trip to at least one other region, multiplying write latency by inter-region network RTT (tens of milliseconds at best, over 100ms for distant region pairs) for every single sample, which is unacceptable at ingestion volumes discussed elsewhere in this domain (hundreds of thousands of samples/sec). Async replication of already-compressed chunks decouples durability from the write's critical path entirely.

Sizing the actual replication cost: 3 regions, each ingesting 300,000 samples/sec locally, with the S13/S14-style compression assumption of about 2 bytes/sample post-compression, replicating to the other 2 regions for durability:

regions = 3
per_region_ingest = 300_000
compressed_bytes_per_sample = 2
egress_cost_per_gb = 0.02   # illustrative unit cost, not a live vendor quote

per_region_bytes_sec = per_region_ingest * compressed_bytes_per_sample        # 600,000 B/s = 600 KB/s
per_region_replication_bytes_sec = per_region_bytes_sec * (regions - 1)       # to 2 peers: 1,200 KB/s
total_replication_bytes_sec = per_region_replication_bytes_sec * regions      # 3.6 MB/s aggregate
total_replication_gb_day = total_replication_bytes_sec * 86400 / 1e9          # 311.0 GB/day
monthly_cost = total_replication_gb_day * 30 * egress_cost_per_gb             # $186.62/month

At about 311 GB/day of aggregate cross-region traffic and an illustrative $186.62/month in egress cost, replicating already-compressed telemetry is cheap; the cost argument for async-and-compressed over synchronous-and-raw isn't marginal, it's roughly two orders of magnitude in bandwidth (compression alone gets you the 16-bytes-to-2-bytes reduction shown in the TSDB storage math elsewhere, before even counting that synchronous writes would need to happen per-sample rather than in batched chunks).

Deduplication and consistency

Because each region's replicated copy and the local hot-store copy can briefly diverge (async lag), the query-merge layer needs to deduplicate by (SeriesID, timestamp) when stitching a cross-region federated result, and prefer the ingest-region's copy as authoritative when both exist (rather than picking arbitrarily), to avoid the same data point appearing twice or a stale replica shadowing a fresher local write.

Cost trade-offs across replication strategies

Trade-offs and pitfalls

• Treating replication lag as zero is the most common mistake in the design write-up; alerting and incident dashboards should always default to local data (which has no replication lag) and only fall back to a remote replica when the local region is actually down, otherwise a transient replication delay looks like a data gap during a real incident.
• Full active-active replication sounds like the "safest" answer but multiplies both storage and cross-region bandwidth cost by the region count for a guarantee (any region can serve any tenant at full fidelity) the requirement doesn't actually ask for; the requirement is "stay observable during outage," which write-local/read-local with async replication already satisfies at a fraction of the cost.
• Data residency constraints (a region legally cannot replicate certain data outside its jurisdiction) break the "replicate everywhere" assumption; the design needs a per-region or per-tenant replication policy, not a single global rule, when residency requirements are in play.
• Deduplication logic that doesn't clearly prefer the ingest-region's copy as authoritative can silently double-count or shadow-out fresher data during federated queries, which is a subtle correctness bug that only shows up as slightly-wrong aggregate numbers, not an obvious failure.

Direct answer

I shipped a change to a signup flow that looked correct in every test environment but broke for users on a specific combination of browser and network condition we hadn't covered, and it was a customer, not our monitoring, who found it first, mid-demo, which made the failure both technical and painfully visible. Working out why it failed meant separating the actual technical root cause from the process gap that let it ship at all, and the fix that stuck was the one that closed the process gap, not just the code.

What happened and how I investigated

The change passed our automated tests and looked fine in manual quality testing, but broke for a subset of users because of an interaction between a caching layer and a redirect that only showed up under a specific, uncommon network condition. It surfaced when a prospective customer hit it during a live demo, which told me something important on its own: our alerting wasn't watching for this failure mode at all, so if the customer hadn't hit it live, it could have persisted undetected. Rather than just fixing the immediate bug, I traced two separate things: the technical root cause, the caching and redirect interaction, and the process gap, which was that our test matrix didn't cover that network condition and our monitoring had no signal that would have caught it in production either.

What I said and to whom, while it was still broken

As soon as I confirmed the cause, I told my manager and the account team handling that customer directly, with the specific technical explanation and an honest estimate of the fix timeline, rather than a vague "we're looking into it." That let the account team manage the customer conversation with real information instead of a placeholder.

What changed as a result

The immediate fix addressed the caching and redirect bug. The change that outlived the incident was adding the specific network condition to our test matrix and adding a monitoring alert for that class of redirect failure, so the next similar bug would be caught by our own systems instead of by a customer mid-demo. I also flagged that our sign-off process treated "tests pass" as equivalent to "ready to ship" with no explicit check for untested conditions, which is a narrower and more honest description of what our tests actually covered.

Trade-offs and pitfalls

The pitfall is stopping at the technical fix and treating the incident as resolved, when the more durable failure was the process gap that let something with an untested condition ship in the first place. A failure caught by monitoring and one caught by a customer can share the identical root cause, but they are different signals about how much your detection is actually covering.

Direct answer

Pull Flow Log records for the source and destination ENIs (Elastic Network Interfaces, the virtual network cards attached to each instance) and read the action field. A REJECT for that exact tuple means a security group or NACL (Network Access Control List, a stateless, subnet-level firewall, separate from the per-instance security group) is blocking it, while ACCEPT records with the app still timing out mean the problem is above the network layer entirely.

Structured elaboration

• Query Flow Logs (Athena or CloudWatch Insights) filtered to the incident window and the ENIs/ports involved.
• On REJECT, check both the security group and the NACL, since NACLs are stateless and can block the return leg even when the security group allows the request.
• On ACCEPT with no timely response, look at DNS resolution, the TLS handshake, or the destination process itself, none of which Flow Logs show.
• No records at all suggests routing, a missing route table entry or peering/Transit Gateway (a managed hub that routes traffic between multiple VPCs and on-premises networks over VPN or dedicated connections) misconfiguration, rather than a security rule.

Worked example

action=REJECT for 10.0.1.15:443 -> 10.0.2.20:5432 conclusively points at SG/NACL rules; action=ACCEPT for the same tuple with a client-side timeout redirects the investigation entirely toward the destination service instead.

Trade-offs and pitfalls

Flow Logs sample and aggregate rather than log every packet, so very brief issues can be underrepresented. They carry no payload detail, so ACCEPT doesn't mean the request was handled correctly.

What the interviewer probes next

Why NACLs being stateless matters for return traffic, and how you'd alert on a REJECT spike for a given path.

Approach summary
Brief, repeatable framework to evaluate SaaS vs. in-house across functional, non‑functional, cost, ops, risk, migration — produce a recommendation with decision drivers and sensitivity analysis.

Technical requirements

• Ingestion: peak events/sec, burst handling, backpressure, agent protocols (HTTP, syslog, fluentd), guaranteed delivery.
• Retention: hot/warm/cold tiers, retention policies, archival to S3, restore times.
• Query patterns: ad-hoc full‑text, aggregation, dashboards, alerting latency, ML/anomaly support.
• Integrations: IAM, K8s, cloud logs, tracing, metrics correlation.

Non-functional

• SLA: availability %, RTO/RPO for queries/ingest.
• Compliance: GDPR, HIPAA, SOC2, encryption-at-rest/in-transit, data residency.
• Security: tenant isolation, RBAC, audit logs.

Cost model (3yr TCO)

• SaaS: subscription, data ingress/egress, storage, premium features.
• In‑house: infra (compute, storage), licenses, SRE salaries, monitoring, backups, network egress, depreciation.
• Model: annualize CAPEX, project growth, run 3 scenarios (base, +50% load, +100%).

Operational staffing

• Headcount for build (design, dev, infra), ongoing SRE, on-call.
• Training, runbooks, incident response.

Failure modes

• Ingest overload, index corruption, query performance degradation, costly egress, vendor lock-in, misconfigurations causing data loss.

Migration complexity

• Data migration plan, dual-write period, schema mapping, dashboard rewrite, cutover strategy, rollback plan.

Recommendation structure for CTO

1. Executive summary (1–2 lines) with recommended option and primary rationale.
2. Key drivers (cost delta, time-to-value, compliance gaps, ops burden).
3. Sensitivity analysis (at what load/cost build becomes favorable).
4. Risks & mitigations.
5. Recommended next steps (pilot vendor X for 3 months / MVP build + metrics).

Brief definition

• Encryption at rest: protects stored data (disks, object stores, databases).
• Encryption in transit: protects data while moving across networks (client→server, service→service).

Common cloud implementation patterns

• At rest
  • EBS / managed block storage: cloud-provider SSE (e.g., AWS EBS encryption) or LUKS on VMs.
  • S3 / object stores: server-side encryption (SSE-S3, SSE-KMS) or client-side envelope encryption.
  • Databases: Transparent Data Encryption (TDE) or application-level field encryption.
• In transit
  • TLS for HTTP/GRPC (public and internal services) — use mTLS where possible.
  • VPN / IPsec for network-level protection (site-to-site, hybrid cloud).
  • Service mesh (e.g., Istio) to enforce mTLS between pods.

Where key management fits

• Centralized KMS/HSM (cloud KMS, CloudHSM) stores root keys; used to wrap data keys.
• Envelope encryption: data encrypted with symmetric data key; data key encrypted by KMS key.
• Key lifecycle: rotation, access policies (IAM), audit logs, and secure backup of keys (BYOK considerations).

Pitfalls that lead to noncompliance

• At rest
  • Misconfigured snapshots/shared AMIs containing unencrypted data or keys—snapshots inherit encryption only if created correctly.
  • Relying on “provider-managed” encryption but leaving object metadata or application secrets in plaintext (logs, tags).
• In transit
  • Using deprecated TLS versions or weak cipher suites; failing to enforce TLS for internal traffic.
  • Assuming private networks are secure and not enabling mTLS or VPNs — lateral movement risk.
• Cross-cutting
  • Poor KMS IAM policies or lack of key rotation/rotation tracking; compromised or over-permissioned principals can decrypt data.
  • Storing keys alongside data (in same VM or repo) or exposing keys in logs/configs.

Practical recommendations

• Use envelope encryption with KMS + least-privilege IAM, enable automatic rotation and audit logging.
• Enforce minimum TLS versions and cipher suites; use mTLS for service-to-service, and scan for plaintext secrets in backups/logs.

Direct answer

Before designing, find out how the 5,000 requests per second (RPS) actually splits by region and time of day, whether traffic is single-tenant or multi-tenant with isolation requirements, and what the payment provider's own latency and reliability really are, because each answer changes whether you build one active-active deployment or several regional ones with different capacity and failover needs.

Structured elaboration

Clarifying questions and what they change

Traffic distribution changes the architecture directly

Assume, once asked, the answer comes back as 40% North America, 30% Europe, 20% Asia-Pacific, 10% elsewhere:

Assume each service instance handles 50 RPS at the target P95, with 2x headroom for burst and failover:

Without the regional split, you would size one global pool for 5,000 RPS in one place, which is both the wrong shape (traffic is not colocated) and misses the actual failover unit, which is a region, not the global total.

Latency-budget arithmetic once the provider's numbers are known

Assume component P95s: 40 ms edge/network, 30 ms auth, 40 ms inventory reservation, 50 ms fraud check, 250 ms payment provider call, 20 ms response serialization:

Against a 500 ms budget, that leaves 70 ms (14%) of margin. That margin is the number that tells you how much slower the payment provider is allowed to get before the flow must switch to an async-confirm pattern (accept the order, confirm payment out of band) rather than blowing the SLO on every request.

Trade-offs & pitfalls

• Pitfall: sizing to the global RPS total instead of the regional split; you either over-provision the small regions or under-provision the busy one.
• Pitfall: assuming the payment provider's advertised latency holds under your own peak; treat its P95 as a variable you monitor, not a constant you designed around once.
• Multi-tenant isolation is easy to forget when the ask is phrased purely in terms of aggregate RPS; a single large tenant's flash sale can consume capacity meant for everyone else unless per-tenant quotas exist.
• Choosing an async-confirm path for payment buys latency headroom but costs the user a pending state, and costs you a reconciliation or webhook path instead of a single synchronous answer.

flowchart TB
    Client --> Router[Global traffic router]
    Router --> NA[NA region: checkout service]
    Router --> EU[EU region: checkout service]
    Router --> APAC[APAC region: checkout service]
    NA --> PayNA[Payment adapter + regional store]
    EU --> PayEU[Payment adapter + regional store]
    APAC --> PayAPAC[Payment adapter + regional store]

Direct answer

Run checks in a layered pipeline ordered from cheapest to most expensive: format and lint first (seconds, no cloud access), then static policy/security scanning against the plan or template (still no cloud access), then anything that actually stands up real resources (integration tests, plan review with real credentials), and finally post-deploy verification against the live environment. Each layer is designed to catch a different class of mistake, and putting the cheap checks first means a typo never has to wait for an expensive real-resource test to fail.

The two fundamentally different kinds of check

Before mapping tools to stages, it's worth naming the split explicitly: some checks are fast, syntax- or policy-level, and need no cloud access at all; others genuinely stand up real (usually short-lived) infrastructure to prove it actually works. Confusing the two, or skipping straight to the expensive kind, is the most common mistake in a pipeline like this.

Worked example: a pipeline definition

jobs:
  fmt-and-lint:
    steps:
      - run: terraform fmt -check -recursive
      - run: tflint

  policy-scan:
    needs: fmt-and-lint
    steps:
      - run: checkov -d . --framework terraform

  plan:
    needs: policy-scan
    steps:
      - run: terraform plan -out=plan.tfplan
      - run: terraform show -json plan.tfplan > plan.json
      # a script here fails the job if plan.json contains an unreviewed delete/replace

  integration-test:
    needs: plan
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    steps:
      - run: go test ./test/... -run TestModule -timeout 30m
      # this job is the one that actually provisions and tears down real infra

  apply:
    needs: integration-test
    environment: production   # requires manual approval
    steps:
      - run: terraform apply plan.tfplan

The ordering matters: fmt-and-lint and policy-scan run on every commit because they're free and fast; integration-test, which provisions real infrastructure, is scoped to run only on merges to main, not on every push to a feature branch, to bound cost and time.

Trade-offs and pitfalls

• Skipping straight to integration tests (or relying on them to catch what a linter would have caught in seconds) is slow and expensive for no extra safety, put the cheap checks first and let them fail fast.
• Running the real-resource layer on every commit, rather than on merge or nightly, is the single most common way teams accidentally burn cloud spend on a testing pipeline.
• Static policy scanning only catches what it has rules for; it gives false confidence if the rule set isn't kept current with new resource types the team starts using.
• A plan-review gate that only a human reads doesn't scale, encode the "never allow an unreviewed delete on a protected resource" rule as an automated check on the plan JSON, not just a habit.