Airbnb Senior Systems Engineer Interview Preparation Guide

Direct answer

Put a durable, replicated buffer (a log like Kafka, not an in-memory queue) between collectors and processors so a failed component can be detected and rerouted around without losing the data that was in flight. Track three pieces of state to do this safely: per-partition consumer offsets (so replay resumes from the right place, not from zero), per-component health signals (lag, error rate, heartbeat) that drive rerouting decisions, and an idempotency key on every event so a replay after a failure doesn't get double-counted downstream. What stops the remediation logic itself from cascading is a hard budget on how many auto-remediation actions any one component can trigger in a window, with a circuit breaker that pages a human once that budget is exhausted instead of retrying forever.

Structured elaboration

State to track:

• Consumer offsets per partition per consumer group, checkpointed durably, so a restarted processor resumes exactly where it left off.
• Health signals per component: consumer lag, processing error rate, heartbeat/liveness. These feed the detection logic, not just an external dashboard.
• Idempotency keys on events (a stable ID derived from source + sequence number, not regenerated on replay), so downstream stores can deduplicate when the same event is delivered twice after a replay.

Detection and reroute:

• Liveness and readiness probes remove an unhealthy processor from its consumer group; the group rebalances so the partitions it owned are picked up by healthy consumers.
• Rising consumer lag or error rate on a partition, even without an outright liveness failure, is itself a detection signal (this catches degraded-but-technically-alive processors, not just crashed ones).
• Collectors buffer locally and retry with backoff when they can't reach a processor, rather than dropping data immediately.

Replay:

• On restart, a processor resumes from its last committed offset; the durable log's retention window bounds how far back replay can reach.
• For a deliberate, controlled replay (not just crash recovery), a replay controller can reset a consumer group's offset to an earlier point and replay into an isolated processing path first, so a bad replay doesn't double-write into the live output the way replaying directly into the primary consumer group would.
• Stateful processors restore their state from a changelog or external state store before resuming, so replay doesn't start from a stale in-memory state.

Autoscaling: scale on consumer lag and error rate together, not CPU alone. A processor that's CPU-idle but falling behind (for example, downstream I/O-bound) needs more replicas even though CPU utilization alone wouldn't trigger a scale-up.

Degraded-mode operation: when the pipeline itself is under load it can't fully process (for example a partial outage reducing available processor capacity), fail toward reduced fidelity rather than data loss: temporarily increase sampling (drop a configured fraction of lower-priority telemetry) to keep the pipeline within capacity while preserving full-fidelity handling for high-priority signals, then return to full sampling once capacity recovers.

A specific, common failure mode worth naming: a collector with a slow memory leak. This doesn't trip a liveness check the way a crash does; it degrades gradually. The remediation here isn't "restart on OOM" (that's just a symptom-fix that fires only after the leak has already caused damage), it's tracking a per-collector memory trend and proactively rotating (drain, restart) a collector whose memory is trending toward its limit before it OOMs and drops whatever was buffered in-process at the time.

flowchart LR
    C[Collector] -- push --> BUF[(Durable log: partitioned, replicated)]
    BUF --> P1[Processor pod]
    P1 -- heartbeat/lag --> HM[Health manager]
    HM -- unhealthy: rebalance --> BUF
    HM -- budget exceeded --> CB[Circuit breaker: page human]
    HM -- lag rising --> AS[Autoscaler]
    AS -- scale replicas --> P1
    RC[Replay controller] -- controlled replay --> BUF
    RC --> ISO[Isolated replay path]

Worked example

Sizing the durable buffer to survive an outage. Assume the pipeline sustains $r = 200{,}000$ events/sec at $300$ bytes/event, and the target is to absorb up to a 15-minute processor outage without dropping data (detect + reroute + recover within that window):

(The 1.5x factor is headroom for more than one component being degraded at once, not just a single clean outage.) With replication factor 3 for durability:

That's the concrete number that goes into sizing the durable log's disk footprint for a 15-minute recovery SLO at this ingestion rate: roughly a quarter-terabyte of replicated buffer, not an arbitrary "keep some retention" guess.

Bounding the remediation loop so it can't cascade. Use a token-bucket budget per component: capacity 3 auto-remediation actions (for example, restarts), refilling 1 token every 10 minutes. In the worst case, a flapping component can trigger:

before the bucket is empty and the circuit breaker opens, escalating to a human instead of continuing to retry. Pair this with capped exponential backoff on reroute retries (base 2s, doubling, capped at 60s):

The cap matters here specifically: without it, a component that's actually gone for good would have retry intervals growing unbounded, delaying the eventual "stop retrying, alert a human" decision far longer than necessary.

Trade-offs & pitfalls

Common wrong turns: replaying directly into the live consumer group after a failure instead of an isolated path first, which can double-process events downstream if the failure that triggered the replay wasn't a clean crash (partial writes already landed); sizing the durable buffer for the average outage duration instead of the target recovery SLO, which quietly fails during the outages that matter most; and building auto-remediation without a hard action budget, on the assumption that "it can only help," when a bad remediation action (for example restarting a processor whose real problem is a poison-pill message) can itself be the thing that turns a single-partition issue into a rebalancing storm across the whole consumer group.

An escalation system for a multi-team, multi-timezone org needs three separate mechanisms working together: routing (getting an alert to the right on-duty person without a human deciding that in the moment), noise suppression (so one root cause doesn't fan out into ten pages), and a severity model that decides who gets pulled in and how fast, because a revenue-impacting outage and a data-sensitive incident need different people in the room, not just different urgency.

Core building blocks

• Alert gateway: every alert is deduplicated by fingerprint, enriched with service/team/severity tags, and checked against maintenance windows before it's allowed to page anyone.
• Routing table: maps service ownership and team schedule (including timezone-local business-hours windows) to whoever is currently on duty, kept in the paging tool as the single source of truth rather than a wiki page someone forgets to update.
• Severity model: decides who gets paged first and how many people, based on impact type, not just raw error rate.
• Escalation ladder: a fixed sequence of who gets paged next if no one acknowledges, with a hard time budget at each step.

Severity and routing matrix

Escalation flow

flowchart TD
  A[Alert fires] --> B[Gateway: dedupe, enrich, tag severity]
  B --> C{Severity type}
  C -->|Revenue-impacting| D[Page Primary, ack SLA 5m]
  C -->|Data-sensitive| E["Page Data on-call AND Compliance lead in parallel; notify Legal/Compliance on fixed clock, independent of ack"]
  D --> F{Acked by 5m?}
  F -->|No| G[Escalate to Secondary, ack SLA +10m]
  G --> H{Acked by 15m?}
  H -->|No| I[Escalate to Service Lead, open incident bridge]
  F -->|Yes| J[Primary mitigates]
  H -->|Yes| J
  E --> K{Acked by 5m?}
  K -->|No| L[Escalate both chains in parallel, ack SLA +10m]
  K -->|Yes| M[Data on-call + Compliance mitigate]
  L --> M

Preventing alert-noise from cascading into unnecessary escalations

Most alert storms come from one root cause tripping many downstream checks at once (a database going down pages every service that depends on it). The gateway groups alerts by a correlation key (same root dependency, same time window) before routing, so the escalation ladder above runs once for the incident, not once per symptom. Escalation timers also only start on the first page for a correlated group; late-arriving duplicates reset nothing.

Keeping the matrix trustworthy

A routing matrix that's wrong is worse than no matrix, because it creates false confidence. Two things keep it honest: primary/backup contacts are pulled live from the scheduling tool rather than hand-maintained, and if the on-duty person marks themselves absent (leave, travel) in that same tool, pages route to the next person automatically instead of timing out silently first. A silent timeout during a real on-call absence is exactly the failure mode that erodes trust in the whole system.

Trade-offs and pitfalls

Stricter deduplication reduces noise but risks folding two genuinely unrelated incidents into one correlation group if the correlation key is too broad; the fix is scoping correlation to a real dependency graph, not just a time window. A common wrong turn is building one severity ladder for everything, which either pages security teams for routine downtime or under-escalates a compliance-relevant incident because it didn't look revenue-critical on the dashboard. The severity model has to be impact-type aware, not just impact-size aware.

Direct answer
A canary deployment ships a new version to a small slice of traffic first, watches it closely against the stable version, and only widens exposure if it looks healthy; if it doesn't, you pull the plug on a small fraction of users instead of everyone.

Structured elaboration

1. Initial slice: route a small percentage of traffic, often 1-5%, to the new version while the rest continues on the stable version.
2. Observe: compare metrics between the canary and the stable baseline over the SAME time window, not the canary against yesterday's numbers, since traffic patterns shift by time of day.
3. Decide: if the canary's metrics stay within an acceptable band of the baseline for long enough, promote to a larger percentage; if they degrade, roll back the canary slice.
4. Ramp: repeat at increasing percentages (for example 5% -> 25% -> 100%) rather than jumping straight to full traffic, since a problem that only shows up under real production load or a particular traffic mix might not surface at 1%.
5. Promote or rollback trigger: could be a manual decision from a dashboard, or automated based on a metric threshold; either way it needs an explicit, pre-agreed criterion, not "it felt fine."

Worked example
A checkout service canaries a payment-processing change at 2% of traffic for 30 minutes. Error rate on the canary stays at 0.15% versus the stable version's 0.12%, well within the agreed 0.5% absolute-difference tolerance, so the team promotes to 25% for another 30 minutes, then to 100%.

Trade-offs and pitfalls
Canary buys you a much smaller blast radius than a straight rollout, but it's slower to reach full deployment and needs enough traffic volume for the canary slice to be statistically meaningful; a low-traffic service at 1% might only get a handful of requests, which isn't enough to detect a real but modest regression. The common mistake is treating a clean canary window as proof of correctness rather than as reduced risk: rare edge cases and slow-building problems (a memory leak, a cache-warming issue) can still slip through a short canary window.

Direct answer: To make eventual consistency between service A and B idempotent and reconciliation-friendly, service A publishes events with a stable event ID (or a monotonic sequence number per entity), service B's consumer deduplicates on that ID before applying any change, and a periodic reconciliation job independently compares A's and B's views to catch and repair anything that slipped through despite the idempotency guarantees.

Structured elaboration

Idempotent event processing on the consumer side. Every event from A carries a stable identifier; B's consumer checks (atomically, alongside applying the event) whether that ID has already been processed, using the same "dedup record plus the actual state change in one transaction" discipline as any idempotent write. This is what makes at-least-once delivery (which any reasonable messaging setup between A and B will actually provide) safe: redelivery is a no-op rather than a duplicate application.

Preserving ordering where necessary. If events for the same entity must be applied in order (e.g. "created" before "updated" before "deleted"), B's consumer needs either a strictly-ordered delivery channel per entity (partition by entity ID) or an explicit sequence number in each event that B checks against the last-applied sequence for that entity, rejecting or buffering an out-of-order arrival rather than applying it prematurely.

Supporting replays. Because B's state can, despite everything, still drift from A's (a bug, an extended outage, a schema-migration mistake), the design should support REPLAYING A's full event history into B from scratch (or from a checkpoint) to rebuild B's view, which requires A to retain (or be able to regenerate) its event history for at least as long as any realistic replay window, and requires B's apply logic to be safe to run repeatedly over the same events (which it already is, by the idempotency design above).

Reconciliation as the safety net, not the primary mechanism. A periodic job independently compares A's and B's data (via checksums, row counts, or a full diff on a schedule appropriate to the data's size and criticality) and either auto-repairs small, well-understood divergences or flags larger ones for human review. This is deliberately a SEPARATE mechanism from the event-driven sync path, its job is to catch failures of that path (a dropped event no retry ever recovered, a bug in the consumer's apply logic), not to be the primary way B stays in sync (that would defeat the point of event-driven propagation in the first place).

Worked example. Service A (an Orders service) publishes OrderUpdated{order_id, sequence, payload} events. Service B (a search index) consumes them, checking (order_id, sequence) against the last sequence it applied for that order, skipping (as an idempotent no-op) any event with a sequence it's already seen or older, and buffering (briefly) any event that arrives out of order, applying it once the gap is filled or timing it out into a "request full replay for this order_id" fallback if the gap doesn't close. Nightly, a reconciliation job compares a sample (or full set, for smaller datasets) of orders between A's source of truth and B's index, flagging any order where B's data doesn't match A's for investigation, this is how the team discovered a bug where B's consumer was silently dropping events during a brief scaling event, well before any customer noticed stale search results.

Trade-offs and pitfalls. Skipping the reconciliation job because "the event pipeline is reliable" is a common and risky shortcut, event-driven consistency mechanisms fail in ways that are often invisible until reconciliation (or a customer complaint) surfaces them, since a missed event usually produces no error, just quietly stale data.

Clarify requirements

• Retention: mixed (e.g., 1 year SOC2, 7 years PCI artifacts)
• Constraints: cost-sensitive, immutable, searchable for audits, role-based access, tamper-evidence

High-level architecture

• Ingestion layer: Fluentd/Vector/Logstash agents → Kafka or Kinesis for buffering and resilience
• Hot indexing/search: ELK/Opensearch or Splunk Cloud for 90–180 days (fast queries, dashboards)
• Cold immutable archive: Object store with WORM + cold tier (e.g., AWS S3 with Object Lock in Governance/Compliance mode → transition to Glacier Deep Archive after X days)
• Catalog/index pointer store: Lightweight metadata DB (DynamoDB/Postgres) holding object ids, time ranges, hashes, and search pointers for archived logs
• SIEM integration: Forward alerts/events from hot index; ingest audit retrieval requests into ticketing workflow

Ingestion & indexing

• Agents push structured JSON with metadata (tenant, system, retention class, checksum)
• Stream processor enriches and routes: short retention → index; long retention → both index (if needed) and archive
• Apply hashing (SHA-256) and sign with KMS/HSM; store hash in metadata DB

Immutable storage & retention management

• Use Object Lock (WORM) or vendor immutability for required windows; set retention per artifact class
• Lifecycle policy: hot index retention policy → snapshot to S3 with key prefix and metadata → apply Object Lock → transition to Glacier after 30/90/180 days per cost plan
• Periodic integrity checks: re-hash objects, compare with stored value, log results to audit trail

Access controls & security

• Fine-grained IAM roles, least privilege, MFA for access to retrieval APIs
• Encryption at rest (KMS/HSM) and in transit (TLS)
• Audit logs for access to archived objects stored separately and retained per compliance rules
• Separation of duties: Requests for retrieval must be approved by compliance and logged

Retrieval / audit workflow

• Query metadata DB or hot index; if object archived, generate retrieval job via API (pre-authorized, audited)
• For cold retrieval, automatic workflow: approve → initiate restore → notify auditor → mount/temporary access to restored data into read-only S3 prefix or ephemeral search index
• All retrievals emit audit events and expiration for restored copies

Cost optimization

• Tiering: keep only N days in hot index (expensive), move everything else to S3 + Glacier
• Compression and chunking of logs before archive (parquet/NDJSON gz)
• Use metadata/index to avoid full restores for common queries (store summarized indexes)
• Retention classes per artifact to avoid 7-year retention for everything

Trade-offs

• Fast search vs cost: keep small searchable indexes for critical artifacts; full-text for recent window only
• Complexity: metadata catalog adds complexity but prevents expensive restores

This design provides durable, tamper-evident storage, fast recent search, controlled retrieval workflows for audits, and clear cost/retention trade-offs appropriate for a Systems Engineer to implement.

Workshop Title: Problem-solving & Communication for Systems Engineers — 60 min

Agenda (with times)

• 0–5 min: Welcome, objectives, norms
• 5–15 min: Short model: DESC for communication + structured troubleshooting (Hypothesis → Test → Iterate)
• 15–30 min: Exercise 1 (Individual): Fault triage
• 30–45 min: Exercise 2 (Pairs): Design trade-off discussion
• 45–55 min: Exercise 3 (Group): Incident postmortem roleplay
• 55–60 min: Wrap-up, takeaways, homework

Exercise 1 — Fault Triage (15 min, individual)

• Prompt: Given a service with high latency, logs show increased DB query times and a backlog on worker queue. List top 5 hypotheses, choose two to test, and write 3 commands/queries to run.
• Deliverable: Hypotheses + test steps
• Purpose: Structured reasoning, prioritization

Exercise 2 — Design Trade-off (15 min, pairs)

• Prompt: Choose between vertical scaling vs. sharding for a legacy datastore. One plays PM, one SE. 8 min to prepare, 7 min to present and defend.
• Deliverable: Decision, trade-offs, migration plan
• Purpose: Communication with stakeholders, cost/risks reasoning

Exercise 3 — Incident Postmortem Roleplay (10 min, group)

• Prompt: 2 simulated outages; group assigns roles (SE, SRE lead, QA). Produce 5 action items and root cause summary.
• Deliverable: 5 actionable follow-ups with owners
• Purpose: Cross-team communication, blameless analysis

Evaluation Rubrics (score 1–4)

• Communication
  • 4: Clear, concise, uses metrics, aligns audience, summarizes decisions
  • 2: Some gaps, occasional jargon, unclear next steps
  • 1: Rambling, no clear ask or actions
• Reasoning & Problem-solving
  • 4: Logical hypothesis chain, prioritized tests, risk-aware trade-offs
  • 2: Missing prioritization or tests not actionable
  • 1: No coherent plan or incorrect assumptions
• Collaboration (for pair/group)
  • 4: Balanced participation, resolves disagreements constructively
  • 1: Dominates or withdraws

Facilitator Hints

• Start with leading questions, not answers (e.g., “What observable data would change your hypothesis?”)
• Use graduated nudges: hint → suggest a metric → show a sample log only if stuck
• Avoid solving; model structuring (write hypotheses on board)
• Timebox tightly; cut hints if the group is making progress

Measuring Effectiveness After One Month

• Short survey: self-rated confidence (pre/post), 3 specific behavior questions (use of hypothesis-driven troubleshooting, documenting incident actions, stakeholder summaries)
• Review 4 anonymized runbooks/incidents for evidence of structured triage and clear action items
• Manager check-in: observed changes in on-call handoffs and postmortem quality
• Target: 30% increase in documented hypothesis-driven steps and 20% faster mean time to identify root cause in sampled incidents

This plan focuses on practical, cloud/infrastructure scenarios, measurable outcomes, and coaching-style facilitation to build durable skills.

Clarify assumptions

• 1M req/day total; 10% of daily traffic occurs in a 2‑hour peak window → 100k requests in 2 hours.
• Average CPU per request = 25 ms of vCPU time.
• Requests are stateless; memory per request negligible.
• Target p95 latency = 200 ms (includes request CPU + queuing/network).

Step 1 — Peak throughput

• Peak window requests: 100,000 over 2 hours → requests/sec:

req_per_sec_peak = 100000 / (2 * 3600) = 13.89 rps ≈ 14 rps

Step 2 — vCPU capacity needed

• One vCPU can provide roughly 1 second / 0.025s = 40 request CPU slots/sec (if single-threaded and fully utilized). Conservatively allow 70% CPU utilization for headroom:

effective_rps_per_vcpu = 40 * 0.7 = 28 rps
vcpus_needed = ceil( req_per_sec_peak / effective_rps_per_vcpu ) = ceil(14 / 28) = 1 vCPU

• For resilience and multi-threading, choose minimum 2 vCPUs per instance.

Step 3 — Instance count

• With 2 vCPUs/instance → capacity ~56 rps. Instances required at peak:

instances_at_peak = ceil(14 / 56) = 1 instance

• For high availability and rolling upgrades, run at least 2 instances. So initial fleet = 2 instances (2 vCPUs each).

Step 4 — Autoscaling policy (initial)

• Metrics: scale on CPU utilization and request rate per instance.
• Target: keep CPU ~50–65% to meet p95 latency.
• Scale out: add 1 instance when avg CPU > 65% for 60s OR per-instance rps > 40 for 60s.
• Scale in: remove 1 instance when avg CPU < 30% for 5 minutes and instances > 2.
• Cooldown: 2 minutes after scale out, 10 minutes after scale in.
• Max instances: 10, Min instances: 2.

Rationale & next steps

• Conservative headroom preserves p95 latency; HA requires >=2 instances.
• Validate with load test, measure tail latencies, adjust vCPU per instance and scaling thresholds. Add autoscaling by request queue length or latency if supported.

Direct answer: The immediate priority is to stop the slowdown from consuming your own resources: set aggressive timeouts, open a circuit breaker so you stop calling the failing dependency, and isolate the connection/thread pool used for that call so it can't starve everything else. The longer-term fix is architectural: decouple the caller from the dependency's latency entirely, usually via an async queue or by making the call non-blocking, so a slow downstream degrades throughput instead of taking the whole service down with it.

Structured elaboration

Why this happens (the mechanism): by Little's Law, the number of requests in flight $L$ equals arrival rate $\lambda$ times the time each request spends in the system $W$: $L = \lambda W$. If a downstream call's latency goes from 50ms to 500ms while your request rate stays at, say, 200 requests/second, the in-flight count grows from $L = 200 \times 0.05 = 10$ to $L = 200 \times 0.5 = 100$, a 10x increase, purely from the latency change with no change in incoming traffic. If your thread or connection pool was sized for ~10-20 concurrent in-flight requests to that dependency, it's now exhausted, and requests start queueing on your side, which is exactly the symptom described.

Immediate mitigations (minutes, not a redesign):

Longer-term architectural fix:

• Decouple via an async queue: put a durable queue between the caller and the slow dependency so the caller can return quickly (accept-and-acknowledge) and the dependency is drained at its own sustainable pace, rather than the caller blocking on it synchronously. Trade-off: the caller can no longer return a synchronous success/failure for that operation; the interaction model has to change to something the client and product can tolerate (a "pending" state, a webhook, a poll).
• Idempotent retries with backoff and jitter: if retries are needed, they must be capped, exponential, and jittered so a fleet of callers doesn't retry in lockstep and re-create the exact overload it's recovering from. Trade-off: added complexity, and retries must be provably idempotent on the downstream side or they risk duplicate side effects.
• Capacity planning against the tail, not the average: provision the dependency (or the pool sized to call it) based on observed p99 latency, not p50, since it's the tail that determines when queues start building. Trade-off: costs more standing capacity for headroom that's idle most of the time.

Applying this to concrete variants of the same pattern: the reasoning above is the same whether the slow dependency is a payment-validation service (immediate: circuit breaker + fast-fail with a clear "try again" to the user rather than a silent hang; long-term: async payment confirmation via webhook), a message-queue consumer falling behind (immediate: shed or dead-letter the oldest low-priority messages, bulkhead the consumer pool by message type; long-term: scale consumers horizontally and partition by priority), a retry storm from a flood of client-side 503s (immediate: the client-side backoff-with-jitter above is the direct fix; long-term: make the shedding threshold adaptive so it doesn't itself become the trigger for a thundering herd), or a synchronous order-processing pipeline backing up (immediate: bulkhead the slow stage's pool; long-term: convert that stage to the async-queue pattern above).

Trade-offs & pitfalls

• Every immediate mitigation above trades some availability or correctness for stability: timeouts drop requests that might have succeeded, circuit breakers deny service during their open window, load shedding sacrifices some requests to save the rest. The point isn't to avoid the trade-off, it's to make it deliberately and visibly rather than let an unbounded queue make it for you via an eventual crash.
• A common wrong turn: adding retries as the first response to a slowdown. Naive retries without backoff amplify load on an already-struggling dependency and can turn a partial slowdown into a full outage (a retry storm).
• Circuit breakers and bulkheads need to be tuned against real traffic and latency distributions; thresholds copied from a different service's runbook are a common source of either false trips (unnecessary unavailability) or no protection at all (thresholds too loose to matter).

Direct answer

A tags policy denies any Terraform resource change whose after state has a missing tags attribute, or a tags map without both owner and environment keys, evaluated against Terraform's JSON plan output. Wired into CI as a conftest (a CLI that runs Rego policies against JSON/YAML input) or opa eval step that runs after terraform plan, it fails the pipeline (non-zero exit) before merge, so terraform apply never runs against a non-compliant resource.

Approach

Read Terraform's plan as JSON (terraform show -json), walk resource_changes, skip anything being deleted (a destroy has no future tags to check), and for everything else assert after.tags exists and contains owner and environment. Emit one deny message per violation so CI can print exactly which resource and which key failed, not just "policy failed."

Policy (Rego)

package terraform.tags

import rego.v1

deny contains msg if {
	some rc in input.resource_changes
	some action in rc.change.actions
	action != "delete"
	attrs := rc.change.after
	tags := object.get(attrs, "tags", null)
	not is_map(tags)
	msg := sprintf("resource %v missing tags map", [rc.address])
}

deny contains msg if {
	some rc in input.resource_changes
	some action in rc.change.actions
	action != "delete"
	attrs := rc.change.after
	tags := object.get(attrs, "tags", null)
	is_map(tags)
	not tags.owner
	msg := sprintf("resource %v missing tags.owner", [rc.address])
}

deny contains msg if {
	some rc in input.resource_changes
	some action in rc.change.actions
	action != "delete"
	attrs := rc.change.after
	tags := object.get(attrs, "tags", null)
	is_map(tags)
	not tags.environment
	msg := sprintf("resource %v missing tags.environment", [rc.address])
}

is_map(x) if {
	x != null
	type_name(x) == "object"
}

Key points

• Iterate resource_changes, not the raw HCL, so the policy sees what will actually be created or changed, including values interpolated from variables and modules.
• Exclude pure delete actions with some action in rc.change.actions; action != "delete": this is an existential check, so a replace (["delete", "create"]) still gets validated on its create half, while a plain destroy (["delete"]) is correctly skipped.
• Default the tags lookup with tags := object.get(attrs, "tags", null) before ever calling is_map on it. Do not call is_map(attrs.tags) directly: when attrs.tags does not exist, referencing it produces no value at all, and in Rego, negating a call whose argument is itself undefined never resolves to true or false, it just never fires. object.get always returns a defined value (null when the key is absent), so not is_map(tags) becomes a normal negation over a normal function result and correctly denies the "no tags key at all" case.
• One deny rule per failure mode (missing map, missing owner, missing environment) so the CI output tells the developer exactly what to fix.

Complexity and edge cases

Evaluation is O(number of resource_changes) with constant work per resource: no recursion, no external calls. Edge cases worth naming explicitly: resource types that do not support tags at all (need an allowlist of resource types, otherwise this policy false-positives on them); for_each/count resources, which appear once per instance in resource_changes and are each checked independently, so a module with ten instances produces ten independent checks; and a tags value that depends on another resource not yet created, which the plan reports as unknown rather than present. A naive check treats "unknown at plan time" the same as "missing," which is a false positive; a stricter version would inspect rc.change.after_unknown.tags and only deny when the value is genuinely absent, not merely unresolved yet.

Wiring into CI

• Generate the plan in CI: terraform plan -out=plan.binary && terraform show -json plan.binary > plan.json.
• Evaluate it: conftest test --policy ./policy plan.json, or opa eval --input plan.json --data policy.rego "data.terraform.tags.deny".
• Treat any non-empty deny set as a pipeline failure and make that check required on the PR, not advisory, so a merge is mechanically blocked, not just discouraged.
• Surface the deny messages as a PR comment or check annotation so the resource address and missing key are visible without digging into CI logs.

Trade-offs & pitfalls

• A policy this strict on day one breaks every existing untagged module. Roll it out in warn-only mode first, generate a report of current violations, and flip to blocking once the backlog is cleared, not the other way around.
• Checking the JSON plan catches drift-introducing changes before apply, but it cannot catch tags removed later by someone with direct cloud console access. That needs a separate periodic scan against live resources (a detective control), not just this preventive one.
• The after_unknown case above is easy to miss in a first pass and is exactly the kind of thing that turns into a noisy false positive once real modules with cross-resource references start hitting the policy.
• Rego's "negating a call over an undefined argument never resolves" behavior is a well-known footgun: a not f(x) guard that looks correct in review can silently never deny anything for the exact input it was written to catch. Always test the "attribute completely absent" case with the real opa binary, not just the "attribute present but wrong" case, since the two paths through a guard function can diverge.
• This policy is written in OPA v1 syntax (deny contains msg if { ... }), which has been the default parser since OPA 1.0 (2024). An older pinned binary that still defaults to v0 needs either an upgrade or the --v0-compatible flag; don't ship v0-only syntax and call it current.

Direct answer
This is a behavioral question best answered with a specific story showing you distinguished a genuine pattern from a one-off, kept the accountability conversation separate from the group postmortem, and led with coaching rather than punishment, while still being clear about the standard expected going forward.

Structured elaboration
A strong answer covers: the situation (what happened, why it warranted an accountability conversation rather than just the standard blameless process), how you determined it was a genuine pattern rather than bad luck, how you had the conversation itself (privately, focused on support and root cause rather than judgment), and the outcome, including how you protected the person's and the team's trust in future incident reporting.

Worked example
"An engineer on my team was involved in a second incident within two months, both traced to skipping the same pre-deploy verification step under time pressure. Before assuming it was negligence, I checked whether this was a genuine pattern: both incidents shared the same specific gap, so it was, rather than two unrelated issues that happened to involve the same person. I had a private conversation, not as part of the group postmortem, focused on understanding what was driving it. It turned out they'd been covering an unsustainable amount of on-call load for a teammate on leave and had started rushing to keep up. I addressed both sides: the workload issue got fixed by redistributing on-call coverage, which was the real systemic driver, and I paired them with a more experienced engineer on deploy discipline for a few weeks as direct support, framed clearly as investment in them, not punishment. Separately, the second incident's actual group postmortem stayed fully blameless and produced an automated pre-deploy gate that makes that specific step impossible to skip regardless of who's deploying, which is the fix that protects the whole team, not just this one person. Three months later there'd been no recurrence, and, just as important, the engineer's incident-reporting behavior stayed just as open as before, which told me the accountability conversation hadn't damaged their trust in the process."

Trade-offs and pitfalls
The most common weak answer either skips the distinction between pattern and coincidence (treating any second incident as automatic proof of a problem) or focuses entirely on the disciplinary angle without addressing what was actually driving the pattern, which misses the more senior insight that repeated mistakes are frequently a symptom of a systemic issue (workload, unclear process, inadequate training) rather than a character flaw.