Solutions Architect (Staff Level) Interview Preparation Guide - FAANG Standards

Direct answer

Prioritize by expected business impact combined with how quickly the failure mode compounds if unaddressed: a region outage first, because it's a full-availability event with no partial-degradation option; a sudden 10x traffic spike second, because it threatens availability but usually has partial mitigations (throttling, degraded modes) available immediately; and an upstream schema change third, because it's typically detectable and containable with fast rollback before it causes user-facing damage, even though it can silently corrupt data if left uncaught.

Structured elaboration

For each curveball, separate the immediate runbook response from the longer-term architectural change it justifies.

Region outage. Immediate: fail over reads and writes to a secondary region using health-checked traffic routing, and pause non-essential batch work to reduce write pressure during the transition. Architectural change: multi-region active-passive (or active-active) replication for the data layer, with regularly rehearsed failover drills; a design that was never built to fail over won't fail over correctly under real pressure, only under a rehearsed one.

Sudden 10x traffic spike. Immediate: autoscale the serving tier, shed or degrade non-critical functionality (serve cached or slightly stale results rather than fail outright), and throttle low-priority background jobs to protect the real-time path. Architectural change: pre-warmed capacity headroom, adaptive rate limiting, and a defined degraded mode that's tested before it's needed, not designed during the incident.

Upstream schema change breaking the data pipeline. Immediate: fail fast on schema-validation errors at ingestion rather than let malformed data propagate, quarantine the bad batch, and roll the downstream transform back to the last known-good schema. Architectural change: enforce a schema contract at the pipeline boundary (a strongly typed serialization format with a compatibility check, such as Avro or Protocol Buffers) so a breaking upstream change is caught at ingestion rather than discovered downstream after it has already corrupted derived data.

Worked example

An illustrative prioritization exercise, scoring each curveball on business impact (1 low to 5 high) and detectability/containability (1 hard to 5 easy) to make the ranking auditable rather than a gut call: region outage scores high impact (5/5: full outage, all users) and moderate containability (3/5: requires a rehearsed failover, not just a code fix); 10x traffic spike scores high impact if unmitigated (4/5) but higher containability (4/5: autoscaling and shedding are standard, fast-acting levers); schema break scores lower immediate user-facing impact (2/5: the pipeline can often keep serving stale-but-correct data while paused) but containability that depends entirely on whether validation exists at the ingestion boundary (2/5 without it), if it doesn't, undetected corruption can silently spread for a long time before anyone notices, which is exactly why validation is the priority architectural investment for that curveball specifically, even though it's ranked last for immediate response.

Trade-offs & pitfalls

• Ranking these purely by which is scariest in the abstract, rather than by business impact and how fast each compounds if left unaddressed, produces a plausible-sounding but ungrounded priority order; tie the ranking to a concrete criterion.
• A schema break that lacks ingestion-time validation is deceptively low-priority in the short term and highest-priority for silent, compounding damage; don't let "least immediately visible" become "least urgent to architect for."
• Building all three mitigations simultaneously from scratch during a single design pass is rarely realistic; sequence the architectural investments and say explicitly which curveball's mitigation ships first and why.
• Rehearsing failure (game days, chaos testing, restore drills) is what turns a runbook from theory into something that actually works under pressure; a runbook that has never been executed is a plan, not a capability.

Direct answer. Pick one system, say what threat you designed against, name the two or three decisions you made early, name what you pushed back on and why, and finish with how you verified it was safer. Below is a story shape using a plausible example (a document-processing service for many customers); replace the details with your own real system.

Situation. A team was designing a service where many customers (tenants) upload documents that are parsed and searched. The design was one shared database and one shared storage bucket, with tenant filtering done in application code. I joined at the design stage, before any code existed.

Task. Make cross-tenant data exposure structurally hard, without delaying the first release.

Actions I took

• Decided on isolation by construction. Each tenant's objects went under a tenant-scoped prefix (a folder-like path such as tenant-123/ in the bucket, one per customer), and the service accessed storage using short-lived credentials (access keys that expire after minutes, so a leaked one soon becomes useless) restricted to that prefix. The point: a bug in the filter code can no longer read another tenant's files, because the credential itself cannot. This is least privilege (give a component only the access it needs) and blast-radius limiting (shrink what one failure can reach).
• Closed the shared-database gap. Storage prefixes do not protect rows in the shared database, where the filter in application code was still the only barrier. I asked for row-level security (the database itself refuses rows whose tenant column does not match the session's tenant, so a missing filter in code returns nothing instead of another customer's data), and I made the cross-tenant test cover API reads of database records as well as files. Had the team not been able to do that before launch, I would have recorded the database as a dated, owner-signed risk acceptance instead of calling the design safe.
• Pushed back on a shared admin credential the team wanted for the background workers to "keep it simple". I offered a per-job scoped credential instead, with the same developer effort because a platform library already issued them. The exchange went roughly like this (illustrative). Team: "One admin credential for the workers keeps it simple." Me: "A credential scoped to one tenant's prefix costs you the same one line from the platform library, and a worker bug can then only touch one tenant's files." The team agreed because the cost to them was unchanged.
• Pushed back on logging full document contents for debugging. I proposed logging document IDs and sizes, which kept debugging possible and removed a second copy of the sensitive data.
• Kept the review early and small. I asked for a one-page data-flow diagram and gave feedback within days so design was not stalled.

Result and how we knew it was safer. I did not rely on opinion. We measured with three checks:

1. A cross-tenant test in the pipeline: a test tenant tries to read another tenant's object by guessing the ID and must get a denial. It ran on every build, so the control could not silently regress.
2. A pre-launch penetration test (an authorised, simulated attack by security specialists) focused on tenant isolation, with no cross-tenant read found.
3. An alert on denied cross-tenant attempts, so after launch we could see the control working and see any probing.

The effectiveness measure for an early control is the count of failing cross-tenant tests (should stay at zero) plus the number of denied attempts that reach the alert. I would avoid quoting a made-up percentage; the honest claim is "for stored files, and for database rows once row-level security was in place, the class of bug was removed, and a test proves it stays removed".

What I would do differently. I would have written the cross-tenant test before the design review ended, not after, so the requirement was executable from day one.

Pitfalls. Do not tell a story where you only said no. Strong stories show a pushback with a cheaper alternative offered, and an outcome that someone other than you could verify.

RTO (Recovery Time Objective) is how long you're allowed to be down: the maximum acceptable gap between an outage starting and service being restored. RPO (Recovery Point Objective) is how much data you're allowed to lose: the maximum acceptable gap, measured in time, between the last durably captured write and the moment of failure. The requirement "payments must be restored within 30 minutes with no more than 5 minutes of data loss" is literally RTO = 30 min and RPO = 5 min stated in plain language, and each number drives a different part of the design.

What each number drives

RPO = 5 minutes drives replication and backup frequency. A nightly or even hourly backup can't meet this: if the outage happens 4 hours after the last backup, you'd lose 4 hours of transactions, not 5 minutes. A 5-minute RPO effectively requires continuous replication (near-synchronous in-region, or streaming WAL (write-ahead log: a durable, ordered record of every change, written before it's considered applied) or CDC (change-data-capture: a stream of those same row-level changes read off that log) shipping to the DR site) with replication lag actively monitored and alarmed well below the 5-minute budget, plus point-in-time recovery for protection against logical corruption that replication alone would just copy.

RTO = 30 minutes drives standby readiness and failover automation. A cold-standby DR site that has to be provisioned from scratch after the fact will blow past 30 minutes just on infrastructure boot time. A 30-minute RTO points toward a warm standby (already running, sized down, kept current via the same replication that satisfies the RPO) with an automated failover runbook: health-check detection, automated promotion, and DNS/routing cutover, because a manual, human-paged process realistically eats 10-15 minutes just in detection and decision-making before any recovery action starts.

Worked example: what these numbers cost against an annual SLA

A useful way to make the 30-minute number concrete is to check it against annual downtime budgets at standard availability tiers, using $525{,}600$ minutes per year (365 $\times$ 24 $\times$ 60):

A single incident with a 30-minute RTO, if the service is held to a 99.99% SLA, consumes:

of the entire year's downtime budget in one event. That reframes "30 minutes sounds generous" into "this design can absorb roughly one such incident a year and still hit four nines," which is exactly the kind of number that should drive whether the DR design gets warm-standby automation now or gets revisited after the first real incident eats most of the annual budget.

Trade-offs and pitfalls

The most common mix-up is treating RTO and RPO as interchangeable "how bad was it" numbers instead of two independent design constraints: a system can have a great RTO (back up in 2 minutes) and a terrible RPO (lost the last hour of writes) if it fails over to a backup instead of a live replica, or the reverse (RPO≈0 via synchronous replication, but a slow, manual promotion process blows the RTO). Both have to be solved, and usually by different mechanisms: RPO is a replication/backup-cadence problem, RTO is an automation/standby-readiness problem. A second pitfall specific to payments: RPO=0 sounds like the obviously "safer" number to chase, but strict synchronous replication that blocks writes during a replica outage can turn a replication hiccup into an availability incident, trading a data-loss risk you might never hit for a downtime risk you're now taking on every day.

Requirements & constraints:

• Business: RTO/RPO targets, expected QPS, data sovereignty/compliance, budget cap, expected growth (3–5 yrs).
• Non‑functional priorities (example): Availability and backup most important for OLTP; performance and vendor‑lockin matter more for latency‑sensitive apps.

Scoring model (example weights totaling 100):

• Availability / SLA & SLO adherence — 25
• Performance (latency, throughput under load) — 20
• Backup & Recovery (RTO, RPO, restore testing) — 15
• Compliance & Data residency — 12
• Operational Tooling & Observability (monitoring, automated ops) — 10
• Vendor lock‑in (proprietary features, data exportability) — 8
• Migration cost & effort (data movement, schema changes, downtime) — 7
  Scoring scale: 0–5 per criterion; normalized by weight.

Evidence gathering:

• SLA analysis: compare contractual SLAs, credit terms, historical uptime reports, published SLOs.
• Benchmarks: run focused microbenchmarks (read/write latency p95/p99, transactions/sec) and realistic workload replay using stored production traces. Use standardized tools (sysbench, YCSB) and run across regions/instance sizes.
• PoC: deploy a representative workload for 2–4 weeks exercising failover, backups, scaling, schema changes, and security features. Execute chaos tests (node kill, network partition) and full restores from backups.
• Cost modeling: TCO for 3 years including egress, backups, instance sizes, licensing, and expected autoscaling.
• Compliance validation: third‑party audits (SOC/ISO), data residency controls, encryption at rest/in transit, access controls.

Sensitivity & risk analysis:

• Run sensitivity scenarios by varying weights (e.g., +50% to availability or +50% to migration cost) and re-score to identify decision stability.
• Monte Carlo or deterministic scenarios for performance variance and cost spikes (egress, storage growth).
• Highlight criteria thresholds where rankings flip.

Recommendation & presentation:

• Deliver an executive summary with top pick, score breakdown chart, and one‑page risk/mitigation map.
• Provide appendix: raw benchmark data, PoC runbooks, cost model, SLA excerpts.

Migration risks & contingency plans:

• Risks: data egress costs, hidden incompatibilities (SQL dialect/features), failover behavior differences, longer cutover, runbook gaps.
• Contingencies: staged migration (dual‑write / read‑replica sync), blue/green cutover, rollback plan with verified snapshot restores, pilot on noncritical tenant, contracts with exit terms and support SLAs for migration assistance.
• Operational readiness: runbooks, run‑throughs, runbook automation, training, and a 72‑hour escalation matrix with vendor.

Outcome justification: show weighted scores, sensitivity stability, PoC evidence that meets RTO/RPO and perf targets, and a migration plan that reduces business risk — this provides a defensible, auditable vendor selection for stakeholders.

Direct answer

Before committing to a lateral pivot, prove the fit cheaply and prove the readiness credibly. Validate genuine interest and aptitude through a low-commitment experiment, a rotation, a shadow assignment, a small real project in the new discipline, before asking for the move, and build a small portfolio of evidence in the destination discipline's own terms, not your current discipline's terms.

Structured elaboration

Separate validating fit from proving readiness, they use different evidence. Fit is whether you actually enjoy and are suited to the day-to-day of the new discipline, learned through direct, low-stakes exposure. Readiness is whether you can perform credibly at an entry level in the new area, proven through a real deliverable.

Validate fit cheaply first. Shadow someone already doing the destination role for a defined period, take on a small real piece of that work alongside your current job, or an informal rotation if your organization supports one. The goal is finding out, before committing a year of your career, whether the actual daily texture of the work matches what you imagine it to be.

Prove readiness in the destination discipline's terms. A common mistake is presenting your current discipline's evidence and expecting it to translate automatically. It rarely does. A few illustrative pairs and what the evidence tends to look like:

• Moving from an engineering role toward product: a small product decision you drove, with the reasoning about user or business trade-offs made explicit, not just a technically strong build.
• Moving from an individual contributor (IC) technical role toward research: a well-scoped investigation with a clear question, method, and honestly reported result, not just a strong implementation.
• Moving from an analyst role toward engineering: something you built that runs reliably and that others depend on, not just an analysis that was correct once.

Build the relationships the destination discipline actually relies on before you need them for the move, so the people who'd eventually evaluate you already have direct exposure to your work in it.

Worked example

"I was drawn to an adjacent discipline but was honestly unsure whether I'd like the daily reality of it or just the idea of it. Rather than asking for the move outright, I asked to shadow someone in that role for a short period and separately took on one small, real piece of that kind of work alongside my existing responsibilities, with my manager's agreement that it was a bounded experiment, not a scope change. The shadowing told me quickly which parts matched what I expected and which didn't. The small real piece of work gave me something concrete, a deliverable that someone already doing that role could evaluate on its own terms, not on the terms of my original discipline. When I later raised the possibility of a fuller move, I brought that piece of work and named it plainly as evidence, rather than asking to be trusted based on enthusiasm alone."

Trade-offs & pitfalls

• Committing to a full pivot based on the idea of the new discipline rather than direct exposure to its actual day-to-day risks discovering the mismatch only after the move.
• Presenting evidence built for your current discipline and expecting a destination-discipline evaluator to translate it themselves. That's your job to do, not theirs.
• Treating the validation experiment as a favor you're owed rather than something you actively design and propose with a clear scope and end date, so it doesn't become an open-ended distraction.
• Be honest with yourself about a negative result. If the shadowing or small project reveals weaker fit than expected, that's a successful use of a cheap experiment, not a failure to be pushed past.

Direct answer

Sensitivity analysis in a TCO (total cost of ownership) model means changing one input at a time across a plausible range and recording how far the answer moves, so you learn which assumptions the conclusion actually depends on. You run it by choosing the output (here, which option is cheaper and by how much), setting a plausible low and high for each input, recomputing the output at each end, ranking inputs by the size of the swing, and then spending your effort firming up the top few. For example, one supplier is cheaper per unit but fails more often, so the cheaper unit price may or may not survive the cost of the failures. For a critical component, the two variables to test first are the defect or failure rate of the cheaper supplier and the cost each defect causes, because both are uncertain and both multiply every unit.

The method

1. Fix a base case and the output metric. Here: the three-year cost advantage of supplier A over supplier B, where a positive number means A is cheaper.
2. List the inputs and give each a plausible range from evidence (quotes, history, pilot data), not a blanket plus or minus 10%. A firm quoted price has a narrow range; a new supplier's defect rate does not.
3. Vary one input at a time, others at base, and record the output at the low and high end.
4. Sort by swing (a tornado chart is this table drawn as bars, longest at the top).
5. Find break-even values: the input value at which the decision flips, so the two options cost exactly the same.
6. Repeat for the top inputs together (two-way table), because inputs interact.

Worked example (illustrative inputs)

Two suppliers of a critical component, 50,000 units a year for three years. A sells at $42 with a 2.0% defect rate; B sells at $46 with a 0.5% defect rate. Each defect costs $200 (rework, expedited replacement, warranty). Expected cost per unit: A is $42 plus 2.0% of $200 ($4), so $46; B is $46 plus 0.5% of $200 ($1), so $47. A is cheaper by $1 a unit, which is $150,000 over three years.

base = dict(volume=50_000, price_a=42.0, price_b=46.0, defect_a=0.020, defect_b=0.005, defect_cost=200.0)
YEARS = 3

def advantage_of_a(p):
    """3-year cost advantage of supplier A over supplier B, in $ (positive means A is cheaper)."""
    per_unit_a = p["price_a"] + p["defect_a"] * p["defect_cost"]
    per_unit_b = p["price_b"] + p["defect_b"] * p["defect_cost"]
    return (per_unit_b - per_unit_a) * p["volume"] * YEARS

ranges = {                      # (low, high) plausible range for each input
    "price_a":     (41.0, 43.0),
    "price_b":     (45.0, 47.0),
    "defect_a":    (0.010, 0.040),
    "defect_b":    (0.003, 0.010),
    "defect_cost": (100.0, 400.0),
    "volume":      (30_000, 70_000),
}
print(f"base advantage of A: {advantage_of_a(base):,.0f}")
rows = []
for name, (lo, hi) in ranges.items():
    at_lo = advantage_of_a(dict(base, **{name: lo}))
    at_hi = advantage_of_a(dict(base, **{name: hi}))
    rows.append((abs(at_hi - at_lo), name, lo, hi, at_lo, at_hi))
for swing, name, lo, hi, a, b in sorted(rows, reverse=True):
    print(f"{name:<12} {lo:>9} to {hi:<9}  advantage {a:>10,.0f} to {b:>10,.0f}  swing {swing:>9,.0f}")
# Break-even defect rate for A, holding everything else at base
d = (base["price_b"] + base["defect_b"] * base["defect_cost"] - base["price_a"]) / base["defect_cost"]
print(f"A stops being cheaper when its defect rate passes {d:.2%}")

base advantage of A: 150,000
defect_a          0.01 to 0.04       advantage    450,000 to   -450,000  swing   900,000
defect_cost      100.0 to 400.0      advantage    375,000 to   -300,000  swing   675,000
price_b           45.0 to 47.0       advantage          0 to    300,000  swing   300,000
price_a           41.0 to 43.0       advantage    300,000 to          0  swing   300,000
defect_b         0.003 to 0.01       advantage     90,000 to    300,000  swing   210,000
volume           30000 to 70000      advantage     90,000 to    210,000  swing   120,000
A stops being cheaper when its defect rate passes 2.50%

Reading it:

• The top two are A's defect rate (a swing of $900,000) and the cost of a defect ($675,000). Both are large and both are poorly known for a supplier with no history. The quoted prices swing $300,000 each, but a signed quote can be firmed up, so they rank lower in effort.
• Volume swings the least ($120,000). The per-unit gap is only $1, so volume scales a small difference. Volume would rank first if you were asking about total spend instead of the difference between options.
• Break-even. A stops being cheaper when its defect rate passes 2.50%, only half a percentage point above the base of 2.0%. That is a thin cushion, and the decision should wait for evidence: a pilot lot or a quality audit.

Choosing which two to test first

Rank variables by uncertainty times leverage: how wrong could we be, and how much does the output move per unit of error. A high-leverage variable with a firm value (a contracted price) does not need testing; a high-leverage uncertain one (defect rate, lead-time reliability for a critical part) does.

Related tools and pitfalls

• A scenario analysis moves several inputs together into coherent stories (supplier fails audit; demand doubles). Sensitivity moves them singly. Use both.
• One-at-a-time hides interactions: a high defect rate and a high defect cost together cost more than the two swings added up, which is why the top two get a two-way table. Here is that table, A's three-year advantage in $ for A's defect rate against the cost of a defect:

Moving only the defect rate to 4.0% gives -$450,000 and moving only the defect cost to $400 gives -$300,000; moving both gives -$1,500,000, far worse than either alone, because the two multiply. Each cell is (4 + defect cost x (0.5% minus A's defect rate)) x 150,000.

• Ranges taken from a single optimistic source make the tornado look short and the conclusion look safe.
• Report the break-even, not only the ranking, because a decision-maker can judge whether 2.5% is plausible.

Direct answer

I would deploy the platform as independent regional stacks, one set inside the EU and one set inside APAC (Asia-Pacific), and give every tenant a home jurisdiction fixed at signup. All of a tenant's customer data (databases, files, backups, logs, analytics copies) lives only in stacks inside that jurisdiction. A small global layer knows only which jurisdiction each tenant belongs to, so it can route traffic without storing customer data. Accidental cross-region writes are prevented in layers: the router sends requests to the right place, each regional service refuses tenants that do not belong to it, and cloud permissions and encryption keys make writing outside the jurisdiction fail even if the code is wrong.

Terms first

• Data residency: a requirement that data is stored (and usually processed) inside a named geography, set by law, regulation or contract.
• Jurisdiction boundary: here "EU" and "APAC". Read the contracts carefully: "APAC" in a contract may mean one specific country (for example Australia only), which changes which regions count as inside.
• Failover: switching traffic to a standby copy when the primary site fails.

Architecture

flowchart TB
  U[User or API client] --> G[Global edge and router]
  G --> DIR[(Tenant directory: tenant to jurisdiction, no customer data)]
  G -->|EU tenants| EU1[EU primary: Frankfurt]
  G -->|APAC tenants| AP1[APAC primary: Sydney]
  EU1 -. replicate and back up .-> EU2[EU standby: Ireland]
  AP1 -. replicate and back up .-> AP2[APAC standby: Melbourne]

Routing

1. At login the user gets a token containing the tenant ID. The global router looks the tenant up in the directory and sends the request to that tenant's primary region. Tenant-specific hostnames (acme.eu.example.com) make this visible and cacheable in DNS.
2. The directory holds only tenant ID, jurisdiction and current primary region: configuration, not customer data. It can be replicated globally.
3. The router terminates TLS (decrypts the HTTPS connection) at the edge only to read enough to route the request. But terminating TLS means the router necessarily sees the whole request in plaintext at that moment, including its body, not just its destination hostname; so if that edge location sits outside the tenant's jurisdiction, the request body technically passes through, even briefly and even if nothing is stored, outside it. Whether that transient processing outside the jurisdiction is acceptable is a legal question; the conservative choice is regional edge endpoints for regulated tenants, so the TLS termination point itself, not just the storage, stays inside the jurisdiction.

Storage placement

• Each regional stack has its own databases, object storage, search indexes, caches and queues.
• Each region uses its own encryption keys held in that region's key management service (a managed service that generates and stores encryption keys and controls who may use them, so application code asks it to encrypt or decrypt rather than ever handling the raw key itself). A copy of EU data in APAC would be unreadable without EU keys, which gives a second line of defence.
• Global services (billing, product analytics, feature flags: runtime switches that turn a piece of behaviour on or off for specific tenants without a deploy) receive only non-personal or aggregated data. That is a data-classification decision made per field, not per service.

Failover inside the jurisdiction

• EU primary Frankfurt (AWS eu-central-1) replicates to a standby in Ireland (eu-west-1); APAC primary Sydney (ap-southeast-2) replicates to Melbourne (ap-southeast-4). If an APAC tenant's contract allows Singapore, that is an alternative standby.
• The failover runbook (a written, tested set of steps for handling this specific kind of incident, followed by whoever is on call rather than improvised in the moment) changes the directory's primary region for affected tenants to the standby in the same jurisdiction. There is deliberately no automation path that fails an EU tenant over to APAC.
• The honest trade-off: a disaster that takes out both regions in one jurisdiction means downtime for those tenants, because failing over abroad would breach the contract. State that in the SLA (service-level agreement).

Preventing accidental cross-region writes

Enforcement points, from the edge inward:

Worked example: an EU tenant during an incident

1. Acme (EU jurisdiction, primary Frankfurt) calls the API. The router reads the directory, sees eu-central-1, forwards to the EU stack.
2. Frankfurt has a regional outage. The on-call engineer runs the failover runbook: Acme's directory entry becomes eu-west-1; the Ireland standby is promoted (switched from a read-only replica into the read-write primary that now accepts Acme's writes).
3. A bug in a newly deployed reporting job tries to write Acme's export to an APAC bucket. The regional middleware passes it (the job believes it is in the EU), but the EU account's permission policy denies the call to ap-southeast-2, the write fails, an alert fires, and nothing crossed the border.
4. Acme's nightly backup copy lands in Ireland only; the global usage dashboard receives "Acme: 1.2M API calls today" and no personal data.

Trade-offs and pitfalls

• Cost: two regions per jurisdiction doubles standby cost for each jurisdiction you support, and regional stacks mean more deployments. The alternative, a single global database with row-level region tags, is cheaper but cannot give a clean residency guarantee.
• Support staff abroad viewing EU data is itself cross-border access in many legal frameworks; route support access through tooling that runs in-region with approval and audit.
• Common wrong turns: treating "stored in the EU" as enough while logs, caches, search indexes or a third-party email or monitoring vendor still copy data abroad; allowing a tenant's jurisdiction to be changed by a normal admin action rather than a controlled migration; and forgetting that global disaster-recovery automation will happily fail over across the border unless it is told not to.

Direct answer

Infrastructure monitoring tells you the pipes are flowing; it says nothing about whether the right water is flowing. Detecting semantic failures (a missing "order-created" event, a conversion rate that quietly drops) requires a second observability layer on top of infrastructure metrics: data-quality checks that compare actual event volume/shape against an expected baseline, synthetic transactions that exercise the real business flow end to end, contract checks that catch shape drift at the source, and lineage tracing that proves each business action produced its full expected downstream chain. Alerting on this layer has to be baseline-relative and multi-signal, not a single static threshold, or it drowns on-call in noise.

Structured elaboration

Data-quality (DQ) metrics. These compare what actually happened in the event stream against what should have happened, not against a fixed number:

• Reconciliation ratio: count of "order-created" events versus count of actual order rows written to the source-of-truth database in the same window. A ratio that drifts from ~1.0 means events are being dropped or duplicated upstream of the broker.
• Required-field completeness: percentage of events missing a field the contract marks required (for example, customer_id).
• Cross-event ratio: the expected relationship between two event types, such as order-created to payment-attempted. If checkout normally emits a payment attempt for close to every order, a sustained drop in that ratio is a business-logic failure even though every individual event is well-formed and every broker metric is green.
• Freshness/staleness: time since the last event of a given type arrived, per event type per tenant/region, to catch a producer that silently stopped emitting.

Synthetic transactions. Inject a canary business transaction (a fake "order") on a schedule (for example every 2-5 minutes) through the real pipeline, tagged as synthetic in a dedicated field so it never counts toward real revenue or conversion metrics. Assert that the full expected event chain (order-created, payment-attempted, payment-succeeded, fulfillment-requested) completes within the pipeline's normal latency budget. This catches "the whole business flow silently stopped" even when there is zero real traffic to notice it, and it isolates infrastructure health (can an event even complete the round trip) from organic-traffic health.

Contract checks. Validate events against a schema/contract at publish time (reject or quarantine non-conforming events rather than let them reach consumers silently) and again at the consumer boundary with consumer-driven contract tests in CI, so a producer team cannot ship a breaking field rename without a build failure on the consuming team's side. Quarantine, not silent drop: a quarantined event should itself increment a DQ metric so schema drift shows up in the same dashboard as volume drift.

Lineage tracing. Every business action gets a correlation id that is propagated through every event it produces or triggers, so you can answer "did this specific order-created event's full expected downstream chain (inventory reserved, payment charged, confirmation email sent) actually complete, and if not, where did it stop." Storing this as a queryable lineage graph (keyed by correlation id, with expected-vs-actual hop completion) turns "conversion rate dropped" from a vague signal into "42% of last hour's orders stopped at the inventory-reservation hop."

Alerting that avoids noisy false positives. Two disciplines matter most:

1. Baseline-relative, not static thresholds. Compare each DQ/ratio metric to a trailing seasonal baseline (same hour-of-day, same day-of-week average over the last N weeks) rather than a fixed number, since order volume is naturally non-stationary.
2. Multi-signal correlation before paging. Require two independent signals to agree before escalating to a page (for example, the order-created/payment-attempted ratio breaches its baseline by more than a set margin AND the lineage-completion rate for the same window also drops) while a single-signal breach opens a lower-severity ticket. This trades a small amount of detection latency for a large reduction in false pages, because most single-metric blips are noise (a slow batch of retries, a deploy-time blip) rather than a real business-logic break.

Worked example

Suppose the trailing 7-day baseline for the order-created to payment-attempted ratio, computed per matching hour-of-week, is 0.94 with a typical band of plus or minus 0.03. In the current 5-minute window the ratio reads 0.68, a drop of 0.26, which is roughly 8-9 times the normal band width, so it clears a "significant deviation" bar by a wide margin. Because this is a single signal, policy routes it to a Sev2 ticket, not an immediate page. Two minutes later the lineage tracer reports that 61% of orders in the same window stopped at the payment-attempted hop entirely (never generated the event), which is the second independent signal. With both signals agreed, the alert escalates to Sev1 and pages the on-call payments engineer, who can start from "payment-attempt emission is broken for most orders" rather than "some dashboard number looks off."

Trade-offs and pitfalls

Synthetic transactions that are not clearly tagged can pollute real conversion and revenue metrics, or worse, create real side effects (charges, emails) if they are not fully sandboxed end to end. Static thresholds age badly as the business grows or has seasonal spikes (a holiday sale looks identical to a metric explosion to a naive rule), so baselines need periodic recalibration. Lineage graph storage grows with event volume and retention; most teams sample or aggregate older lineage data rather than keeping full per-event graphs forever. Finally, a common wrong turn is over-fitting alert rules to the last incident (a rule that only catches exactly what happened last time) instead of generalizing to "expected event-to-event ratio and lineage completion for any business flow," which is the pattern that transfers to the next, different incident.

flowchart LR
    P[Producers] --> B[(Event Broker)]
    B --> C[Consumers]
    B --> ST[Synthetic Transaction Probe]
    C --> M[Business Metric Aggregator]
    M --> DQ[Data-Quality Checks]
    B --> SC[Schema Contract Validator]
    C --> LT[Lineage Tracer]
    DQ --> AL[Alerting Layer]
    ST --> AL
    SC --> AL
    LT --> AL
    AL --> ON[On-call Runbook]

Direct answer

An ADR (architecture decision record: a short document capturing one decision, the options and the consequences) for decomposing a monolith must record why now, at least three real alternatives (including "modularise but stay deployed as one" and "do nothing"), and an honest account of what happens to database transactions once data is split. To keep it from reading as a design essay, give it question headings, put the decision in the first lines, cap each section, and link out to the design document for everything else.

Question-per-heading structure

Each heading is the question the reader has, and each answer is 2-5 lines:

# ADR-19: Extract order, payment and inventory from the shop monolith
Status: Accepted | Date: 2026-09-28

## What is the decision?
Extract Payment first, then Inventory, then Orders, using the strangler
pattern (route traffic to the new service one endpoint at a time while the
monolith keeps serving the rest). The catalog and accounts stay in the monolith.

## Why now?
Deploys are weekly because every change ships the whole app; checkout
latency at peak is limited by contention on shared tables (requests queueing for locks on the same tables).

## What did we consider?
1. Do nothing, tune queries and scale the monolith vertically (buy a bigger machine).
2. Modular monolith: enforce module boundaries inside one deployable (one program, one deploy, with strict internal borders).
3. Extract services along business seams, gradually (chosen).
4. Full rewrite (rejected: no delivery for a long stretch, high risk).

## What happens to transactions?
The single database transaction across stock, payment and order disappears.
We replace it with a saga (local steps plus undo steps), an outbox for
reliable events and idempotent handlers. For a few seconds an order may
exist before stock is confirmed; Support owns cleanup of stuck orders.
Saga, outbox and idempotency mechanics are in the linked design doc (explained in plain words after this excerpt).

## What does it cost us?
Distributed debugging, more infrastructure, data duplication, a period with
two ways of doing things. Undo plan: each extraction is behind a routing
flag (a switch that sends traffic to the old or new code path); the monolith code path stays until two release cycles pass cleanly.

## Where is the detail?
Link to the design doc and the migration plan (not copied here).

Handling transactional boundaries in the consequences

Inside the monolith, "reserve stock, charge card, create order" is one database transaction (ACID: atomic, consistent, isolated, durable, so it either all happens or none of it does). After the split there are three databases, so that guarantee is gone. Record the replacement explicitly:

• Keep together what must be atomic. If two tables truly need one transaction, they belong in one service.
• Saga (a sequence of local transactions where each step has an undo, called a compensating action): reserve stock, then charge, then create order; if charge fails, release the stock.
• Outbox pattern: write the business change and the message to publish in the same local transaction, then a relay publishes it, so an event is never lost or sent for a rolled-back change.
• Idempotent handlers (safe to process the same message twice), because messaging delivers at least once (a message may arrive twice but is never dropped).
• Accepted anomalies: for a few seconds an order may exist before stock is confirmed. State it, along with who handles the cleanup.

Worked example with data: order 1001 buys 2 units of SKU-A for $40. Step 1, Inventory reserves stock (10 units become 8): succeeds. Step 2, Payment charges $40: succeeds. Step 3, Orders inserts the order row: fails (say a database constraint error). That failed insert rolled back its own local transaction, so anything written in that same transaction, including an outbox row, is gone too. The Orders service therefore catches the error and, in a fresh local transaction, writes an OrderFailed(1001) row to its outbox; the relay publishes it. The saga reacts by running the compensations in reverse: refund the $40 (Payment) and release the 2 units (stock 8 back to 10). If the relay delivers OrderFailed twice, the idempotent handlers make the second refund and release do nothing.

Keeping it off the design-essay path

• Decision in the first two lines; status and date at the top.
• Options as a short list, one line of reasoning each, not paragraphs.
• No diagrams of every service or full schema. Link to them.
• One decision per ADR. The extraction order, the messaging choice and the data-ownership rules are separate ADRs that reference each other.
• Word budget: if it scrolls past two screens, move the detail out.

Trade-offs and pitfalls

• Splitting a database is the hard part; services that still share tables gain the costs of distribution and none of the independence.
• Recording only the winning option makes the ADR useless when someone asks "did we consider a modular monolith?"
• Extracting too many services at once. Sequencing belongs in the record, with a review trigger such as "reassess after Payment is live for one quarter".

Direct answer

Infrastructure as Code means defining servers, networks, and other infrastructure in version-controlled configuration files instead of clicking through a cloud console or running one-off commands, so provisioning becomes repeatable, reviewable, and automatable. It matters because manual changes drift out of sync with what anyone remembers doing, are hard to reproduce for a second environment, and leave no audit trail of who changed what.

Why it matters day to day

• Consistency: the same module produces the same VPC, subnet layout, and security group rules every time, in every environment.
• Recoverability: rebuilding a region or an environment from scratch is a terraform apply, not a multi-day scramble reconstructing console clicks from memory.
• Review and audit: infrastructure changes go through the same pull-request review as application code, with a diff showing exactly what will change before it happens.
• Faster iteration: spinning up a throwaway staging environment is minutes of automation instead of a manual request queued behind other tickets.

Tools and typical use cases

• Terraform: declarative, multi-cloud provisioning. Typical use: standing up a VPC, subnets, and a fleet of compute instances across AWS and GCP from one codebase.
• Ansible: agentless configuration management. Typical use: installing packages, applying OS patches, and pushing config file changes to a fleet of already-provisioned servers.
• Puppet: agent-based configuration management with continuous enforcement. Typical use: keeping long-running servers (users, cron jobs, installed services) in a defined state across hundreds of nodes over time, correcting drift automatically on each agent run.

Trade-offs & pitfalls

• Provisioning tools (Terraform) and configuration tools (Ansible, Puppet) solve different problems. Using one to do the other's job, such as Terraform provisioners running shell scripts to configure software, usually loses the idempotency and drift-correction the dedicated tool gives for free.
• IaC only prevents drift for changes that go through it; a console click that bypasses the pipeline still causes drift, so IaC needs to be paired with either locked-down console permissions or a periodic drift-detection scan to stay true.
• Writing the first Terraform module for a legacy environment often means importing existing resources into state (terraform import) before any of these benefits apply, which is real, sometimes tedious, up-front work.