Airbnb Solutions Architect Interview Preparation Guide - Mid Level

Approach: model the evaluation as a lightweight project with clear artifacts (vendor, PoC, compliance, approvals). Configure either Jira or Azure DevOps with consistent issue types, custom fields, workflows, Confluence links, and two dashboards (engineering/executive).

Issue types (Jira) / Work item types (Azure DevOps)

• Vendor (Epic / Feature): owner, vendor scorecard link
• Evaluation Task (Story / User Story): specific evaluation activities (install, integration, benchmarks)
• PoC (Task / Task): environment, success criteria, test plan
• Compliance Check (Sub-task / Test Case or Checklist Work Item): controls mapped to requirements
• Approval (Issue / Approval work item or Pull Request-like flow): approver, decision, SLA
• Risk/Blocker (Bug/Impediment)

Suggested fields

• Vendor Name, Product Version, License Model
• Priority, ROI Estimate, TCO Estimate
• Success Criteria (acceptance), Test Plan link
• Security/Compliance Status (Compliant/Conditional/Non-compliant)
• Evaluation Score (numeric), Last Updated, Estimated Effort
• Owner, Approver, Target Decision Date

Workflows

• Vendor: Candidate → Under Evaluation → PoC Running → In Review → Approved / Rejected
• PoC: To Do → In Progress → Results Submitted → Validation → Closed
• Compliance: Not Started → Assessing → Mitigation Planned → Approved
• Approval: Requested → Under Review → Approved / Rejected (with automated notifications)

Linkages & automation

• Link Evaluation Tasks and PoC tasks to the Vendor Epic/Feature; tie Compliance Checks as sub-tasks or linked test cases
• Automate transitions: when all PoC tasks Done and Compliance Approved, auto-transition Vendor to In Review and create Approval work item
• Add PR-like approvals or use Azure DevOps Approvals & Gates for gating production sign-off

Confluence integration

• Maintain a vendor evaluation template page (requirements, test plan, scorecard). Link Confluence page to the Vendor issue/feature.
• Embed test results, architecture diagrams, and runbooks in Confluence; surface links in issue fields.
• Use Jira macros or Azure DevOps Wiki links for one-click context.

Dashboards / Views

• Engineering dashboard:

  • Widgets: active PoCs, open evaluation tasks, test failures, compliance gaps, build/deploy pipeline status
  • Filters: by environment, owner, priority, SLA
  • Purpose: task-level progress, blockers, test evidence

• Executive dashboard:

  • Widgets: vendor comparison table (scorecard), decision timeline, cost/TCO summary, risk heatmap, approvals pending
  • KPIs: % PoCs completed, average evaluation time, projected TCO, # of compliant vendors
  • Purpose: concise decision support; link to Confluence deep-dive pages

Best practices

• Standardize scorecard and success criteria so dashboards can roll up quantitative comparisons.
• Use automation rules (Jira Automation or Azure Pipelines/Service Hooks) to reduce manual status updates and send reminders for approvals.
• Keep audit trail: require attachments for PoC evidence and compliance artifacts; enable versioned Confluence pages.

This configuration ensures traceability from initial vendor candidate to final approval, gives engineers the granular task view they need, and provides executives a concise, data-driven decision snapshot.

Direct answer

A production database monitoring dashboard needs metrics from four categories, resource utilization, replication, query performance, and capacity, because a healthy reading in one category can hide a real problem in another. Nine metrics cover the ground well: CPU utilization, buffer cache hit ratio, disk input/output operations per second (IOPS) utilization, disk space used, active connections against the configured maximum, replication lag, replica connectivity state, the 95th and 99th percentile (p95 and p99) query latency, and lock wait time or blocked query count.

How to think about it

This list holds whether the instance is standalone or serving a mixed workload of online transaction processing (OLTP, many small read and write transactions) and online analytical processing (OLAP, fewer, larger queries that scan and aggregate data); a mixed workload just means watching the query-performance metrics separately per workload type, since an analytical query's normal duration and a transactional query's normal duration are entirely different baselines, and one threshold cannot serve both honestly.

Worked example

Real output pulled from a running instance, illustrating what these look like in practice:

-- buffer cache hit ratio
 cache_hit_pct | blks_hit | blks_read
         99.95 |  1562989 |       718

-- per-table scan pattern (a missing-index signal when seq_scan dominates)
      relname       | seq_scan | idx_scan
 order_items_denorm  |        1 |        1
 orders               |        5 |        0

And the replication query returning correctly empty on an instance with no replica attached:

SELECT application_name, state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes
FROM pg_stat_replication;
-- (0 rows)

Zero rows here is the correct, real answer for this specific instance, not a broken query, a genuine replication setup returns the lag columns shown in a read-replica health check instead.

Trade-offs and pitfalls

Alerting on every metric at the same severity causes alert fatigue, and real pages get ignored along with the noise. Thresholds copied from someone else's blog post instead of your own baseline will page every day if your normal steady state already runs at that level. Missing the replica connectivity state specifically, because a lag graph can look fine while actually frozen at its last reading. Monitoring only resource metrics or only query-performance metrics, never both, hides real incidents either way: high CPU with no p99 alarm can be an entirely healthy scheduled batch job, and a p99 spike with normal CPU is often a lock or an external dependency, not the database's own fault at all. Connections being rejected outright rather than degrading gracefully once the cap is hit is exactly why that threshold should warn well before 100%, not at it.

Direct answer
An API gateway centralizes the cross-cutting concerns that every client-facing request needs handled the same way, so individual services don't each have to reimplement them: routing requests to the right backend service, authenticating the caller, enforcing rate limits, terminating TLS, and doing basic request/response shaping, all in one place at the edge of the system.

Structured elaboration
Routing means the gateway is the single entry point clients call, and it decides which backend service actually handles a given request based on the path or other request attributes, so clients don't need to know the internal service topology or how it changes over time. Authentication centralizes verifying who's calling (validating a token or API key) once, at the edge, rather than every individual service independently re-implementing token validation. Rate limiting enforces per-client or per-API-key usage limits at the edge before a request even reaches a backend service, protecting every service behind the gateway from a single abusive or misbehaving caller without each service needing its own limiting logic. TLS termination handles the encryption/decryption overhead for external traffic in one place rather than in every service. Request/response transformation covers lighter-weight shaping, like aggregating a few backend calls into one client-facing response, or adapting an older API shape for backward compatibility, without every service needing to handle those concerns itself.

Worked example
Without a gateway, if a client needed to call five different backend services directly, each of those five services would need its own authentication logic, its own rate limiting, its own TLS termination, and the client would need to know all five service addresses and how they change; a gateway collapses that to one well-known entry point, one place to update authentication logic when it needs to change, and one place to see aggregate traffic and enforce limits across the whole surface a client touches.

Trade-offs and pitfalls
The most common mistake is treating the gateway purely as a routing proxy and skipping the cross-cutting concerns it's actually valuable for (authentication, rate limiting), which leaves each backend service to reimplement them inconsistently. The opposite mistake is overloading the gateway with business logic that belongs in a backend service (like assembling a complex, service-specific business response rather than a light aggregation), which turns the gateway itself into an operationally risky single point of coupling for logic that should be independently deployable elsewhere.

Direct answer

I would first decide whether the deal justifies trying at all, then compress by cutting scope rather than adding people or demanding overtime, and negotiate with the customer a written, pilot-sized acceptance for day 28 with a dated plan for the rest. A pilot is a limited first release used for real work on a small scale, to prove the value before the full build. Squeezing 12 weeks of work into 4 needs three times the throughput (12 / 4 = 3), which people and overtime do not honestly deliver. What can change is what we promise to have by day 28.

Structured elaboration

Step 1: decide whether to compress. Ask:

1. Is the four-week condition firm, or an opening position? Ask Sales to find out what the customer is really trying to do by then (a board demo, a budget cycle, a competing vendor).
2. What is the deal worth, and what does the team stop doing for four weeks to make room?
3. Is the capability reusable by other customers, or a one-off that becomes a support burden?
   If the deal is real, the value is high, and the capability is reusable, I try. If it is a one-off for a modest deal, I decline the date and offer a later one.

Step 2 (worked example): find the slice that fits four weeks. Break the 12 weeks into components, then ask what the customer must see on day 28.

Every row's slice plus deferred equals its full size, and the columns total 12, 4 and 8.

Why those amounts: the customer's day-28 success is their own data flowing through their own rules, so data import gets 2 of its 3 team-weeks (their real file formats, not every format), configuration gets 1 of 3 (their first two rules, not a general rules engine), and reporting gets 0.5 (the one monthly report they named). Single sign-on and audit log are not needed to see the workflow work, so they get 0. Hardening gets 0.5 so the pilot does not fall over on their first real file.

Step 3: why not just add people or work longer. New engineers take weeks to ramp up and slow the existing team while they do, so a late addition often delays a short project. Overtime for four weeks trades quality for speed, and the defects land in front of the customer we are trying to impress.

Step 4: negotiate with the customer. Ask what must be true on day 28 for them to call this a success; usually it is their own workflow working on their own data, not every feature. Propose the slice as a pilot, and agree in writing:

• Accepted when: their sample files import, their first two rules run, and the monthly report comes out correct, with no critical defects open.
• Explicitly not in the pilot: single sign-on, audit log, full-scale load.
• Their dependencies: sample data and test users by day 5; slip there moves the date.
• The rest: dated, with a firm schedule re-baselined (re-estimated and re-agreed) at the end of week 4 once real use shows the true effort.

What I would say to the customer: "We can have your own data and your first two rules working, with the monthly report, on day 28. Single sign-on and the audit log come in the next phase, and I will give you a firm date for them at the end of week 4, once we have seen your real files. What would you need to see on day 28 to call it a success?"

Step 5: internal discipline. Sales agrees not to promise beyond the written scope. I re-estimate at the end of week 1; if even the slice looks like it will miss, the customer hears in week 2, not week 4.

What would change my call

If the customer's security team will not accept a pilot without single sign-on, the slice grows by 2 weeks and no longer fits, so I would say so and propose a later date or a different pilot setup. If the displaced work is a committed reliability fix, I would decline.

Trade-offs and pitfalls

• A pilot promised as "basically the full thing" is how a four-week deal becomes a twelve-week argument.
• Do not hide the cut scope in the contract; surface it in the first conversation.
• Track what the team stopped doing, and tell its owners.

Direct answer

Keep a single, easy-to-find decision log tied directly to the work it affects: what was decided, the options considered, the reasoning, and who owns it, updated by whoever is making the decision at the moment it is made, not reconstructed later from memory.

Structured elaboration

What belongs in an entry

A short, consistent structure works better than a long one, because people will actually fill it out: a title, the date, who owns it, the context in one or two sentences, the options considered with their trade-offs, the decision itself, and the reasoning behind it in a few bullet points.

Where it lives

The log needs to be one discoverable place, linked from the tickets, docs, or roadmap items it affects, not scattered across meeting notes and chat threads. A shared doc or wiki page with a simple table works; the tool matters less than the discipline of always linking to it.

Who keeps it current

The person who owns the decision, not a rotating scribe with no stake in it, writes or finalizes the entry, ideally right after the decision is made, while the reasoning is still fresh and easy to state accurately.

How it gets used afterward

In retrospectives, revisit decisions that affected the outcome and check whether the original assumptions held. For onboarding, a short list of the most consequential recent decisions gives a new team member the context that would otherwise take weeks of osmosis to pick up.

Worked example

A team is deciding between two ways to notify users of an event: a push notification versus an in-app banner. The entry, once decided, looks like this: title, "Notification channel for event alerts"; date and owner, the decision owner's name and the date; context, users were missing time-sensitive alerts under the current in-app-only approach; options considered, push notification (faster delivery, requires a new permission prompt), in-app banner only (no new permission needed, slower to be seen), and both channels (best coverage, more engineering and support surface); decision, push notification with an in-app banner as a fallback for users who decline the permission; reasoning, the delay in the in-app-only approach was the specific problem being solved, and the fallback covers users who opt out.

Anyone who later asks why the team does not just use an in-app banner, since it is simpler, can read this entry and see the trade-off was already considered, rather than re-litigating it from scratch.

Trade-offs and pitfalls

A log nobody updates is worse than no log: it creates false confidence that the reasoning is captured somewhere, while actually going stale. The fix is keeping entries short enough that updating one takes minutes, rather than requiring a formal write-up every time.

A log can also be used as a weapon later, such as insisting a past decision still holds in a situation where circumstances genuinely changed and revisiting was the right call. The log should record reasoning, not lock in a decision forever; a review date or a note on when to re-evaluate keeps it a living reference instead of a trap.

Direct answer

For a payment pipeline, at-most-once risks silently losing a transfer, at-least-once risks silently double-charging unless the application deduplicates, and exactly-once removes both risks but shifts real coordination cost onto the system: distributed state, transactional writes, and monitoring that has to prove correctness, not just uptime. As a site reliability engineer (SRE) evaluating this, the practical recommendation is usually to skip chasing full exactly-once delivery and instead build strong idempotency on top of at-least-once delivery, which gets the same business outcome with a materially smaller operational surface to run and monitor.

Structured elaboration

The three semantics for a payment pipeline specifically

• At-most-once: fastest, simplest, but a dropped message means a transfer that should have happened never did, silently, with no automatic signal that anything is wrong.
• At-least-once: no transfer is silently lost, but a redelivered message can double-apply unless the consumer deduplicates; the operational cost moves from "did we lose something" to "did we correctly deduplicate everything."
• Exactly-once: neither loss nor duplication, but achieving it end to end (through the broker, the application, and often an external payment gateway that the team does not control) is the hardest of the three to actually guarantee, not just to claim.

Operational complexity and monitoring needed to support exactly-once, if chosen

• Cross-system transactionality: the broker, the application's database, and often a third-party payment gateway all need to agree on outcome, which typically means either distributed transactions/two-phase commit, or an idempotency-key-based reconciliation across all three, since a payment gateway rarely offers you a transactional hook into its own state.
• Distributed tracing correlating message identifiers, idempotency keys, and the payment provider's own reference identifiers end to end, so a specific transfer can be followed through every hop when something looks wrong.
• Dashboards and alerts specifically for correctness, not just availability: duplicate-detection rate (how often the dedup layer actually catches a redelivery), transactional commit latency and abort rate, and a rising rate of "uncertain outcome" transactions (calls to the payment gateway that timed out with no confirmed result either way) as a leading indicator of a coordination problem.
• Stateful capacity planning: a dedup or transaction-coordination store adds its own storage growth, garbage collection, and time-to-live (TTL) tuning that has to be sized and monitored like any other stateful service, not treated as an afterthought bolted onto a stateless consumer.
• Runbooks and on-call practice oriented around correctness incidents (a transaction stuck in an uncertain state, a suspected duplicate charge) rather than only the usual latency/error-rate incident playbooks, since a payment-correctness incident often has no clear "is it still broken" signal from standard availability monitoring alone.

An alternative design that achieves idempotency without full exactly-once support

• Assign a globally unique payment identifier per logical payment at the point the payment is initiated (not generated fresh on each delivery attempt), and treat every downstream write as an idempotent operation keyed by that identifier: INSERT the payment row if payment_id does not already exist, otherwise return the existing recorded outcome unchanged, rather than an unconditional apply this payment.
• This lets the transport stay ordinary at-least-once (retries, redelivery, no special broker transaction support required); the payment_id-keyed idempotent write is what actually prevents a double-charge, regardless of how many times the message is redelivered.
• Use short-lived optimistic-concurrency checks (a version number on the payment row) for any state transition that must happen exactly once even under concurrent redelivery, for example moving a payment from pending to settled.
• Operationally this is materially simpler to monitor than full exactly-once: the key health signal becomes "reads and writes against the payments table by payment_id," a much smaller, more familiar surface than cross-system distributed transaction coordination.

Worked example

A payment consumer processes transfer messages, each carrying payment_id, amount, and destination_account.

• A message for payment_id=pay_9f31 and amount=$500 is delivered, processed, and the payments table gets a row {payment_id: pay_9f31, amount: 500, status: settled} via an idempotent upsert: INSERT ... ON CONFLICT (payment_id) DO NOTHING, then a status check confirms the row already reflects settled.
• The broker redelivers the same message (a normal at-least-once redelivery after an acknowledgment was lost in transit, not a system failure). The consumer runs the identical idempotent upsert again: the ON CONFLICT clause means no second row is inserted and no second $500 debit happens; the consumer reads back the existing settled row and returns the same success response it would have on the first delivery, with zero difference in outcome to the caller.
• Total effect after both deliveries: exactly one $500 transfer applied, verified by the payments table containing exactly one row for pay_9f31, achieved without any distributed transaction spanning the broker and the payments database.

Trade-offs and pitfalls

• Common wrong turn: pursuing full exactly-once delivery as a goal in itself, before checking whether an idempotency-key design achieves the same business correctness with far less operational surface to build and monitor.
• Common wrong turn: monitoring a payment pipeline only for availability and latency, without a dedicated signal for correctness (duplicate rate, uncertain-outcome rate), which means a duplicate-charge incident can go undetected until a customer or an auditor finds it.
• Common wrong turn: generating the idempotency key at delivery time (for example, from the message's own broker-assigned offset) instead of at payment-initiation time; a key tied to delivery mechanics changes on redelivery through a different path and stops actually deduplicating the thing that matters, the logical payment.
• Senior signal: recommending the idempotency-key design as the default and reserving full exactly-once machinery for the rare case where it is genuinely required, while being explicit about what monitoring exactly-once, if chosen, would actually demand.

The four standard DR tiers form a spectrum from cheapest-and-slowest to most-expensive-and-fastest, and each one trades infrastructure spend for recovery speed (RTO, recovery time objective: how long restoring service takes) and data freshness (RPO, recovery point objective: how much data, measured in time, you could lose): backup-and-restore keeps only backups running, pilot light keeps a minimal always-on core, warm standby keeps a scaled-down full copy running, and active-active multi-site keeps a full copy running and serving live traffic.

Comparing the four tiers

The RTO/RPO ranges above are the typical shape of the trade-off, not a fixed number for any specific system: the exact figures depend on data volume, automation maturity, and how the replication is actually implemented within each tier.

Worked example: a budget-constrained startup

A mid-sized SaaS with a fixed infrastructure budget doesn't have to pick one tier for the whole system; the standard move is to mix tiers by criticality. Say the product has three logical components: authentication/billing (must never meaningfully go down, since it blocks every paying customer from doing anything), the core application (needs to come back reasonably fast but a short outage is tolerable), and internal admin tooling (only the ops team notices if it's down for a few hours).

A budget-conscious allocation: active-active for auth/billing (the one component where the cost premium is justified because its outage blocks revenue entirely, and it's usually small enough in infrastructure footprint that duplicating it fully is affordable), pilot light for the core application (keep the database replicated continuously so RPO stays low, but only spin up the app-server fleet in DR when actually needed, since that's the majority of the compute cost), and backup-and-restore for admin tooling (cheapest tier, acceptable because nobody customer-facing is blocked by it being down for hours). This gets the highest-blast-radius component the fastest recovery while keeping the overall DR bill proportional to what each component actually costs the business if it's down, instead of buying active-active everywhere by default.

Trade-offs and pitfalls

The most expensive mistake in this space isn't picking the "wrong" tier, it's picking a tier and never testing failover into it: a pilot-light setup that's never actually been promoted to full capacity under load is a theoretical RTO, not a real one, and the first real DR event is a bad time to discover the app layer doesn't actually scale up cleanly from zero. A related pitfall is under-provisioning a warm standby's capacity: "scaled down" often means it can absorb DR traffic at reduced performance, and teams sometimes forget to validate that the scaled-down size can actually handle 100% of production load once promoted, not just serve health checks. Finally, active-active's real cost isn't just the duplicate infrastructure line item, it's the ongoing engineering cost of keeping a multi-writer data model correct, which is easy to underestimate when comparing tiers purely on an RTO/RPO/dollar table.

Direct answer

A WebSocket is pinned to one server for its whole life, so "stateless WebSocket servers" really means keep only the socket on the gateway, and keep everything else (who is connected where, subscriptions, message history) outside it. For large-scale operations I recommend a combination: gateways that hold connections but no durable state, a routing registry (user to gateway, in an external store such as Redis, an in-memory key-value database fast enough for a lookup on every message) and a pub/sub layer that addresses gateways, not users, with resumable sessions so that a gateway failure costs a reconnect and nothing else. Live connection handoff is useful only on the same host during deploys; across machines, a client reconnect with a resume token is the practical form of handoff.

The four approaches

• (1) Sticky load balancer with state local to the server. The load balancer keeps sending a user to the same server (by cookie or IP hash, routing a client to a server chosen by hashing its IP address, so the same client lands on the same server every time). Everything about the session (subscriptions, unacknowledged messages) lives in that server's memory. To reach a user connected elsewhere, servers must talk to each other directly or broadcast.
• (2) Centralized session store. A store such as Redis holds user -> {gateway_id, connection_id, subscriptions}. Any service that wants to reach a user looks up the gateway, then calls or messages that gateway.
• (3) Stateless servers with tokens plus external pub/sub. Identity comes from a signed token presented at connect, so the gateway needs no session lookup to authenticate. Messages are routed through a pub/sub (publish/subscribe) system, where senders publish to named channels and every subscriber of a channel gets a copy (Redis Pub/Sub, NATS, or similar brokers); gateways subscribe on behalf of their connections.
• (4) Connection handoff across frontends. Moving a live connection from one frontend to another without the client noticing. On the same host this is real and routine: a new process receives the open socket's file descriptor (the small integer handle the operating system uses internally to refer to that open connection) from the old process over a Unix domain socket (a communication channel for processes on the same machine, faster than a network socket because it never has to leave the kernel), using SCM_RIGHTS file-descriptor passing (the specific Linux message type used to hand an open file descriptor from one process to another), which enables zero-downtime restarts. Across hosts, live TCP migration requires kernel-level connection repair (rewriting the TCP connection's state inside a different machine's operating system so packets keep arriving there, which most kernels and networks do not support) and is rarely worth it; the practical cross-host version is a server-initiated reconnect with a resume token.

Comparison

Recommended design for large scale

flowchart LR
  C[Clients] --> LB[TCP load balancer]
  LB --> G1[Gateway A]
  LB --> G2[Gateway B]
  G1 --> R[(Routing registry)]
  G2 --> R
  SVC[Chat or notify service] --> R
  SVC --> PS[Pub/sub: one inbox per gateway]
  PS --> G1
  PS --> G2
  SVC --> DB[(Message store with sequence numbers)]

1. Gateways hold sockets only. On connect: verify the signed token, write user -> gateway_id to the registry with a TTL, and subscribe the gateway (once) to its own inbox channel gw:<id>.
2. Route by gateway, not by user. A sender's service looks up the recipient's gateway and publishes to gw:<id>. The broker then carries a few hundred gateway channels, not millions of user channels, which keeps subscription churn off the broker when users reconnect.
3. Durability lives in the message store, not the pipe. Every message gets a per-conversation sequence number in a durable store before delivery. The pub/sub hop can lose a message; the client notices the sequence gap (or reconnects) and fetches from the store. That makes delivery reliable even though pub/sub is at-most-once (each message delivered zero or one time).
4. Gateway liveness. Each gateway renews a lease (a key that expires automatically unless its owner renews it) every few seconds. If it dies, its lease expires and a sweeper (a background job that scans for and removes expired entries) removes its registry entries, so senders stop publishing into a dead inbox.
5. Deploys use same-host handoff where possible, and drain-with-resume otherwise.

The load balancer is still "sticky" in the only sense that matters (a socket stays on its gateway), but nothing is lost when that gateway goes away. This is approach 3 for identity and message transport, approach 2 for routing, and approach 4 in its same-host form.

Worked capacity example

Assume 10 million concurrent connections and 50,000 connections per gateway (an ESTIMATE to confirm by load test).

• Gateways: 10,000,000 / 50,000 = 200 gateways at full load, plus headroom for failures and deploys.
• Registry memory: about 100 bytes per entry (user ID, gateway ID, timestamps) × 10,000,000 = about 1 GB, small enough for a modest sharded Redis cluster with replicas (extra copies of each shard kept in sync, so one node failing does not lose that data).
• Failure blast radius: one gateway crash disconnects 50,000 clients. If they reconnect with jitter spread over 30 s, the rest of the fleet must accept 50,000 / 30 ≈ 1,667 new connections per second, each needing a TLS handshake, a token check and a registry write. Size for this, and it is why jitter is non-negotiable.
• Broker load: with per-gateway inboxes, the broker holds about 200 subscriptions instead of 10 million.

Trade-offs and pitfalls

• Option 1 looks cheap until a hot server (a celebrity's followers all hashed to one box) cannot be rebalanced without disconnecting users.
• Per-user pub/sub topics create millions of subscribe/unsubscribe operations per reconnect surge and are a common broker-overload cause.
• Treating the registry as truth without TTLs produces "ghost" entries pointing at dead gateways and silently dropped messages.
• Believing pub/sub is durable. Redis Pub/Sub drops messages for disconnected subscribers; reliability must come from sequence numbers plus a durable store.
• Over-investing in cross-host live handoff. Clients must handle reconnects anyway (networks drop), so make reconnect cheap rather than trying to make it never happen.

With six regions and a three-minute decision target, the design needs three layers working together: independent, corroborated health signals feeding a central aggregator; a quorum vote across regions (agreement from a required minimum number of them, not any single region acting alone) rather than any single region's self-report deciding anything; and a human-approval gate sized to how reversible each step is, so the fast reversible steps happen immediately and only the consequential ones wait on a person.

Health checks. Each of the six regions runs several independent probes locally, external synthetic transaction success, internal error rate, and replication lag, reported to a central aggregator. Crucially, each of the other five regions also independently probes its peers, so a single region's self-report is never the only signal, since a region that's specifically partitioned from the aggregator might self-report healthy while genuinely unreachable to users.

Aggregation and voting. Failing over a given region requires agreement from a quorum of the other five regions' independent observations, for example at least four of five must independently confirm the target is unhealthy, not the target's own report and not a single observer alone. This corroboration requirement is what keeps the false-positive rate low: it takes a broad, independently agreeing signal, not one flaky probe, to trigger anything. It also directly addresses a subtler failure: a network problem between just two of the six regions can look like "region X is down" to one observer when the actual fault is the path between two specific regions, not region X's health, which is exactly why requiring multiple independent observers to agree matters here.

Count the observers before you fix the threshold. Four of five is a supermajority, not a majority, and that choice decides how the system behaves in exactly the incident it was built for, so enumerate it rather than asserting it. One region down leaves five peers able to report, so four of five has a spare vote and tolerates one broken probe. Two regions down simultaneously leaves only four peers able to report about either of them, so the threshold is met only if all four agree, and a single flaky probe now blocks the failover outright. Three regions down leaves three peers, and four of five becomes arithmetically unreachable: the automation declines to act during the largest outage it will ever see, which is the precise inverse of what it was bought for. Decide explicitly whether a peer that is itself unreachable counts as a vote against, which is fail-closed and produces the behaviour above, or drops out of the denominator, which is fail-open and keeps the rule satisfiable at the cost of the corroboration you raised the threshold for. A defensible middle is a majority of reachable peers with a floor under it, for example at least three peers must have actually reported and a majority of those must agree: that stays satisfiable down to three surviving observers while still refusing to act on one observer's word. Whatever you pick, write down the number of simultaneous regional failures the rule is designed to survive, because with six regions that is a design decision and not something that falls out of the word quorum.

Human-in-the-loop gating. A full human approval gate before any action at all would blow the three-minute target, so gate by blast radius instead. The moment quorum agrees, automatically execute the low-risk, fully reversible first step (withdraw new traffic routing from the suspect region) immediately. Reserve human approval for the high-risk, harder-to-reverse step (promoting a new primary and accepting divergent writes): page on-call automatically with full context and a short approval window, for example two minutes, after which a pre-approved default action fires anyway rather than stalling indefinitely. This two-tier structure meets the speed target for the safe mitigation while still keeping a person in the loop for the consequential decision.

Automation steps. Quorum confirms an unhealthy region, traffic routing is withdrawn immediately, on-call is paged with a short approval window, the designated secondary is promoted and fenced on approval or timeout-default, and reconciliation begins for whatever writes were in flight during the transition.

Conflict resolution during failover. Writes accepted by the failing region in the seconds before detection completes are expected, not an edge case, given real signal and quorum latency. Route them through the same tiered policy used for reconciliation generally: automatic merging for fields with a safe combination rule, and a human-review queue for anything financial or ambiguous, rather than assuming failover happens at a perfectly clean instant with zero writes in flight.

Schema-compatibility check before promotion. Verify the candidate secondary is running an application and schema version compatible with what the primary was serving before promoting it; a secondary a few deploys behind could reject or misinterpret a write shape the primary was already accepting, turning a regional outage into a data-corruption incident. If a mismatch is found, the safe default is to block automated promotion and escalate to a human rather than proceeding anyway.

Rollback safety. Keep the original region's data intact and readable rather than wiping it, so failing back later is a routing change, not a rebuild. Log every decision in the sequence, which quorum votes fired, who approved or which timeout fired, and what the schema check returned, so a post-incident review can reconstruct exactly why the system acted as it did.

Direct answer: For complex JSON documents with nested objects, arrays, and domain-specific merge rules, the design attaches per-field (or per-path) merge metadata rather than one blanket strategy for the whole document, applies structural rules by field TYPE (scalars use LWW or a domain rule, arrays use set/sequence semantics depending on intent, nested objects recurse the same merge logic), embeds explicit domain rules where the natural type-based default is wrong (ratings take max, tags union), and falls back to surfacing an unresolvable conflict rather than guessing when no rule cleanly applies.

Structured elaboration

Per-path metadata. Rather than one merge function for the whole document, the design tracks, for each JSON path (e.g. $.ratings.overall, $.tags), what KIND of merge applies there: last-write-wins, numeric-max, set-union, or a custom domain function. This is necessary because a single document genuinely mixes fields with very different correct merge semantics, treating them uniformly (e.g. LWW for everything) would be simple but wrong for many fields.

Default rules by structural type. Scalars (numbers, strings, booleans) default to LWW unless a domain rule overrides it. Arrays default to set-union semantics if order doesn't matter for that field (deduplicating by value or by an element ID), or to a sequence-CRDT-style position-preserving merge if order genuinely matters (e.g. an ordered list of steps). Nested objects recurse: the SAME per-path merge logic applies to each nested field independently, so a deeply nested document is handled compositionally rather than needing a special case for depth.

Domain-specific rule examples. "Ratings take max": for a field like $.qualityScore where two concurrent updates might reflect two different reviewers' concurrent submissions, the domain decides the highest score should win rather than an arbitrary last-write, a business rule, not a generic data-structure default. "Tags union": for $.tags, the domain decides concurrent additions from different sources should ALL be kept (set-union) rather than one replacing the other, since tags are additive by nature.

Fallback strategies for impossible merges. Some conflicts have no automatic resolution: two concurrent edits to a deeply structural field (e.g. one edit reshapes $.address from a string to a nested object, while a concurrent edit modifies the OLD string-shaped value), these are genuinely irreconcilable by any generic rule and need to be surfaced explicitly (both candidate document versions, with metadata about the conflicting paths) for human or application-level resolution, rather than the merge engine guessing and potentially producing an internally inconsistent document.

Worked example. A product-review document: $.title (LWW), $.rating (max, per the domain rule that a higher rating from a concurrent update should win, reflecting the reviewer's most positive assessment), $.tags (set-union, so tags added from two different concurrent edit sessions all survive), and $.photos (an OR-Set of photo IDs, so concurrently-uploaded photos from two devices both appear rather than one overwriting the other). Two concurrent edits: edit A sets rating: 4, tags: ["great_service"]; edit B (concurrent, unaware of A) sets rating: 5, tags: ["fast_delivery"]. Merge result: rating: 5 (max rule), tags: ["great_service", "fast_delivery"] (union rule), no conflict surfaced, since every field had a clean, applicable rule.

Trade-offs and pitfalls. The biggest practical risk in this design is an UNDEFINED merge rule for a path silently defaulting to something wrong (e.g. LWW applied to a field that actually needed union semantics) rather than failing loudly during development; a strong version of this design validates, at schema-definition time, that every field in the document has an EXPLICIT merge rule assigned, rather than relying on an implicit fallback that might quietly produce a business-incorrect result in production.