DoorDash Solutions Architect (Junior Level) Interview Preparation Guide

Direct answer

A distributed monolith is a system that is split into separately deployed services but still behaves like one program: the services cannot be changed, tested or released independently. You pay every cost of a distributed system (network latency, partial failures, harder debugging) and get none of the main benefit, which is independent teams shipping independently. Two strong signs are lockstep deployments (a feature routinely requires several services to release together, in a specific order) and long synchronous call chains (one user request cannot complete unless many services are all up at the same moment). A concrete first mitigation is to make deploys independent with versioned, backward-compatible APIs checked by consumer-driven contract tests, so each service can ship without waiting for the others.

What it looks like

Think of an online store split into Cart, Pricing, Inventory, Orders and Payments. On paper these are five microservices. In practice:

• Adding a "gift wrap" option means changing all five, and the release plan is a spreadsheet saying "deploy Pricing first, then Orders, then Cart, within 30 minutes, or checkout breaks".
• Placing an order calls Cart → Pricing → Inventory → Orders → Payments, one after another, synchronously.
• Orders and Inventory read and write the same products table.

Sign 1: lockstep deployment

What you observe: releases are coordinated across services; there is a shared release train (a fixed, recurring release schedule that several services are bundled onto and ship together, rather than each shipping on its own whenever its change is ready) or a "deploy order" document; rolling back one service requires rolling back others.

Why it happens: services share a data model or API that changes without versioning. If Pricing renames a field in its response, Cart breaks until Cart is updated, so they must deploy together.

How to detect it: from the deploy log, compute the fraction of releases in which a service was deployed within the same short window as another service for the same ticket. If Cart ships with Pricing in most of its releases, those two are one deployable in disguise.

Sign 2: synchronous call chains that multiply failure

What you observe: a distributed trace (a record, stitched together from the timestamped spans each service emits for the same request, showing every service that request touched and how long each one took) of one request shows a deep chain of blocking calls, and an outage in any one service takes down the whole user flow.

Why it matters, with numbers: if a request needs 5 services in series and each is independently available 99.9% of the time, the chain is available only when all 5 are up:

• 0.999 × 0.999 × 0.999 × 0.999 × 0.999 = 0.999^5 ≈ 0.99501, about 99.50%.
• Over a 30-day month (30 × 24 × 60 = 43,200 minutes), one service at 99.9% is down about 0.001 × 43,200 = 43.2 minutes.
• The 5-service chain is down about (1 − 0.99501) × 43,200 ≈ 215.6 minutes, roughly five times as much.

A monolith would have had one component's worth of downtime; the distributed monolith has five, with the same features. (This simple model assumes failures are independent; correlated failures change the number but not the lesson.)

Other signs worth naming

• Services sharing a database schema, so a column change needs coordination.
• A shared library of domain objects that every service must upgrade together.
• End-to-end tests that need every service running before anyone can merge.

The mitigation: break deploy coupling first

Why start here: lockstep deploys are the most expensive symptom (they slow every team, every release) and fixing them is incremental. Steps:

1. Version the contract. API changes become additive: add a new field, never rename or remove one in place. Consumers use a tolerant reader (ignore fields you do not recognise, do not fail on missing optional ones).
2. Add consumer-driven contract tests. Each consumer (Cart) publishes the exact requests and response fields it depends on; the provider (Pricing) runs those contracts in its own CI. Tools such as Pact implement this. Now Pricing learns in its own build, before deploying, whether it would break Cart.
3. Deploy in expand-then-contract order. To rename price to unit_price: Pricing ships a version returning both; Cart switches to unit_price whenever it likes; once contracts show no consumer reads price, Pricing removes it. Three independent deploys, no coordination window.
4. Measure the result. Track the lockstep share from the deploy log. If Cart and Pricing co-deployed in 8 of 10 releases before, the target is near zero within a couple of quarters.

If the contracts reveal that two services cannot evolve separately no matter what (they change together on every feature), the honest fix is to merge them back into one service. A boundary that always moves together was never a real boundary.

Pitfalls

• Fixing the symptom with tooling. A better release-orchestration tool makes lockstep deploys easier to run, which hides the coupling instead of removing it.
• Adding more services. Splitting further to "fix" coupling usually adds more edges.
• Replacing sync calls with async messages carrying the same shared data model. The deploy coupling survives; it just moved into the message schema.

Direct answer

I gave myself a single weekend to build and deploy a small end-to-end project using a message-queueing system I'd only used at a surface level at work, with one rule I set in advance: it had to run somewhere real and handle actual (small) load, not just run on my laptop, because that's where the parts documentation skips over actually live.

Structured elaboration

The constraint I imposed on purpose was what forced real understanding instead of a demo: deploying it and pointing real traffic at it, rather than stopping once the happy path worked locally. Before I started, I set the success criterion explicitly, so the project could fail informatively rather than just fizzle out: I'd only count it as understood if I could kill a consumer process mid-message and correctly predict, in advance, whether that message would be reprocessed or silently lost.

What building surfaced that reading hadn't: an edge case in exactly when a message gets acknowledged relative to when processing finishes, which changes the answer to that mid-crash question and isn't obvious from a conceptual overview. I spent roughly a weekend plus a couple of follow-up evenings on it. What transferred back to my day job: a few months later I proposed a specific change to a retry policy on a production system, grounded directly in the acknowledgment-timing behavior I'd deliberately broken and observed in the side project, not in something I'd only read about.

Worked example

In a similar project on a different tool, I contributed a small fix to an open-source library I depended on, specifically to force myself to learn its internals rather than just use it. The maintainers' review comments were the actual learning mechanism there: they caught an assumption I'd made about thread-safety that I hadn't questioned, holding the change to a bar I hadn't set for myself. That's a distinct kind of learning project from the deploy-it-yourself one: someone else's quality bar does the falsifying for you, instead of a self-imposed test.

Trade-offs and pitfalls

The main risk with this kind of project is that it stays a toy: without a real constraint forcing depth (deploy it, break it on purpose, get it reviewed by someone with a real bar), it's easy to stop the moment the happy path works and call that learning. The other risk is over-scoping: a project sized to take "a couple of weekends" that drags on for months rarely produces anything that solidifies into something you'd actually reuse.

A CRDT-based counter (CRDT: Conflict-free Replicated Data Type, a data structure whose replicas can be updated independently and merged with a deterministic rule that always converges to the same value) lets every region increment locally with no coordination, so writes never block and never fail because of a remote outage. A consensus-backed counter (built on Raft or Paxos, where a majority of replicas must durably agree on each state transition before it counts as committed) gives you a single, linearizable value where every acknowledged read reflects every previously acknowledged write, at the cost of needing a live majority and at least one round trip per operation. The concrete thing the CRDT approach gives up is that read guarantee: a client can observe a stale, undercounted value during the window before a remote increment has propagated, with no signal that the value is incomplete, whereas consensus makes that undercounting structurally impossible.

What each approach actually guarantees

Worked example: where the freshness guarantee actually breaks

Take a G-Counter (a grow-only counter CRDT) tracking a global increment total across three regions, A, B, and C, each holding its own local counter; a read sums the counters a replica currently knows about.

1. Start: A = 0, B = 0, C = 0.
2. Region A processes one local increment: A = 1.
3. Region C processes two local increments: C = 2.
4. Before anti-entropy has propagated A's and C's updates to B, a client reads the counter at region B. B's local view is still A = 0, B = 0, C = 0, so the read returns 0, even though three increments are already durably accepted somewhere in the system.
5. An anti-entropy round runs: B receives the vectors {A: 1} and {C: 2}, merges by taking the elementwise maximum, and B's state becomes A = 1, B = 0, C = 2. A read at B now returns 3, the correct eventual total.

No increment was lost between steps 2 and 5, which is the CRDT's core promise. But the read at step 4 was not just a little stale, it was materially undercounted with no indication of that to the caller. A consensus-backed counter cannot produce this outcome, because an increment is not considered committed until a majority has durably logged it, and any successful read after that point is defined to include it.

A related but distinct choice: automatic CRDT merge vs. application-level conflict resolution

The same coordination-free idea shows up one level up in shared-document collaboration, where two users edit the same paragraph while offline. There a third option exists beyond CRDT and consensus: application-level conflict resolution, where conflicting edits are detected and handled by explicit business logic (show both versions to the user, prefer the longer edit, run a custom three-way merge) instead of a mathematically guaranteed merge function. A CRDT gives the same coordination-free availability as the counter case, with a merge that is provably correct for that specific data type; application-level resolution can encode arbitrary rules a generic CRDT cannot express, such as preferring the document owner's edit, but only for data types someone is willing to hand-write a merge function for, and it carries no formal convergence guarantee if that logic has a bug.

Trade-offs and pitfalls

This is not a universal ranking of CRDT versus consensus, it is a question of which correctness property a specific use case cannot live without. A monitoring signal or a rolling feature aggregate feeding a dashboard, or a non-blocking model feature, can absorb a bounded, self-correcting undercount, so a CRDT is the right default: no leader, no quorum loss during a regional outage, cheaper per write. A signal that gates an irreversible action, such as a billing or quota threshold, a fraud rule, or an inventory decrement, cannot absorb that undercount, because the moment a threshold decision fires is exactly the moment staleness turns from cosmetic delay into a correctness bug, so it belongs on a linearizable path even though every write costs more there. A common mistake is defaulting to CRDTs everywhere for their operational simplicity and only discovering the freshness gap when a threshold check fires on stale data; the fix is rarely to abandon the CRDT for everything, but to route the one decision that needs freshness through a linearizable read, or a periodic consensus-backed reconciliation snapshot, while leaving the bulk of the aggregation coordination-free.

Direct answer

Owning a migration that many people depend on is fundamentally a sequencing and risk-management problem, not a technology problem. You inventory what is actually affected before touching anything, prove the new platform can do what the old one does on a small slice before committing the whole system, cut over in a way you can reverse, and you do not call it done until you have actively verified nothing regressed, rather than waiting to see if anyone complains.

Structured elaboration

A migration plan for shared infrastructure has six parts:

1. Inventory. Enumerate every consumer, integration point, and edge-case usage of the current system, not just the documented ones. The single biggest risk in a migration like this is the undocumented dependency nobody remembers until it breaks.
2. Compatibility and parallel-run strategy. Before migrating anything real, prove the new platform matches the old one's behavior on a representative slice, ideally by running old and new side by side on the same input and diffing the output (comparing the two outputs field-by-field to spot any mismatch), so you find divergence before anyone actually depends on the new path.
3. Sequencing and cutover checklist. Migrate the lowest-risk, most self-contained consumer first, not the biggest or most visible one, to build confidence and catch problems cheaply. Each cutover step needs an explicit go or no-go checkpoint tied to a measurable signal, not a calendar date.
4. Rollback plan. For every cutover step, know in advance, before you need it, exactly how you would reverse it and how long that reversal takes. A rollback plan you have not thought through until the moment you need it does not really exist.
5. Communication plan. Dependent teams need to know what is changing, when, what (if anything) they need to do on their end, and how to reach you if something breaks. Give enough lead time that nobody is surprised, and confirm the message landed rather than just broadcasting it.
6. Parity verification. After cutover, actively check that outputs, behavior, and performance match the old system's, on real traffic where possible, rather than waiting for a complaint. Absence of complaints is not evidence of parity.

Worked example

Take migrating a shared internal library used by a dozen services to a new major version with breaking changes.

Inventory turned up 12 documented consumers plus two undocumented ones found by searching for actual usage, one of which called a function everyone assumed was already retired. The parallel-run phase put the new version behind a flag in the three lowest-risk consumers first, running both code paths against the same inputs for two weeks and comparing outputs before trusting the new path alone. The cutover checklist then moved the remaining nine consumers in two more batches, each batch gated on the prior batch showing no divergence for a full week, rather than moving everyone at once. The rollback plan kept the old version available at each step for a window matching that same soak period (the week-long stretch of watching a batch for divergence before trusting it and moving to the next one), so any batch could revert without a redeploy of every consumer. Communication gave each consuming team two weeks of notice, a migration guide, a named point of contact, and one specific action to take, a config flag to flip, and the plan tracked an explicit acknowledgment from each team lead rather than assuming an email had been read. After full cutover, parity verification compared error rate and latency for each consumer against its own pre-migration baseline for two more weeks before the project was closed, since a subtle regression is more likely to look like a slow drift than an outage.

Trade-offs and pitfalls

The most common failure is starting with the most important consumer to "prove it fast," which puts your first, least-proven attempt on the path with the largest blast radius (the widest set of dependent consumers or systems it could break if the attempt goes wrong). A second is treating communication as a one-time announcement instead of a confirmed, two-way acknowledgment, since silence is not the same as consent. A third is skipping parity verification because nothing broke immediately, when many real regressions show up as gradually rising latency or a rare edge case rather than a clean outage. And a rollback plan that exists only on paper, never rehearsed, is close to having no rollback plan at all, because its first real use will be under the worst possible time pressure.

Situation: A Director of Sales repeatedly promises features engineering can't deliver, creating friction, missed commitments, and risk to customer trust.

Coaching script (step-by-step, 1:1 and follow-ups):

1. Prepare: gather facts — specific promises, timelines, engineering capacity, customer impact.
2. First 1:1 (60 mins) — Empathize and align on shared goal.
   • "I value your drive to close deals. I want to ensure we deliver on those wins without risking churn. Can we walk through the specific commitments you made and the customer's must-haves?"
   • Listen, restate commitments, and surface technical constraints.
3. Diagnose: explain technical feasibility plainly — trade-offs, dependencies, delivery timelines.
   • "To deliver X, we need Y engineering cycles or a phased approach; otherwise quality/scale will suffer."
4. Co-create options: present three buyer-facing alternatives (off-the-shelf, phased MVP + roadmap, professional services workaround) with timelines and risks.
   • Ask: "Which option preserves the sale while keeping delivery realistic?"
5. Commit: agree on language for customer conversations and escalation path. Role-play the customer call and craft a “safe” promise script.
   • e.g., "We can commit to delivering core capability A within 90 days, with B on our roadmap for Q3 and regular demos in between."
6. Follow-up: weekly sync for first 8 weeks, review wins, blocked items, and calibration with engineering.

Alignment tactics (team-level):

• Create a predefined “commitment matrix” tying feature categories to approval levels (Director-level can promise configs; anything requiring >2 engineers or >8 weeks needs VP/product sign-off).
• Introduce a lightweight Deal Review (Solutions Architect + Product + Engineering) for high-risk deals before verbal commitments.
• Provide sales with templated, customer-facing commitment language and solution diagrams showing what’s included vs. roadmap.
• Use shared success metrics: win-rate weighted by delivery accuracy and customer satisfaction.

Organizational changes to prevent recurrence:

• Formalize a pre-sales governance process: threshold-based Deal Review board and SLA for technical turnaround (e.g., 48–72 hours for feasibility).
• Empower Solutions Architects as mandatory approvers on Statement of Work / proposal templates for technical deliverables.
• Introduce a Product-Sales playbook mapping common asks to delivery patterns (supported, professional services, roadmap) and estimated time-to-market.
• Create incentives: tie a portion of sales compensation or recognition to “delivery alignment” KPIs (e.g., percentage of deals delivered as promised).
• Invest in a rapid prototype/POC pool (small engineering reserve) to de-risk commitments quickly.

Why this works:

• It preserves sales momentum by offering practical, customer-friendly alternatives.
• It builds trust between Sales and Engineering through structured collaboration and clear approval paths.
• It protects customers and the company from over-commitment while enabling controlled, accountable exceptions when strategically necessary.

Requirements (scope & objectives):

• Enforce technical, legal, and operational controls before onboarding any third‑party vendor that will access company data, systems, or customers.
• Applicable to SaaS/PaaS/IaaS, managed services, professional services, and embedded components.
• Objectives: ensure confidentiality/integrity/availability, legal compliance, operational readiness, maintainable offboarding.

Policy Sections and Key Clauses

1. Governance & Applicability

• Clause: This policy applies to all business units, procurement, IT, Security, Legal, and Engineering teams involved in vendor selection/onboarding.
• Clause: Ownership — CISO owns policy; Vendor Management Office (VMO) enforces.

2. Pre‑Onboarding Gates (must be satisfied before contract signature)

• Security Assessment Clause: Vendors must complete a risk classification and pass a security assessment (including architecture review, penetration test report or attestation, data flows, encryption, IAM, vulnerability management, and SOC2/ISO27001 evidence where applicable).
• Legal Review Clause: Legal must approve contract terms for data protection, liability, indemnity, export control, breach notification timelines, data processing addendum (DPA), and IP/CLA considerations.
• Pilot Sign‑Off Clause: A defined pilot (scope, success criteria, duration) must be executed in a non‑production environment; product owner + Security + Operations must sign off in writing.
• Operational Readiness Clause: Runbooks, playbooks, and checklists must be published and validated (deployment, rollback, incident escalation, monitoring, backup/restore, SLOs/SLAs).
• Business Acceptance Testing (BAT): BA owner validates functional and non‑functional requirements.

3. Runbooks & Operational Artifacts

• Clause: Minimum artifacts: deployment runbook, runbook for common incidents, on‑call rotation, escalation matrix, support SLAs, maintenance window procedures, capacity planning.
• Clause: Artifacts versioned in company repo; access controlled.

4. Exceptions & Risk Acceptance

• Exception Process Clause: Exceptions require a written Risk Acceptance signed by CISO and Business Sponsor, documenting compensating controls, duration (max 90 days), and mitigation plan with milestones.
• Review Cadence Clause: Exceptions reviewed every 30 days until closed.

5. Deprecation & Offboarding

• Deprecation Timelines Clause: When a vendor or integration is deprecated, vendor and business owner must implement a 90/180/365‑day timetable based on classification, with documented migration/retirement plan.
• Exit Strategy Clause: Every contract must include: data export format & timelines, secure data deletion certification, portability of configs, transitional support, knowledge transfer, and service continuity plan for at least 60 days post‑termination.
• Termination Testing Clause: Annual offboarding dry‑run for critical vendors.

6. Monitoring, Metrics & Reporting

• Metrics Clause: Track and report: % vendors with completed security assessments, % with legal/DPA signed, % with runbooks published, mean time to remediate assessment findings, % onboarding without exception, number of active exceptions, time-to-onboard (days).
• Dashboard & Reporting Clause: VMO publishes monthly dashboard to Risk Committee; critical deviations flagged immediately.
• Audit Clause: Quarterly audits by Internal Audit; results shared with Exec Risk.

7. Enforcement & Remediation

• Non‑Compliance Clause: Systems or integrations deployed without sign‑off must be quarantined; business sponsor accountable for remedial costs.
• Remediation SLA Clause: High‑risk findings require remediation within 30 days or formal risk acceptance.

8. Roles & Responsibilities

• Define accountable roles: Business Sponsor (requirements/acceptance), Solution Architect (technical review), Security (assessment), Legal (contract), Operations (runbooks), VMO (policy enforcement), Procurement (contracting).

9. Continuous Improvement

• Clause: Policy reviewed annually or after major incidents; lessons from pilots, postmortems fed back into templates.

Examples & Templates (appendices)

• Security questionnaire template, runbook template, pilot sign‑off checklist, exception form, exit checklist.

Rationale & Trade‑offs

• Balances speed-to-market with risk control via formal exception path and pilot requirement; emphasizes operational readiness to reduce production incidents and ensure clean offboarding.

Direct answer
The choice between synchronous and asynchronous service communication is really a choice about whether the calling service can afford to wait for an answer right now, and getting this wrong in either direction shows up as either a slow, fragile user experience or a system that's needlessly hard to reason about.

Structured elaboration
Trade-offs across the key dimensions:

• Latency: synchronous calls add the full latency of the downstream service directly to the caller's response time; asynchronous decouples the caller's response time from the downstream work, which can complete later.
• Reliability: synchronous calls fail together (if the downstream service is down, the caller's request fails immediately); asynchronous calls (via a durable message queue) can be retried and processed once the downstream service recovers, without the original caller needing to know or retry itself.
• Complexity: synchronous is simpler to reason about (a request either succeeds or fails, immediately, in one place); asynchronous introduces real complexity, including message ordering, duplicate delivery, and the need to communicate eventual results back to whoever's waiting for them.
• Error handling: synchronous errors surface immediately to the caller, who can decide how to respond in the moment; asynchronous errors require an explicit mechanism (a dead-letter queue, a status callback) or they risk failing silently, invisible until someone notices the expected downstream effect never happened.
• Debugging: synchronous call chains are easier to trace (a single request's path through the system is linear); asynchronous flows require distributed tracing across a queue and multiple independent consumers, which is a real operational investment to build well.
• User experience: synchronous is appropriate when the user is actively waiting for the result and needs it to proceed (checking out an order); asynchronous is appropriate when the user doesn't need to wait (an email confirmation, a background report generation), and forcing them to wait synchronously for non-blocking work degrades their experience for no real benefit.

Examples: a payment authorization during checkout should be synchronous, since the user needs an immediate answer to proceed or be told why they can't; a post-purchase email receipt and a recommendation-model retraining triggered by that purchase should be asynchronous, since neither needs to complete before the user's checkout flow finishes.

Validating these choices under load: load-test the synchronous path specifically for how it behaves when the downstream dependency itself slows down under load (does the caller's own capacity get consumed waiting, causing a cascading slowdown), and load-test the asynchronous path for queue depth and processing lag under a sustained high-volume burst, confirming the eventual-completion time stays within an acceptable bound even during peak load, not just under average conditions.

Trade-offs and pitfalls
The most common mistake is defaulting to synchronous calls throughout a system because it's simpler to build initially, creating a fragile chain where a slowdown in any downstream service cascades directly into user-facing latency everywhere upstream. The second common mistake is making something asynchronous that the user is actually waiting for, which just adds indirection and complexity without improving the actual experience, since the user still can't proceed until the async work completes anyway.

Diagnosing the intermittent per-customer latency

• First confirm it's genuinely per-customer: segment the existing latency metrics by tenant or customer ID. If that dimension doesn't exist yet, adding it is itself the first architecture gap to fix, since without it, "which customer is affected and why" stays a manual, slow investigation every time.
• Once confirmed, pull distributed traces for that customer's slow requests specifically during a peak-hour window and look for what's different about their request shape: larger payloads, a different API path, a noisy-neighbor effect from being colocated with a high-traffic tenant on shared infrastructure, or a downstream dependency, such as a per-tenant database shard, that's disproportionately loaded.

Architecture and observability changes to recommend

• Add tenant ID as a first-class label on latency metrics, logs, and trace spans everywhere in the call path.
• If tenants currently share infrastructure with no isolation, a common root cause of symptoms that are "intermittent" and "only during peak," introduce per-tenant rate limiting or resource quotas so one tenant's peak usage can't degrade another's latency.
• Add a per-tenant latency dashboard and automated anomaly detection, for example alerting if one tenant's p95 (the 95th percentile: the latency value 95% of requests finish faster than) diverges meaningfully from the fleet-wide p95, so this kind of issue surfaces proactively instead of through a customer complaint.

Tiered SLAs and per-tenant quality-of-service, QoS, controls

• Define explicit latency service level agreements, SLAs, per tier, for example a premium-tier target of p95 under 150ms and a standard-tier target of p95 under 400ms, backed by matching infrastructure guarantees. A tier promise without a QoS mechanism behind it is just a document.
• Implement QoS through prioritized queuing and resource reservation: give premium-tier requests a separate, or priority, queue and a reserved slice of connection-pool and thread-pool capacity, so they aren't waiting behind standard-tier load during a shared-infrastructure surge.
• Apply per-tenant rate limits sized to each tenant's tier and contracted usage, so no single tenant, regardless of tier, can consume enough shared capacity to degrade others. This is what actually protects the SLA promise operationally, not just what's written in the contract.
• Consider dedicated infrastructure, a dedicated shard, node pool, or even region, for the highest tier if shared-infrastructure QoS controls still can't reliably meet their SLA under worst-case contention.

Direct answer

SLI, SLO, and SLA are three layers of the same idea, stated with increasing weight. An SLI (service level indicator) is what you actually measure. An SLO (service level objective) is the internal target you set for that measurement. An SLA (service level agreement) is the external promise, usually contractual, built on top of that target, with consequences if you miss it. In plain terms: the SLI is the speedometer, the SLO is the speed limit you've set for yourself, and the SLA is the speed limit you've promised a customer you won't exceed, with a penalty if you do.

Picking the example and the threshold

• Choose one measurable thing the executive already cares about, not an internal engineering metric they have no context for. "Percent of requests that succeed" beats a raw latency percentile for this audience, because success or failure needs no further explanation.
• State the SLO as a number deliberately below what looks achievable. This is the part executives most often misread: an SLO of 99.9% isn't "we're at 100% and slipping a little," it's a chosen buffer that leaves room to ship changes and absorb normal failures without over-investing in reliability nobody needs.
• The SLA number sits below the SLO, with a consequence attached, and that gap is itself worth explaining: it exists so that missing the internal target doesn't automatically mean breaking a customer promise.
• The same three-layer structure holds outside web services too, whether you're onboarding a new product manager on a team's SLOs for the first time or defining an SLI/SLO for a streaming data pipeline (there the SLI might be how stale the data is, instead of whether a request succeeded), the relationship between the three layers doesn't change, only what's being measured does.

Worked example

Say the team sets it up this way for a checkout API. SLI: percent of checkout requests that return successfully within two seconds. SLO: 99.9% of checkout requests meet that bar, measured over a rolling 30 days, the number engineering is held to internally. SLA: 99.5% of checkout requests meet that bar, measured monthly, written into the enterprise customer contract; falling below it triggers a service credit specified in the contract.

Business consequence of missing the SLA (say the month comes in at 99.3%): this is a contractual breach. The customer is owed the agreed credit, and depending on the contract, may have grounds to escalate or walk away. It's a direct, quantifiable cost and a trust hit that shows up outside engineering entirely.

Business consequence of exceeding the SLO (say the month comes in at 99.97% against a 99.9% target): this isn't a "consequence" in the SLA sense, it's a signal. Consistently beating the SLO by a wide margin means either the target is stale and could absorb more risk (ship faster, take on more ambitious changes), or the team is over-investing effort in reliability the product doesn't need. Either way it's a prompt to revisit the number, not something to report as a win on its own.

Trade-offs and pitfalls

The most common executive misunderstanding is treating the SLO as the promise, when the SLA is the promise and the SLO is the internal cushion above it. Say that gap out loud every time, or the SLO number will get quoted externally by mistake. The second pitfall is picking a metric that's technically correct but means nothing to the audience, an uptime percentage without saying what "down" costs the business, always translate the metric into what the customer actually experiences before attaching a number to it.

Direct answer

Each service owns its own data and announces changes as events on a durable log (Kafka or similar; Kafka calls a named log a topic, and splits each topic into ordered partitions, so "in order" is a guarantee within one partition, not across the whole topic), written through a transactional outbox so an event is published if and only if the change committed. The booking itself is a saga: a sequence of local steps (hold dates, authorize payment, confirm, capture) where each step emits an event that triggers the next, and each failure triggers a compensating step (release the hold, void the authorization). Every consumer is idempotent (it records which event IDs it has applied), tolerates out-of-order delivery by comparing a per-entity version number, retries transient failures with backoff, and parks poison messages (events that fail processing no matter how many times they are retried, usually a bug or a malformed payload) in a dead-letter queue with a replay tool. The one place I would not go asynchronous is the final "are these dates still free" check: that is a synchronous, transactional hold in the availability service.

Terms, briefly

• Event: a fact that already happened ("BookingConfirmed"), published by the service that owns it.
• At-least-once delivery: the log guarantees each consumer eventually receives every event, possibly more than once. Exactly-once processing end to end is achieved by combining at-least-once delivery with idempotent consumers.
• Outbox pattern: the service writes the event into an outbox table in the same database transaction as its state change; a relay (a small background process that reads new outbox rows, publishes them to the log, and marks them sent) does the publishing. Without it, a crash between "commit" and "publish" silently loses the event.
• Saga: a multi-service transaction built from local transactions plus compensations, instead of one distributed lock across services.

Event map

A note on ordering, since the table alone does not show it: the initial hold is a synchronous call, not an event (see the sequence below), so it never appears as something Availability "consumes". Availability's event consumption starts with BookingConfirmed, which turns a held date into a permanently booked one, and PaymentFailed or BookingCancelled, which release the hold. Booking only publishes BookingRequested after that synchronous hold has already succeeded, which is what "(after hold)" means on the Payments row: by the time Payments sees BookingRequested, the dates are already protected.

Payment terms: authorization reserves the amount on the guest's card without moving money; capture actually takes it (at confirmation or check-in, depending on policy); a void cancels an authorization that was never captured.

The booking flow

sequenceDiagram
  participant G as Guest
  participant B as Booking
  participant A as Availability
  participant P as Payments
  G->>B: Request booking
  B->>A: Hold dates (synchronous)
  A-->>B: Held until T plus 15 min
  B->>P: BookingRequested event
  P-->>B: PaymentAuthorized event
  B->>A: BookingConfirmed event
  A-->>A: Hold becomes booked
  B-->>G: Confirmed

Why the hold is synchronous: two guests clicking "book" for the same nights must get a definitive answer immediately, and only one can win. The availability service does it with a transactional conditional write (for example, a unique constraint on (listing_id, night) in a relational table, so the second insert fails). Everything after the hold can be asynchronous because the dates are already protected, and the hold's expiry (15 minutes here) guarantees they are released if the saga stalls.

Compensations: PaymentFailed makes Booking emit BookingCancelled, and Availability releases the hold. A hold that expires emits HoldReleased; if a PaymentAuthorized arrives after that, Booking voids the authorization rather than confirming dates it no longer holds.

Retries

• Transient failures (a dependency timeout): retry in the consumer with exponential backoff and jitter (random spread, so a thousand consumers do not retry in lockstep), for example 1 s, 2 s, 4 s up to a cap.
• Do not block the partition for long: after a few in-place attempts, move the event to a retry topic (a separate topic that holds only events waiting out a delay before another attempt) with a delay (a 1-minute topic, then a 10-minute topic), and let the main partition continue. Caveat: this breaks ordering for that entity, which is why the out-of-order handling below is mandatory, not optional.
• Payments side effects carry an idempotency key (the booking ID plus the step name) on every call to the payment provider, so a retried authorization cannot charge twice. Payment providers such as Stripe accept such a key on their API requests for exactly this reason.

Out-of-order delivery

The log keeps order within one partition, so choosing the partition key is the first defence: every event about a booking uses booking_id, every event about a listing's calendar uses listing_id. That covers most cases, but order still breaks when events for one entity come from different producers (Booking and Payments both talk about one booking), or after a retry-topic detour.

Second defence: each entity carries a version that the owning service increments on each change, and each event carries the version it produced. Consumers keep the last version applied per entity and ignore events with a lower or equal version. For read models (a service's own copy of data built for fast queries, such as search's index, kept in sync via events rather than owning the source of truth) such as search, that is enough: a stale "DatesFreed v7" arriving after "DatesBooked v8" is simply dropped.

For state machines that consume events from several producers (Booking), use guards instead: "PaymentAuthorized for a booking in state CANCELLED means void it", "HoldReleased for a booking in state CONFIRMED is ignored".

Deduplication

Every event has a unique event_id. Consumers apply the event and insert its ID into a processed_events table in the same transaction, with a unique constraint on the ID; a duplicate either fails the insert or is detected before applying. Keep the table for longer than the maximum redelivery window (for example 7 days), then expire it. Consumers that write to stores without transactions (a search index) instead make the write naturally idempotent: "set listing 42's booked nights to this version-8 set", never "add one booked night".

Dead-letter queues

An event that fails every retry (a malformed payload, a bug) moves to a DLQ (dead-letter queue) with the error, the consumer name and the attempt count. Rules that make a DLQ useful:

• Alert on DLQ depth per consumer; a DLQ nobody watches is a slow data-loss channel.
• Provide a replay tool that re-publishes selected DLQ events to the original topic after a fix. Replays are safe only because consumers are idempotent.
• Classify: a payment or booking event in the DLQ is a customer-visible incident; an analytics event is not.

Scale check

Suppose 200,000 bookings/day with about 7 events per booking (requested, held, authorized, confirmed, captured, booked, notifications). That is 200,000 × 7 = 1,400,000 events/day, or 1,400,000 / 86,400 ≈ 16.2 events/s on average, a few hundred per second at peak. Throughput is not the challenge; correctness under retries, duplicates and reordering is. Calendar updates from hosts syncing external calendars can outnumber booking events and should be rate-limited per host.

Trade-offs and pitfalls

• Events for everything, including the double-booking check: availability becomes eventually consistent (replicas can briefly disagree; the system only guarantees they converge to the same answer eventually, not instantly) and two guests can both "win". Keep that one step synchronous and transactional.
• Publishing directly after commit without an outbox: loses events on crashes, and the missing event is found weeks later in reconciliation.
• Choreography vs orchestration: I described choreography (services reacting to each other's events). With more than four or five steps, or complex timeouts, I would move the booking saga to an explicit orchestrator (a workflow engine that drives the steps and records saga state), because debugging a flow spread across five consumers is hard.
• Unbounded retries in place: one poison event blocks the whole partition and every listing hashed to it.