Marketplace, Dispatch, and Logistics System Design Questions

Direct answer

Make the quote a first-class, immutable record with an ID, and make the booking reference it. At quote time the pricing service writes the quote (price, its components, the inventory it applies to, the pricing version, an expiry) and returns a quote_id. At booking, the order service accepts only a quote_id, re-reads that stored quote, checks it is unexpired and unused, places the inventory hold, and writes the order carrying the quote's price in one transaction. The client never sends a price. A continuous reconciliation job (one that independently compares records that should agree and flags where they don't) then joins quotes, orders, holds and payment captures and flags any order whose charged amount differs from its quote. Honoring the quote is enforced by construction; reconciliation proves it happened and catches the cases construction missed.

The flow

sequenceDiagram
  participant C as Client
  participant P as Pricing service
  participant O as Order service
  participant I as Inventory
  participant L as Event log
  C->>P: request quote
  P->>P: store quote (immutable)
  P-->>C: quote_id, price, expires_at
  C->>O: create order (quote_id, idempotency key)
  O->>O: load quote, check expiry and unused
  O->>I: conditional hold on item
  O->>O: insert order plus outbox event (one transaction)
  O-->>C: confirmed at quoted price
  O->>L: publish order event from outbox
  P->>L: publish quote event

The pieces, in the order they matter

1. Idempotency keys. The client generates an idempotency key (a unique ID for one booking attempt, resent unchanged on every retry) when the checkout screen opens. The order table has a unique constraint on it, so a retry after a timeout returns the existing order instead of creating a second one. A second, business-level uniqueness rule makes a quote single-use: a unique constraint on orders.quote_id, so two different attempts cannot both redeem the same quote.

2. Concurrency on the inventory: optimistic versus pessimistic. Both approaches stop two orders from taking the last unit:

I would use the optimistic conditional update as the default, because in a marketplace almost every item is lightly contended and a single conditional statement is also the shortest possible critical section (the smallest span of code that touches shared data, where letting another transaction interleave could corrupt the result). For known hot items (a limited release, a stadium event) I switch those items to the pessimistic path or to unit rows claimed with SKIP LOCKED (splitting the inventory into one row per physical unit, where each worker selects and locks one still-unclaimed row while skipping rows another transaction already has locked, so buyers compete for distinct rows instead of queuing on one shared counter), because a high retry rate under optimistic concurrency wastes more work than queueing does.

3. Event ordering guarantees. Downstream consumers (reconciliation, analytics, notifications) need to see, per order, created before paid before cancelled. Global ordering is expensive and unnecessary. What I guarantee:

• Per-key ordering: events are published to a partitioned log (Kafka or similar: a distributed, append-only stream split into independent partitions so many producers and consumers can work in parallel) keyed by order_id, and a log partition (one of those parallel sub-streams) preserves the order events were written in, so all events for one order, which always land in the same partition because they share a key, arrive in the order they were written.
• Atomic publish: the order row and its event are written in the same database transaction into an outbox table, and a relay (a small background process that reads new outbox rows and publishes them to the log) does that publishing. Without the outbox, a crash between "commit order" and "publish event" loses the event (the order exists but reconciliation never sees it).
• Per-entity sequence numbers: every event carries order_version. A consumer that sees version 3 before version 2 (possible after a retry or replay) holds or discards it rather than applying it out of order.
• Quote events and order events are on different keys, so they can arrive in either order. The reconciliation join must tolerate an order arriving before its quote, which is why it waits a grace period before declaring a quote "missing".

4. Reconciliation. Reconciliation means independently comparing records that should agree and flagging differences. A job runs every few minutes over a sliding window (with a final daily pass) and checks:

5. Dispute resolution. When a customer says "I was shown a different price", support looks up the order, follows quote_id to the immutable quote record, and compares: shown price (quote), booked price (order), charged price (payment). The rules are simple and pre-agreed: if the charge exceeds the quote, refund the difference automatically; if the quote itself was wrong (a pricing bug), honour it anyway and file the incident. The quote store is the evidence, so it must be append-only.

Schemas

CREATE TABLE quotes (
  quote_id        TEXT PRIMARY KEY,
  customer_id     TEXT NOT NULL,
  item_id         TEXT NOT NULL,          -- listing, room type, SKU (stock keeping unit: a seller's unique code for one product)
  item_params     JSONB NOT NULL,         -- dates, quantity, options (JSONB: PostgreSQL's binary JSON column type, for a flexible bag of fields)
  currency        CHAR(3) NOT NULL,
  subtotal_minor  BIGINT NOT NULL,        -- amounts in minor units (cents)
  fees_minor      BIGINT NOT NULL,
  tax_minor       BIGINT NOT NULL,
  total_minor     BIGINT NOT NULL,
  pricing_version TEXT NOT NULL,          -- model or rule set that produced it
  created_at      TIMESTAMPTZ NOT NULL,
  expires_at      TIMESTAMPTZ NOT NULL
);                                        -- insert-only: no UPDATE grants (the database role the app connects as is never given permission to run UPDATE on this table, so an accidental edit is rejected by the database itself)

CREATE TABLE orders (
  order_id        TEXT PRIMARY KEY,
  idempotency_key TEXT NOT NULL UNIQUE,
  quote_id        TEXT NOT NULL UNIQUE REFERENCES quotes(quote_id),
  total_minor     BIGINT NOT NULL,
  status          TEXT NOT NULL,
  version         INT NOT NULL,
  created_at      TIMESTAMPTZ NOT NULL
);

CREATE TABLE recon_findings (
  finding_id  BIGSERIAL PRIMARY KEY,
  order_id    TEXT,
  quote_id    TEXT,
  check_name  TEXT NOT NULL,
  expected    BIGINT,
  actual      BIGINT,
  status      TEXT NOT NULL,              -- open, auto_refunded, resolved
  found_at    TIMESTAMPTZ NOT NULL
);

Store money as integer minor units, never floats: 0.1 + 0.2 in binary floating point is not exactly 0.3, and reconciliation that compares amounts for equality will produce false mismatches.

Worked example

A guest gets quote q_77 at 14:00:00 for 2 nights: subtotal 480.00, fees 43.20, tax 34.56, total 557.76 USD, expiring 14:15:00. Surge rises at 14:03 so a fresh quote would now be 612.00. At 14:05 the guest books with quote_id = q_77 and idempotency key k_91. The order service loads q_77, sees it is unexpired and unredeemed, runs the conditional hold on the listing's two nights (1 row updated, so the hold succeeded), and inserts the order with total_minor = 55776 plus its outbox event in one transaction. The guest's app times out and retries with k_91: the unique constraint returns the existing order, so no second booking. Payment captures 557.76. That evening the reconciliation pass finds quote total = order total = captured amount and records nothing. If a bug had captured 612.00, the check payment.captured_amount = order.total would open a finding with expected 55776 and actual 61200, and the auto-refund rule would return 54.24.

Trade-offs and pitfalls

• Trusting a client-supplied price is the classic hole: anyone can edit a request body. Only the quote_id crosses the boundary.
• Quote validity is a business lever. Longer validity improves conversion but lets customers lock in low prices before surges; 10 to 15 minutes is a common middle ground for dynamic pricing.
• Storing quotes costs writes: every quote is a write even though most are never booked. If quotes vastly outnumber bookings, store them in a cheap append-only store with a TTL (time-to-live, after which records are deleted automatically) and copy the redeemed ones to the order database at booking.
• Reconciliation that only runs daily finds overcharges after customers do. Run it in near real time and keep the daily pass as the complete one.

Direct answer

At this scale I would build a cell-based, region-partitioned platform (a "cell" here is one geographic partition of the road graph, the same thing the rest of this answer calls a "partition"): each partition, with its live driver state and traffic, is owned by one region, and a small global overlay stitches partitions together for trips that cross them. Consistency is deliberately weak and versioned on the data path (telemetry, traffic, graph metrics are eventually consistent with bounded staleness) because an ETA (estimated time of arrival) is a prediction, not a balance; the only strongly consistent records are the trip and dispatch decisions, which live in the trip's home region. The p99 target of 200 ms (99% of API calls answered within 200 ms) is met by keeping every hot-path hop inside one region, giving each service its own latency and error budget derived from the end-to-end one, and having a precomputed or cached answer to fall back on at every step.

Scale, derived

Two readings of these numbers. Ingest is the heaviest stream and it is write-mostly, so it must never touch a durable database synchronously. And if a variant of this requirement quotes 10 million requests per second, that is roughly 145x the load above: it only works if the vast majority of requests are served from precomputed or cached ETAs (for example, zone-to-zone tables refreshed every minute), with live routing reserved for requests that change a decision (dispatch, an active trip).

Services and data flow

flowchart LR
  subgraph Region[One region, owns partitions P1..Pn]
    GW[Edge gateway] --> MM[Map-matching]
    MM --> ST[Driver state]
    MM --> TF[Traffic fusion]
    TF --> MET[Metric publisher]
    MET --> RT[Partition routers]
    ST --> ETA[ETA API]
    RT --> ETA
  end
  OV[Global overlay router] <--> RT
  MB[Offline map build and profiles] --> RT
  ETA --> C[Clients and dispatch]

• Edge gateway: terminates TLS (encryption in transit), holds persistent connections, routes each device to its partition's region by geography.
• Map-matching: snaps noisy GPS to road segments using a hidden Markov model style matcher (it chooses the most likely sequence of road segments given the sequence of GPS points).
• Traffic fusion: turns matched movements into per-segment speeds, blends with third-party feeds and historical profiles, and publishes a new metric version (a complete set of edge travel times) every 30 to 60 s.
• Partition routers: hold one graph partition in memory with a speed-up structure (for example, a customizable contraction hierarchy, which pre-computes shortcut edges once and cheaply re-weights them when traffic changes).
• Global overlay router: holds only the boundary nodes between partitions and the travel times between them, so a cross-partition route is: origin partition to its boundary, overlay across, boundary to destination partition.
• ETA API: route time plus learned corrections, with caches and fallbacks.

Graph partitioning and shard boundaries

Partition along natural cuts (rivers, mountain ranges, sparse rural roads) so few edges cross a boundary, not on a latitude/longitude grid that slices through a downtown. Each partition stores a halo (a margin of neighbouring roads) so a short trip near a border stays inside one router. Assign partitions to regions by where the trips originate, not where the servers are cheapest.

Consistency assumptions

Cross-region concerns

• Long trips across partitions and regions (for example, an airport run that crosses a regional boundary): the origin region plans the whole route using the overlay; the far part of the trip uses historical profiles because live traffic 90 minutes ahead is not predictive.
• Time zones: store every timestamp in UTC; look up historical speed profiles by the edge's local time of day, advancing the clock as the route proceeds, so a trip that crosses a time-zone boundary at 16:50 uses that zone's rush-hour profile.
• Failover: every partition has a warm standby (a second region kept running and continuously up to date, but not serving live traffic, so failover is a reconnect rather than a cold boot) in a second region with the same map build and profiles. Live driver state is not replicated synchronously; after failover, drivers re-report within one update interval and the state rebuilds itself.
• Egress cost (the per-gigabyte fee a cloud provider charges for data leaving a region, usually far higher than the fee for data that stays in-region): keep raw telemetry in its region and replicate only aggregates. Replicating the full 1.5 TB/day to two other regions would move 3 TB/day across regions for data almost nobody reads there.

Latency budget and error-budget coordination

p99 latencies compound. If one API request fans out to 10 parallel calls that each meet 100 ms at p99, the chance all 10 are fast is 0.99^10, about 90.4%. In other words, about 9.6% of requests (1 minus 0.904) include at least one call that missed its 100 ms target, not the 1% a single p99 target would suggest, so the request's own p99 is far worse than 100 ms. Each hot-path dependency therefore needs a much tighter target: at p99.9 per call, 0.999^10 is about 99.0%, meaning only about 1% of requests would include a slow call instead of 9.6%, which is what keeps the whole request's own p99 close to the 100 ms target rather than far above it. I would write the end-to-end SLO (service-level objective: the internal reliability target) first, then derive a latency and error budget per service, for example: gateway 10 ms, state lookup 10 ms, routing 80 ms, corrections 20 ms, network and serialization 40 ms, with 40 ms slack. Services share one error-budget dashboard: an error budget is the amount of failure an SLO allows before it is considered breached, and "burning budget" means a service has been failing enough recently to be spending that allowance down. So a team burning budget freezes risky deploys until it recovers, not everyone.

Cost and latency trade-offs

Failure modes and mitigations

Direct answer

Matching is local: a rider in Paris is never matched to a driver in Tokyo. So "active-active across 3 regions" should mean every region serves traffic, and every city is homed in exactly one region at a time, not that one city is matched simultaneously in several regions. Each region runs the full matching stack for its cities with in-region consensus for assignments; a global directory, changed only through fenced epochs, says which region owns each city and each driver. Location indexes are never replicated across regions (they rebuild from driver pings in seconds); trip and account metadata replicates asynchronously to a designated standby region. That keeps matching latency at in-region speed, makes "the same driver in two regions" a directory conflict with one defined winner, and makes failover a controlled re-homing of cities.

Terms

• Active-active: all regions take production traffic, as opposed to one active and others idle.
• Leader election: choosing one node to be the single decision-maker for some data, typically with a consensus protocol such as Raft (a protocol where a majority of replicas must agree before a decision counts).
• Lease and epoch: a leader's authority expires unless renewed (lease), and each new leader gets a higher number (epoch) that is attached to its writes so an old leader's writes can be rejected (fencing).
• RPO / RTO: recovery point objective (how much recent data you may lose) and recovery time objective (how long recovery may take).

Requirements I am designing to

• Matching p99 (99th-percentile) latency in the low tens of milliseconds, measured server-side.
• Survive the loss of a whole region with city traffic back within minutes.
• No driver ever holding two concurrent trips.

Why cross-region synchronous replication is ruled out

Light in fibre travels at about 200,000 km/s. For regions 6,000 km apart, one way takes at least 6,000 / 200,000 = 30 ms, so a round trip is at least 60 ms before any processing. A synchronous cross-region write on every assignment would by itself exceed a low-tens-of-milliseconds budget. So: synchronous within a region (replicas in three availability zones, which are separate data centres in the same metro, so that a majority, two of three, can keep agreeing even if one zone is lost), asynchronous across regions.

Architecture

flowchart TB
  DIR[Global directory: city and driver home, epochs]
  subgraph RA[Region A]
    MA[Matching for its cities] --> SA[Trip and driver status, Raft in 3 zones]
  end
  subgraph RB[Region B]
    MB[Matching for its cities] --> SB[Trip and driver status, Raft in 3 zones]
  end
  DIR --> MA
  DIR --> MB
  SA -. async replication .-> SB
  SB -. async replication .-> SA

Data partitioning and replication

The global directory is the only data that uses cross-region consensus, and it is written only when a city or driver changes home, never per match.

Leader election for assignments

Within a region, each city (or each geographic shard of a large city) has one assignment leader elected by Raft, holding a lease with an epoch. Every assignment writes driver status with a compare-and-set that includes the epoch; a deposed leader's writes fail. Across regions, only the directory can move a city's home, and moving it bumps the city epoch, so even if an old region is still running after a network partition (the regions temporarily lose the ability to talk to each other, though each keeps running on its own), its writes carry the old epoch and are rejected once it reconnects.

When the same driver appears in two regions

This happens in three ways: a driver near a border between cities homed in different regions, a driver relocating (flies from one country to another), and a split brain (both regions believing they are the sole authority for the same driver) during a network partition.

Region-selection rules for drivers:

1. A driver's home region is the home of the city whose operating area they are in when they go online.
2. Mid-trip, a driver never changes home; the trip finishes in the region that created it, even if it ends across the border.
3. Between trips, if the driver's position is in a city homed elsewhere for more than a short dwell time (for example 2 minutes, to avoid flapping at a border), the directory moves their home with an epoch bump. The old region stops offering them the moment its epoch is stale.
4. A relocating driver who goes offline in one city and online in another is simply re-homed at go-online.

Conflict resolution when prevention fails (for example a partition in which both regions believe they own a driver): the directory's epoch decides. The assignment made with the current epoch stands; the other is cancelled, and its rider is re-matched at the front of the queue with a notification. Prevention is the main defence: during a partition, a region that cannot confirm its epoch for a city stops assigning in that city (choosing consistency over availability for assignment only, refusing to assign rather than risk a wrong or double assignment; browsing and quoting keep working).

Failover

1. Health checks declare region B down (several independent signals, not one probe, to avoid failing over on a monitoring glitch).
2. The directory re-homes B's cities to their standby region, bumping each city's epoch.
3. Driver and rider apps reconnect to the new region (DNS or client-side region lists). Within one ping interval (for example 4 s) the location index is populated.
4. In-flight trips are restored from the async replica; the last few seconds may be missing and are reconciled from the apps, which hold their own copy of the active trip.

Capacity: with 3 regions and one failed, the failed region's load splits over the other two, so each needs its own load L plus L/2, which is 1.5L. Normal utilisation must therefore stay at or below 1 / 1.5, about 67%.

Trade-offs and pitfalls

• Truly multi-master matching (letting more than one region independently accept assignments for the same city at once) for one city across regions needs cross-region coordination per assignment; the latency arithmetic rules it out.
• Replicating the location index across regions costs bandwidth for data that is stale by the time it arrives and is rebuilt in seconds anyway.
• Automatic region failover on a flaky signal causes more incidents than it prevents; require corroboration.
• Forgetting border dwell time makes drivers flap between regions.

Direct answer

For most marketplace bookings (a hotel room-night, a restaurant table, a seat on a class) the right model is optimistic concurrency implemented as an atomic conditional write in one strongly consistent database: "reserve one unit only if one is still free", checked and applied in a single statement, with a database constraint as a backstop. It adds no extra round trips when there is no contention and cannot double-book. Switch to partitioned ownership for extremely hot inventory (a concert on-sale) where thousands of requests hit one item, and use leader-based locking only when the critical section spans systems you cannot put in one transaction, always with fencing tokens. Distributed consensus belongs inside your database or coordination service, not in your booking code.

The four models

1. Leader-based locking

A lock service (or a single leader process) grants an exclusive lease, a lock that expires after a set time, on an inventory key; the holder reads availability, books, and releases.

• Latency: one round trip to acquire, the booking work, one to release. Under contention, requests queue behind the holder, so waiting time grows with the hold time.
• Failure modes: the holder pauses (a long garbage-collection pause, where the language runtime stops the program to reclaim unused memory and can unpredictably take seconds, or a network stall), its lease expires, a second client acquires the lock, and both write: split brain (two parties each believing they are in charge). The fix is a fencing token, a number that increases with each lease grant and that the storage layer checks, rejecting writes carrying an older token. Without it, a lock alone does not prevent double-booking. The lock service is also a dependency whose outage stops all bookings.
• Prefer when: the booking touches an external system (reserving with a third-party supplier plus your own record) that cannot share a transaction with your database.

2. Optimistic concurrency

No lock is held while the user thinks; the write succeeds only if the state is still what it expected. Two forms:

• Version check: read version = 7, then UPDATE ... SET ..., version = 8 WHERE id = ? AND version = 7. Zero rows updated means someone else won; re-read and retry.
• Conditional atomic update: UPDATE slot SET booked = booked + 1 WHERE id = ? AND booked < capacity. The condition is re-evaluated by the database at write time, so concurrent bookers of different units do not conflict at all.
• Latency: a single statement at low contention. On a hot row the database serializes the writers on the row lock for a very short time; with the version-check form, losers must retry, and retries grow quickly with contention.
• Failure modes: retry storms on hot items (version form), and application bugs that read then write in two separate steps without a condition, which is exactly the double-booking bug.
• Prefer when: almost always, for ordinary inventory.

3. Distributed consensus

Every booking write is committed through a consensus protocol such as Raft or Paxos (two well-known leader-based consensus protocols): a leader proposes the write and it counts as committed once a quorum (a majority of replicas) has stored it, so any two majorities overlap and no two conflicting bookings can both commit.

• Latency: at least one round trip from leader to a majority per write; with replicas in different regions, one inter-region round trip per booking, plus stalls during leader election.
• Failure modes: losing a majority makes writes unavailable (this is the price of never serving a wrong answer); leader elections cause brief write pauses.
• Prefer when: you need bookings to survive a region failure with no lost confirmed booking. In practice you buy this by choosing a database that replicates with consensus internally (for example a Raft-based distributed SQL database) and then still use the conditional write from model 2 on top. Writing your own consensus for a booking flow is a mistake.

4. Partitioned ownership

Each inventory key is owned by exactly one worker (an actor: a single-threaded worker that owns one slice of state and processes its messages one at a time, also called a shard owner) at a time, and all requests for that key are routed to it and processed one at a time in memory, then persisted.

• Latency: the lowest per decision (in-memory, no lock round trips), and it absorbs a hot key gracefully by queueing rather than retrying.
• Failure modes: hot keys cap at one worker's throughput; during failover a new owner must wait for the old owner's lease to expire (a pause for that key); and without fencing on the persisted write, an old owner that did not know it lost ownership can still write.
• Prefer when: a few items receive enormous concurrent demand (ticket on-sales, limited drops), usually with a waiting-room queue in front (an entry queue that holds excess requests outside the booking system and admits them gradually, so a flash-sale spike never hits the database directly).

Comparison

Worked example: a conditional write under a real race

The script below starts PostgreSQL in Docker, creates a slot with 50 seats, and fires 80 concurrent clients, each running one conditional update plus an insert in a single statement, using a WITH ... RETURNING construct (a CTE, or common table expression: a named, temporary result computed once and reused later in the same statement). The CTE s holds one row if and only if the UPDATE actually changed a row; the INSERT ... SELECT ... FROM s that follows it then runs once for each row s produced, so a client whose UPDATE matched zero rows (the seat was already full) also inserts zero booking rows, silently, no error and no booking, instead of needing a separate check. A CHECK constraint backs the rule at the storage layer.

#!/usr/bin/env bash
set -euo pipefail
C=booking-demo
docker run -d --name $C -e POSTGRES_PASSWORD=pw postgres:16-alpine >/dev/null
# The official image runs a transient server for initdb, shuts it down, then starts the
# real server; a plain "psql -c select 1" probe can succeed against that transient
# instance and then fail moments later while the real server is still coming up. Wait
# for the "ready to accept connections" line to appear twice (temp instance, then real).
until [ "$(docker logs $C 2>&1 | grep -c 'database system is ready to accept connections')" -ge 2 ]; do
  sleep 0.5
done

docker exec -i $C psql -U postgres -q <<'SQL'
CREATE TABLE slot (
  id int PRIMARY KEY,
  capacity int NOT NULL,
  booked int NOT NULL DEFAULT 0,
  CHECK (booked <= capacity)
);
CREATE TABLE booking (id serial PRIMARY KEY, slot_id int NOT NULL, user_id int NOT NULL);
INSERT INTO slot VALUES (1, 50, 0);
SQL

# 80 clients race for 50 seats at the same moment.
rm -rf /tmp/booking-demo && mkdir -p /tmp/booking-demo
for u in $(seq 1 80); do
  docker exec $C psql -U postgres -qtA -c "
    WITH s AS (
      UPDATE slot SET booked = booked + 1
      WHERE id = 1 AND booked < capacity
      RETURNING id)
    INSERT INTO booking (slot_id, user_id) SELECT id, $u FROM s RETURNING id" \
    > /tmp/booking-demo/client-$u.out &
done
wait

echo "clients that got a seat: $(cat /tmp/booking-demo/client-*.out | grep -c .)"
docker exec $C psql -U postgres -qtA -F ' ' -c "
  SELECT 'slot.booked =', booked FROM slot WHERE id = 1
  UNION ALL SELECT 'booking rows =', count(*) FROM booking"
docker rm -f $C >/dev/null

Output, with the readiness wait above (5 real runs, this exact script, unchanged otherwise):

clients that got a seat: 50
slot.booked = 50
booking rows = 50

Every run lands on the same three numbers, because the conditional UPDATE is correct regardless of exactly how many of the 80 clients happen to race each other; capacity, not timing, decides the count. (A naive readiness probe such as psql -c "select 1" looks correct in testing but is not: the official Postgres image starts a temporary server for initdb, shuts it down, and starts the real one, and a probe that only checks "did psql connect" can pass against the temporary instance, so the script then races the real server's startup. Across 5 runs with that naive probe this raced visibly: one run aborted before creating any tables, one run logged roughly two dozen client connection failures before still landing on the correct 50/50/50, and three ran clean. The fix above waits for the ready message to appear twice, which removed the race across 5 further runs.) Exactly 50 of 80 clients got a seat, and the other 30 received zero rows (a clean "sold out", not an error). This works because under PostgreSQL's default isolation level (READ COMMITTED: the rule that controls what concurrent, not-yet-committed changes a transaction is allowed to see), a second writer on the same row waits for the first to commit and then re-checks booked < capacity against the new value before updating. This re-check is specific to READ COMMITTED: under the stricter REPEATABLE READ or SERIALIZABLE levels, the second writer would instead abort with a serialization error once it tried to commit, and the application itself would have to retry the whole transaction; the single-statement conditional UPDATE stays correct under any of the three, but only READ COMMITTED turns the race into a silent, zero-row "sold out" instead of a visible error to handle. The naive version, SELECT booked in one statement and UPDATE slot SET booked = <value read + 1> in another, has no such re-check under any isolation level and is how double-bookings happen.

For a multi-unit booking (three consecutive hotel nights), the same idea applies per night inside one transaction: update all three rows conditionally and roll back if any returns zero rows. Update the rows in a fixed order (by date) so two concurrent transactions cannot deadlock (each one waiting forever for a lock the other transaction holds, with neither able to proceed) by locking nights in opposite orders.

Holds and the user's think time

Users pick a room and then spend minutes on payment. Do not hold a lock for that. Insert a hold row with the same conditional write and an expires_at (for example 10 minutes); confirmation converts the hold into a booking; a sweeper, or a condition that ignores expired holds, returns unconfirmed units. This keeps every model above working on short operations only.

Trade-offs and pitfalls

• A distributed lock alone is not safe. Without a fencing token checked by storage, a paused holder can write after its lease expired.
• Version-check retries on hot items multiply load exactly when load peaks; prefer the conditional update, or move the hot item to partitioned ownership with a queue.
• Consensus is not free at the booking layer: every cross-region confirmation pays an inter-region round trip; keep an item's writes in one home region unless region-failure durability is a hard requirement.
• Caches and search indexes are not the source of truth. Showing "available" from a cache is fine; the conditional write decides.

Direct answer

Mitigate first, diagnose second, fix third. In the first minutes: declare an incident, check for a recent deploy or config change and roll it back, and apply safe generic relief (widen the search radius, switch from batched to greedy matching, shift load off any hot partition). Then locate the failure with data: was supply actually missing, or were drivers present but not found or not offered? Scope it by partition and by pipeline stage (index freshness, matcher CPU and queues, downstream call timeouts, offer delivery and acceptance). The shape of the scope points at the cause. Long-term fixes follow from the cause, plus the detection and fallback gaps that let it run for 30 minutes.

Terms

• Unassigned request: a ride request that ended (timed out or was cancelled) without a driver accepting.
• Index lag: how old the positions in the matcher's location index are (now minus the timestamp of the last update applied).
• IC (incident commander): the one person coordinating the response, so others can investigate.
• Consumer lag: how far a stream consumer is behind the newest message.
• Partition / tile / cell: three names used for the same idea below. The city's matching workload is split into geographic slices; a partition is the underlying stream and processing shard that owns one slice, a tile is the map grid square that slice covers, and cell is used the same way when talking about one slice informally. "Moving a tile off a partition" means reassigning that geographic slice's traffic to a different, less-loaded shard.

Step 1: stabilise (first 10 minutes)

1. Declare and assign roles: IC, one investigator per subsystem, one person on communications (riders, drivers, city operations).
2. Change log first: any deploy, config change, feature-flag flip or infrastructure change in the hour before onset? Roll it back. Most incidents like this are self-inflicted, and a rollback is fast and reversible.
3. Generic relief that is safe to apply blind:
   • Widen the search radius and relax filters (for example allow a slightly longer pickup ETA).
   • Switch batched matching to greedy immediate matching if the batching job is slow.
   • Scale out the matcher and move tiles off any partition at high CPU.
   • Retry unassigned requests automatically rather than letting riders give up.
4. Communicate: in-app banner for riders in the city, status to operations.

Step 2: collect data (keep it; the post-mortem needs it)

Step 3: probable causes and their signatures

The last-but-one row is the organic rush-hour trigger: no-matches and long waits with every system healthy. The triage checklist is then: demand-to-supply ratio by zone, whether surge pricing engaged and how fast, whether the search radius and dispatch filters were too tight for the moment, driver acceptance rates (drivers may be declining long pickups), and whether drivers were stranded in the wrong part of the city. That is a marketplace-balance problem, and the fixes are pricing, repositioning incentives and radius policy rather than engineering.

Worked example (a hypothetical incident, to show the reasoning)

• Normal unassigned rate: 3%. During the 30 minutes: 21% city-wide.
• Online idle drivers: 1,100 before, 1,080 during. Supply was there, so it is not an organic surge.
• By partition: 3 of 16 partitions at 95% unassigned; the other 13 at 4%. If demand is roughly equal per partition, (3 x 0.95 + 13 x 0.04) / 16 = (2.85 + 0.52) / 16 = 0.21, which fully explains the 21%. So the problem is partition-scoped.
• On those 3 partitions: index lag p99 went from about 1 s to 45 s at the onset; CPU was normal. The other 13 were normal.
• The change log shows a location-consumer deploy that triggered a consumer-group rebalance (when the set of consumer instances reading a topic changes, for example because one restarts, the group redistributes which instance reads which partitions), which assigned those 3 partitions to one consumer instance that had crashed during startup, and the group waited for its session timeout (the time the group waits without a heartbeat from a member before deciding it is dead and reassigning its partitions) before reassigning.

Root cause: stale index on 3 partitions. Contributing factors: no alert on index lag, the matcher had no rule to refuse candidates with old positions, and the rebalance took minutes to notice a dead consumer.

Step 4: root-cause analysis and long-term fixes

1. Timeline from the data, then ask "why" until reaching a fixable cause, and separately list why it was not detected for 30 minutes and why it was not mitigated sooner.
2. Fixes by category:

Terms in the fixes table: a circuit breaker stops calling a failing dependency for a cooldown period instead of piling up timeouts on it; load shedding deliberately drops or delays lower-priority requests so the rest of the system stays up; a canary deploy rolls a change out to a small slice of traffic first and watches its metrics before rolling out to everyone; a game day is a scheduled rehearsal where the team deliberately injects a failure to test detection and response before a real one happens; a feature flag is a runtime switch that turns a code path on or off without a new deploy.

3. Blameless review with owners and dates on each action item, and a follow-up that verifies the alert would have fired.

Pitfalls

• Debugging before mitigating while riders wait.
• Looking at city-wide averages, which dilute a 3-partition failure into a modest bump.
• Assuming "no match" means "no drivers" without checking idle supply.
• Closing the incident without fixing detection, so the next one also runs for 30 minutes.