Marketplace, Dispatch, and Logistics System Design Questions

Direct answer

I would split the problem in two: a slow planner that every few minutes looks at the whole network (warehouse backlogs, carrier capacity, SLA risk) and publishes a small set of "prices" for scarce resources, and a fast per-order decider that, within 200 ms, enumerates the feasible warehouse-and-carrier options for one order, drops those that break hard rules, and picks the lowest expected cost, where expected cost includes a penalty for the chance of being late and the planner's price for using a nearly full carrier. The decision is committed by atomically reserving inventory and carrier capacity, with a pre-computed fallback if either reservation fails. Success is measured by on-time rate against the 95% SLA, cost per order, and decision latency at the 99th percentile.

Requirements and load

• 1,000,000 orders/day is 1,000,000 / 86,400 ≈ 11.6 orders/s on average. If the busiest hour carries 10% of the day, that is 100,000 / 3,600 ≈ 28 orders/s, and a flash sale at 5 times that is about 139 orders/s. This is modest; the 200 ms budget, not throughput, drives the design.
• SLA (service-level agreement, the promise to customers): 95% of orders delivered by the promised date.
• Each order may be fulfilled from several warehouses (possibly split across them) and by several carriers whose daily pickup capacity is live and limited.

Architecture

flowchart LR
  OS[Order service] --> RD[Routing decider]
  RD --> INV[Inventory service]
  RD --> CAP[Carrier capacity service]
  RD --> RULES[Rules and overrides]
  RD --> RATES[Rate and transit-time cache]
  PL[Network planner] --> PRICES[Shadow prices]
  PRICES --> RD
  RD --> EV[Decision log]
  EV --> PL
  CAR[Carrier APIs] --> CAP

Coordination between services: the decider calls inventory, capacity and rules synchronously, but only against local or in-memory data. Carrier APIs are never called on the per-order path: they are slow and unreliable, so capacity is mirrored locally and reconciled asynchronously.

The matching algorithm

1. Enumerate options: warehouses that hold every item (plus a small set of two-warehouse splits if no single warehouse can), crossed with carriers that serve the destination. Typically 5 warehouses × 6 carriers = 30 options.
2. Filter by hard rules (see precedence below): hazmat restrictions, delivery time window, carrier capacity left.
3. Score each remaining carrier by expected cost:

Here $p_{\text{on time}}$ is the probability the option arrives by the promise date (from the historical transit-time distribution for that lane), $\lambda$ is what the business says a late order costs (refunds, support contacts, churn), and the shadow price is the planner's estimate of what one more unit of a scarce resource is worth right now (a nearly full carrier at one warehouse gets a positive price, so orders drift elsewhere before the carrier runs out).
4. Reserve: atomically hold inventory and decrement carrier capacity. If either fails (someone else took the last unit), move to the next-best option. If all fail, fall back to the pre-agreed default (the most reliable carrier from the nearest warehouse with stock) and flag the order.

Worked example. Late penalty λ = $20. Three feasible options:

A wins although it is the most expensive to ship, because the lateness risk of C costs more than the $1.20 it saves. Now carrier X at East is at 90% of today's capacity and the planner publishes a shadow price of $0.80: A becomes 6.70 + 0.80 = 7.50 and B (7.20) wins. The planner has steered volume off X before X runs out and later, more urgent, orders find nothing.

Concretely, where that $0.80 comes from: the planner solves a linear (network-flow) optimisation over remaining capacity and forecast demand, and a shadow price is that optimisation's dual value on one resource's capacity constraint, i.e., how much total network cost would fall with one more unit of it. If committing carrier X's last slot to this order means a later order on the same lane must fall back to a standby carrier at $6.90 instead of X's own $6.10 base rate, that $6.90 - $6.10 = $0.80 gap is exactly the shadow price the planner publishes for X, refreshed every planning cycle as forecast demand and remaining capacity shift.

Why this meets the 95% SLA: the planner monitors the running on-time rate of committed decisions (the average of $p_{\text{on time}}$ over today's orders) and raises λ if it drifts toward 95%. A per-order floor (never choose an option with P(on time) below 0.85) prevents trading away an individual customer for a cheap label.

Why not solve every order jointly? A global optimization across the day's orders is better in theory but cannot run in 200 ms, and orders arrive one at a time. Shadow prices give the per-order decision most of the network-level information at a fraction of the cost. I would switch to micro-batching (collect orders for a few seconds and solve together) only for a fulfillment model where orders are not promised a date at checkout.

Business-rule precedence, rule representation and operator override

Rules are evaluated in a fixed precedence, from strongest to weakest:

1. Safety and legal (hazmat, i.e. hazardous materials that need special handling and a certified carrier; restricted destinations): filters, never overridable.
2. Time windows (the promised date, carrier cutoff times at the warehouse): filters.
3. Capacity (warehouse throughput, carrier slots): filters, with the planner's shadow price as a soft pressure before the hard limit.
4. Priority (premium customers, perishable goods): adjusts λ or restricts to faster services.
5. Cost: the objective among whatever survives.

Representation: rules are data, not code. Each rule is a row with a scope (SKU category, warehouse, destination region, customer tier), a kind (filter or score adjustment), a precedence level, an effective time range and a version. The decider loads them into an in-memory decision table indexed by scope, so evaluating 30 options against the relevant rules is a few lookups. A rule change is validated by replaying the last day's decision log against the new rule set (how many decisions would change, and at what cost and on-time impact) before it is activated.

Safe manual override: operators sometimes need "send everything from West via carrier Y today, carrier X has a strike". An override is a rule at its own precedence level, above priority and cost but below safety and legal, so it can never route hazmat by a forbidden carrier. Each override carries its author, reason, scope and a mandatory expiry; it is logged, shown on a dashboard with its measured cost impact, and can be put in shadow mode first (evaluated and logged but not applied). An override with no expiry is how a one-day workaround becomes a permanent, forgotten cost.

Caching strategy

• Rates and transit-time distributions: in-process memory in the decider, refreshed hourly; they change rarely.
• Rules: in-process, invalidated on change notification plus a 60 s maximum staleness.
• Inventory: not cached for the decision. Available-to-promise counts (the stock not yet committed to any order, i.e. what can still be promised to a new customer) are cached for display on product pages, but the reservation always goes to the source of truth, because a stale cache there creates oversell.
• Carrier capacity: authoritative counters in memory, since the carrier API is too slow to be the source of truth at decision time; a reconciliation job compares counters with carrier-reported bookings every few minutes and corrects drift.

Latency budget (200 ms at the 99th percentile)

Retries, fallbacks and scaling

• Retries: a reservation conflict retries with the next option, at most twice; any dependency timeout sends the order to the default option and flags it for review, rather than holding the order. The 200 ms budget above is sized for the common case of at most one retry (its "Retry on reservation conflict" row); a second retry only happens when the first two candidate options are both taken in the same instant, which is rare, and is allowed to run past the p99 target rather than fail the order, since a slightly late routing decision still beats none.
• Idempotency: decisions are keyed by order ID, so a retried request returns the stored decision instead of reserving twice.
• Reservations expire (for example 30 minutes) if the warehouse never confirms the pick, so crashed flows release stock.
• Scaling: the decider is stateless and scales horizontally; inventory is partitioned by warehouse; capacity counters by carrier and warehouse. At about 139 orders/s the hot rows (the database rows many transactions try to update at once, and so the ones most likely to queue behind each other) are the best-selling SKUs at one warehouse, so reservations there use per-SKU row locks (the database blocks other writers from touching that same row while one reservation is mid-update, so two orders cannot claim the same unit of stock) with short transactions.

Success metrics

• On-time rate (target at least 95%) and the gap between predicted and realized on-time probability (calibration).
• Cost per order and split-shipment rate.
• Decision latency at the median and the 99th percentile (p50/p99), fallback rate, reservation conflict rate.
• Override count, age and measured cost impact.

Pitfalls

• Calling carrier APIs synchronously: their latency and outages become yours.
• Optimizing only cost: the cheapest option is often the one that breaks the SLA.
• Rules in code: every rule change becomes a deploy, and nobody can audit which rule fired.

Direct answer

Build one dispatch platform with two policies selected per zone by live density. In dense zones, collect orders for a short window (tied to when the food or parcel will actually be ready) and solve a batched assignment that may stack two or three orders on one courier. In sparse zones, dispatch each order immediately to the best courier, because a waiting window would add latency and almost never find a second order to pair. A density controller with hysteresis picks the mode, and both modes commit through the same single-assignment guard, so a zone switching modes mid-peak can never give one courier two conflicting jobs.

Terms used below

• Batching (order stacking): one courier carries several orders in one trip.
• Zone: a small geographic area used as the unit of decision, typically a set of cells in a spatial grid index such as H3 (Uber's hexagonal hierarchical grid).
• ETA (estimated time of arrival): here, mostly how long a courier needs to reach a merchant or a customer.
• Hysteresis: different thresholds for switching on and switching off, so a value hovering near one threshold does not flip the mode every minute.
• Lease / fencing token: a lease is a time-limited ownership grant from a coordination store (etcd or ZooKeeper, both strongly consistent key-value stores for cluster coordination); the fencing token is a number that increases with every new lease, so a write from an old owner that still thinks it holds the lease can be rejected.

Requirements and scale (assumptions stated, arithmetic shown)

The last two rows are the whole argument for density-adaptive dispatch: a 30 s window in the dense zone gathers about 5 orders to combine, while in the sparse zone the same window usually holds zero or one order and is pure added latency.

Latency targets: order-ready to courier offer within the batching window + 2 s in dense mode; within 3 s in sparse mode; courier location no older than 10 s when used for matching.

Components

flowchart LR
  CA[Customer app] --> OS[Order service]
  KA[Courier app] --> LS[Location service]
  OS --> BUS[Event bus]
  LS --> BUS
  BUS --> DC[Density controller]
  BUS --> ZC[Zone dispatch coordinator]
  DC -->|mode per zone| ZC
  ZC --> RT[Routing and ETA service]
  ZC --> AS[(Assignment store)]
  AS --> OF[Offer service]
  OF --> KA

The event bus is a shared pipe: order and location updates are published to it once, and the density controller and the zone coordinator each read it independently, without polling each other or the originating service.

APIs

Data flow

1. The order is created; the order service estimates ready time and publishes OrderCreated with its zone.
2. The zone's coordinator adds it to the pending pool.
3. Single mode: immediately score idle couriers by ETA to the merchant against the ready time, pick one, and write the assignment.
   Batch mode: hold the order until the window closes (default 30 s, and never beyond "ready time minus courier travel time"), then solve over all pending orders and couriers in the zone.
4. The assignment write is conditional (below). On success the offer service pushes the offer; on decline or timeout the order returns to the pool.

Batch-mode solver

• Bundle generation: candidate pairs and triples of pending orders whose pickups are within 500 m (or at the same merchant), whose ready times are within 5 min of each other, and whose drop-offs lie roughly in the same direction.
• Bundle scoring: courier minutes to complete the bundle plus a lateness penalty for each customer against their promised time. Reject any bundle that makes a customer later than their promise, exceeds 3 orders, or exceeds the courier's carrying capacity.
• Assignment: choose a non-overlapping set of bundles and match them to couriers at minimum total cost, a set-packing problem (picking the best non-overlapping combination out of every possible bundle, a choice that grows combinatorially with the number of candidates). Exact set-packing is expensive, so the practical recipe is greedy selection of the bundles with the biggest courier-minute saving, then an optimal bundle-to-courier match with the Hungarian algorithm (a classic polynomial-time method for minimum-cost one-to-one assignment). Give the solver a 1 s budget and fall back to greedy if it runs over.

Density controller

Metric per zone: pending orders per idle courier, and orders per square kilometre over the last 10 min. Enter BATCH at 1.2 pending orders per idle courier or more; drop back to SINGLE only below 0.8. The gap between 1.2 and 0.8 is the hysteresis that stops flapping. The second measure, order density, is a gate on top of that ratio, not a second threshold to cross on its own: a courier-scarce but geographically sprawling zone (say two orders 6 km apart and one idle courier) can still show a ratio above 1.2, yet there is no nearby second order to stack with, so BATCH only actually engages when density is also at least a handful of orders per km² in the same 10-minute window; below that, the coordinator stays in SINGLE even if the courier ratio alone says otherwise.

Single-assignment guard

Every assignment, from either mode, is one transaction of conditional writes (:name marks a bound parameter):

UPDATE courier_state
   SET status = 'assigned', order_ids = :orders, version = version + 1
 WHERE courier_id = :courier AND version = :read_version
   AND status IN ('idle', 'can_stack')
   AND :fencing_token >= (SELECT token FROM zone_lease WHERE zone_id = :zone);
-- plus, for each order in the bundle:
UPDATE orders SET status = 'assigned', courier_id = :courier
 WHERE order_id = :order AND status = 'pending';

Each condition in the WHERE clause blocks one specific race on its own: version = :read_version fails the write if anyone else changed this courier since the coordinator last read it; status IN ('idle', 'can_stack') fails it if the courier was fully assigned elsewhere in between; and the fencing-token subquery fails it if this coordinator's own zone lease has gone stale, i.e. a newer coordinator has since taken over. All three must hold at once, or the whole statement updates zero rows and the transaction rolls back: someone else got there first, and the coordinator re-plans. This is what protects the zone boundary, where two coordinators can both see the same courier.

Worked example: what stacking buys and costs

Two orders from the same restaurant, drop-offs 1 km apart, courier 8 min from the restaurant, 12 min restaurant-to-first-drop, 4 min from first to second drop (assume a dedicated courier sent straight to the second drop would also take about 12 min from the restaurant, since both drops sit in the same direction from it):

• Single dispatch: two couriers × (8 + 12) = 40 courier-minutes.
• Stacked: 8 + 12 + 4 = 24 courier-minutes, a saving of 16 minutes (40%).
• Cost: the second customer receives food 4 min later than with a dedicated courier. The bundle is allowed only if that stays inside their promise.

For the business side, this is the lever that moves orders per courier-hour and cost per delivery; the constraint is delivery time for the second customer.

Operational constraints and failure modes

Trade-offs and pitfalls

• Window length is a product decision. Longer windows find better bundles and delay everyone. Anchoring the window to ready time is the key idea: if food takes 15 min, a 30 s window costs nothing visible to the customer.
• What would flip the design: where merchants always prepare for 10 minutes or more, dispatch everywhere becomes "assign at ready time minus courier ETA", and the dense/sparse split reduces to whether stacking is permitted in that zone.
• Pitfall: batching optimised for courier-minutes alone. It quietly degrades the second customer; put their promise in the constraints, not only in the cost.
• Pitfall: two dispatch code paths with two sources of truth. Keep one assignment store and one conditional write for both modes.
• Pitfall: zones as hard walls. Orders on a boundary get worse couriers than the ones 200 m away in the next zone; let candidate search cross zones while ownership stays per zone.