Marketplace, Dispatch, and Logistics System Design Questions

Direct answer

I would design the routing engine as a ladder of progressively simpler answers, each rung precomputed and stored where the next failure cannot reach it: live traffic routing, then routing on historical time-of-week speeds, then a precomputed cell-to-cell travel-time table (the metro divided into small map patches called cells; a lookup table storing the approximate travel time between every pair of them), then an on-device estimate. Every response carries a quality tier and data age so clients can widen ETA (estimated time of arrival) ranges and change their wording, and a small, separately replicated closure list is enforced at every rung so a degraded answer can be imprecise but never route someone into a closed road. Degraded modes are proven by regular failure drills and by continuously checking degraded answers against live ones while everything is healthy.

Requirements I am designing to

• Keep serving routes and ETAs for pickups, dispatch and in-trip navigation during (a) loss of live traffic feeds, (b) loss of the upstream map or tile service, (c) a network partition between regions (regions that can each still run fine on their own but can no longer talk to each other), and (d) a whole cloud region going down.
• A degraded answer may be less accurate, but must be safe (no closed roads, no impossible ETAs) and honest (the client knows it is degraded).
• Recover cleanly: when the failure heals, trips that started under degraded data are corrected without jarring jumps.

The fallback ladder

flowchart TD
  L0[Tier 0: live traffic routing] -->|traffic feed stale| L1[Tier 1: historical time-of-week speeds]
  L1 -->|routing cluster unavailable in region| L2[Tier 2: other region, replicated graph]
  L2 -->|graph search unavailable| L3[Tier 3: precomputed cell-to-cell table]
  L3 -->|server unreachable| L4[Tier 4: on-device estimate]
  CL[Closure list, replicated separately] -.enforced at.-> L1
  CL -.enforced at.-> L2
  CL -.enforced at.-> L3

Sizing the precomputed data (so it can live everywhere)

Use H3 (Uber's hierarchical hexagonal grid) resolution-8 cells (resolution is H3's zoom level; a higher resolution number means smaller cells, and 8 is a mid-level choice), which average about 0.737 km². For a 1,500 km² metro:

$\frac{1500}{0.737} \approx 2035 \text{ cells}, \qquad 2035^2 \approx 4.14 \times 10^6 \text{ pairs}$

Storing one travel time as a 2-byte integer (seconds, up to 65,535) for each of the 168 hours in a week:

$4.14\times10^6 \times 168 \times 2 \text{ B} \approx 1.39 \text{ GB}$

Historical edge profiles for a graph with 2 million edges at the same 168 × 2 bytes cost 672 MB. Both fit in memory on a single routing node, so every region holds every city's Tier 1 and Tier 3 data, and a region failure never takes the fallback data with it. Only live traffic, which changes every minute or two and is regenerated anyway, is region-local.

Cache priming

• Before a known risky window (a big event, a planned failover drill), warm the route cache in the standby region with the city's most frequent origin-destination pairs from the same hour last week.
• Keep the standby region's routing processes running with the graph already loaded; loading a large graph from disk and building its indexes can take long enough to matter, so failover must not start with a cold load.
• Ship a compact city pack to phones (historical cell speeds, the active trip's route polyline, the sequence of coordinates that draws the route on a map, and segment times) at trip start, so Tier 4 has real data rather than a guess.

Staleness handling

• Every data input carries a generation timestamp. Each tier has a maximum age: live speeds older than, say, 10 minutes are no longer "live" and the engine drops to Tier 1 for the affected area rather than serving rush-hour speeds at noon.
• Staleness is per area, not global: if one city's feed dies, only that city degrades.
• Blend on the way down and the way back up: when live data for a segment is thin, weight it against the historical profile instead of flipping abruptly, so ETAs do not jump.

Client signaling

Each response includes quality_tier, data_age_seconds and an ETA range, not just a point. The client then:

• Shows "about 12-18 min" instead of "14 min" for Tier 3 and below.
• Suppresses precise countdowns and the "driver is 1 minute away" push.
• Keeps showing the last good route with a subtle "updating" state rather than a blank map.
• Pricing and dispatch read the same tier: pricing can hold the last upfront quote rather than quoting from Tier 3 numbers, and dispatch can widen its matching radius because its ETA ranking is less precise.

Road closures: the one thing a degraded answer must never get wrong

Closure signals (from city feeds, operations staff, or detection such as "no probe vehicles, cars or phones that report GPS location back to the platform and act as a live traffic sensor, have crossed this bridge in 30 minutes while traffic around it is normal") are published as a small event stream fanned out (copied and pushed to every consumer at once) to every region and every tier. The closure list for a city is kilobytes, so it is cheap to replicate aggressively. On a new closure:

1. All tiers mark the edges as impassable immediately.
2. Cached routes are invalidated by keeping an index from edge to cached routes that use it, or more simply by bumping a version on the tiles (fixed map grid squares that group nearby edges and the cached routes through them) containing the edge so every cached route through them misses.
3. Active trips whose remaining route crosses the closure are pushed a re-route.

Tier 3 has no path, so it cannot "avoid" a closure; it must instead add a penalty to cell pairs whose shortest path crossed the closed edge (recomputed offline in the background) and label those answers as low confidence.

Partitions: degrade, then reconcile after heal

During a partition a region may serve its cities from Tier 1 while cut off from the live-traffic producer. After the heal:

• Recompute ETAs for trips still in progress with live data, and move the displayed ETA towards the new value over a minute or two rather than in one jump.
• Trips completed under degraded data are flagged; any fare that depended on a degraded duration is checked against the upfront quote rule so riders are not charged for our estimate error.
• Replay the closure events missed during the partition before re-entering Tier 0.

Runbooks for an upstream map-service failure

1. Confirm scope: which tiles or areas return errors or stale versions.
2. Pin routing to the last known-good map version (map data is versioned, so this is a config switch, not a rebuild).
3. Freeze map-data ingestion so a partially updated graph is not published.
4. Post the degraded status to operations and to the in-app banner configuration.
5. After recovery, diff the new map version against the pinned one before unpinning, and re-run the safety checks below.

Verifying that degraded answers are safe

• Invariant checks on every degraded answer: the route is connected, uses no closed or wrong-way edges, and its ETA is at least distance divided by a plausible top speed for the road class.
• Continuous shadow comparison: while healthy, compute Tier 1 and Tier 3 answers for a sample of live requests and record their error against Tier 0 and against actual arrival. This gives a measured error band per tier, per city and per hour, which is what the client's displayed range should be based on.
• Game days: deliberately cut the traffic feed or evacuate a region in production during low traffic, and verify that tiers switch, clients show degraded wording, and nothing routes through a test closure.

Trade-offs and pitfalls

• Cost of full replication: holding every city's fallback data in every region costs memory, but at roughly 2 GB per large metro it is small next to the cost of a city going dark.
• Silent degradation is the worst outcome. A system that falls back without telling clients shows confidently wrong ETAs; always propagate the tier.
• Flapping: without hysteresis (requiring the live feed to be healthy for several minutes before returning to Tier 0), a feed that recovers and fails repeatedly makes ETAs oscillate.
• Correlated failure: if Tier 1 data is generated by the same pipeline that feeds Tier 0, a bad pipeline release corrupts both. Generate profiles from a separate job and keep the last known-good copy.

Direct answer

Keep third-party traffic providers off the request path. A background ingestion service pulls (or receives) provider data, normalises it onto our own road segments, stamps every value with its observation time, and writes it into a fast "current traffic" table. The ETA (estimated time of arrival) service only ever reads that table, so a slow or dead provider makes ETAs a little less accurate, never slower or failing. Prefer streaming when a provider offers it, poll on a schedule that fits inside the rate limit when it does not, attach a confidence score to each source that decays with age, and fall back in a fixed order: other live provider, our own fleet's recent speeds, then historical speeds for that time of day.

The pipeline

flowchart LR
  P1[Provider A stream] --> ING[Ingest adapters]
  P2[Provider B polled API] --> ING
  FLEET[Own fleet speeds] --> ING
  ING --> NORM[Map to our road segments]
  NORM --> FUSE[Freshness and confidence scoring]
  HIST[Historical speed profiles] --> FUSE
  FUSE --> TBL[(Current traffic table)]
  TBL --> ETA[ETA service]

One adapter per provider isolates that provider's format, auth and quirks. Normalisation matters because each provider describes roads in its own way (its own segment IDs, or map tiles, or location references); mapping everything onto our segment IDs once, at ingest, means the rest of the system never sees provider formats.

Polling versus streaming

Choice: stream when offered, with a heartbeat check (no message for, say, 60 s means treat the feed as down and reconnect). Poll otherwise, and spend the rate limit where it matters.

Worked example: fitting polling inside a rate limit. A provider allows 10 requests per second and returns one map tile per request. The city is covered by 400 tiles.

• Uniform refresh every 60 s: 400 / 60 = 6.7 requests per second, inside the limit but with little headroom for retries.
• Tiered refresh: the 100 tiles where most active trips are refresh every 30 s (100 / 30 = 3.3 per second) and the other 300 every 120 s (300 / 120 = 2.5 per second), for 5.8 per second in total. Busy areas get fresher data and the quota keeps about 40% headroom for retries and bursts.

Enforce the budget client-side with a token bucket (a counter that refills at the allowed rate and is spent one token per call), and honour the provider's own throttling responses (typically HTTP 429 "too many requests", often with a Retry-After header) by backing off rather than retrying immediately.

Freshness and confidence

Every value carries the time the provider observed it, not the time we received it. A reading's confidence decays with age: for example full weight when under 2 minutes old, linearly down to zero at 10 minutes, and ignored after that. Also score each provider's accuracy over time by comparing its speeds against our own drivers' actual travel times on the same segments later, so a provider that is fresh but systematically wrong is trusted less.

Two providers that disagree. Say provider A says a segment moves at 15 km/h and B says 45 km/h. Do not average blindly (30 km/h is what neither observed). Prefer the one with the higher confidence score (fresher and historically more accurate); if our own fleet has recent traversals of that segment, use them as a tie-breaker; if they still disagree and neither is clearly better, use the slower value for dispatch decisions (under-promising is cheaper than a missed pickup) and widen the ETA range shown to the rider.

Protecting against noisy or stale data from a single provider. Validate each value on ingest: reject speeds above the road's plausible maximum, reject values older than the previous one for that segment (out-of-order data), and flag a provider whose readings do not change at all for an unusually long time (a frozen feed looks fresh by timestamp but is not). Clamp how fast a segment's speed can change between consecutive updates unless several sources agree, which stops one glitchy reading from rerouting a whole city.

Fallback when a provider is slow or down

1. Timeouts and circuit breakers on the ingest side: each call has a strict timeout (say 2 s); after repeated failures the circuit opens (stop calling for a cool-off period, then probe with a single call), so a sick provider does not tie up workers.
2. Last good value, with decaying confidence: the table keeps serving the last reading, and its falling confidence shifts weight to other sources automatically.
3. Fallback order per segment: another live provider, then our own fleet's speeds from the last few minutes, then the historical profile for that segment, day of week and time of day.
4. Tell downstream about degraded mode: the ETA response carries a confidence flag, so the app can show a range ("12 to 18 min") instead of a precise number, and dispatch can add margin.

Cache provider responses at ingest with a short time-to-live (TTL) tied to their freshness window, so retries and multiple consumers never re-spend quota for data we already have.

Pitfalls

• Calling a provider inside the ETA request: the provider's slowest response becomes your p99 (99th-percentile latency), and its outage becomes yours.
• Using receive time instead of observation time, which makes a delayed batch look fresh.
• Silent quota exhaustion: alert on the rate-limit headroom, not only on errors.
• Treating historical fallback as good as live data during incidents or events; always surface lower confidence.

Direct answer

The first question is whether trips are actually taking longer or whether only our predictions went up. If actual trip durations rose too, something happened on the roads (an accident, a storm, an event) and the ETA (estimated time of arrival) system is probably right. If actual durations are flat while predictions jumped, our system is wrong, and I check the usual suspects in order of how often they cause sudden, single-city jumps: a recent deployment or config change, stale or broken traffic data, a drop in driver telemetry, and routing engine errors or fallbacks. I mitigate by rolling back or switching the city to a known-good data source first and finishing the diagnosis afterwards, and I capture evidence as I go so the postmortem does not depend on memory.

Triage flow

flowchart TD
  A[ETA spike in one city] --> B{Actual trip durations up too?}
  B -->|Yes| C[Real-world event: confirm, widen ranges, no rollback]
  B -->|No| D{Deploy or config change in the last hours?}
  D -->|Yes| E[Roll back, compare replayed ETAs]
  D -->|No| F{Traffic data age above normal?}
  F -->|Yes| G[Switch city to historical speeds, fix feed]
  F -->|No| H{Telemetry ingestion rate dropped?}
  H -->|Yes| I[Check ingestion pipeline and app release]
  H -->|No| J[Routing engine errors, fallbacks, map version]

What to inspect, and what each finding means

Worked example: the stale traffic table

At 11:00 the ETA for a typical 6 km cross-town trip in one city is 20 minutes instead of the usual 12. Completed trips are still taking about 12 minutes, so predictions are wrong. No deploy happened. The traffic-data age panel shows the city's speed table was last updated at 08:55.

That explains the number exactly. The table froze during rush hour, when this corridor averaged 18 km/h. At midday it normally runs at 30 km/h:

$\frac{6 \text{ km}}{18 \text{ km/h}} \times 60 = 20 \text{ min}, \qquad \frac{6 \text{ km}}{30 \text{ km/h}} \times 60 = 12 \text{ min}$

Every route in the city is being computed at 8:55 a.m. congestion levels, two hours after the congestion cleared. Only this city is affected because each city's table is built by its own job, and that job has been failing silently (for example, an upstream file format change that its parser rejects).

Immediate mitigation (minutes, not hours)

1. Switch the city to the fallback speed source: historical average speed per road segment for this hour of the week. This brings ETAs back to roughly normal immediately.
2. If a deploy is implicated, roll it back rather than fixing forward (pushing a new patch on top of the broken change instead of reverting it) during the incident.
3. Widen displayed ETA ranges and pause "arriving in 1 minute" notifications for the city while quality is uncertain.
4. Protect downstream decisions: surge pricing (the temporary price multiplier that rises when nearby demand outstrips available drivers, meant to pull more drivers into the area) and dispatch both read ETAs, because "how far away is the nearest driver" is itself computed from predicted travel time. Inflated ETAs can make dispatch think all drivers are far away and push surge up; freeze surge at its current cap (stop the multiplier from climbing any further, but leave it wherever it already is) or switch it to a supply-based signal only (compute the demand-supply imbalance from a raw count, open ride requests versus available drivers in the area, that does not depend on ETA at all), until ETAs are fixed.
5. Then fix the root cause (restart or patch the feed job) and let the city return to live speeds once data is fresh for several consecutive refreshes.

Automated remediation, so next time it fixes itself

• Freshness watchdog per city. If the table normally refreshes every 2 minutes, page at 10 minutes of age (five missed refreshes) and automatically fall back to historical speeds at 15 minutes. An hour-plus stale table would then never be served.
• Prediction-vs-reality guard. Continuously compare the median ratio of predicted to actual duration for trips completed in the last 30 minutes. If it moves beyond a set band (for example 20% above normal for 15 minutes with enough trips), trigger the fallback and page.
• Automated rollback triggers. Any ETA model, feature or map release is rolled out city by city with the same guard; tripping it reverts the release automatically.

Capturing evidence for the postmortem

• Start an incident timeline immediately: first alert, each check, each action, with timestamps.
• Snapshot the dashboards (freshness, ingestion rate, latency, fallback share, predicted vs actual) covering an hour before onset through recovery; dashboard data may be downsampled (older points get averaged into coarser buckets to save storage, so minute-by-minute detail is lost) or expire.
• Save a sample of affected requests with their full inputs: route, traffic snapshot ID or timestamp, model and map versions, output. These let you replay the bad ETAs later.
• Preserve the failing job's logs and the offending input file before anyone retries or cleans up.
• Record the impact: number of trips shown inflated ETAs, cancellation rate vs baseline, any pricing effect.
• In the postmortem, the key question is not "why did the job fail" but "why could a stale table be served for two hours": the missing freshness check is the real defect.

Trade-offs and pitfalls

• Rolling back real events. If actual trip times are up (a storm), "fixing" the ETA back down makes it wrong. Always check actuals first.
• Over-sensitive guards. An automatic fallback that triggers on every small wobble swaps good live data for coarser history too often; require sustained deviation and minimum trip counts.
• Fixing only the ETA. Check what else consumed the bad numbers (pricing, dispatch, promised delivery windows) during the incident and whether any customers need remediation.

Direct answer

To find the earliest arrival time (the basis of a routing ETA, or estimated time of arrival), run time-dependent Dijkstra: the same algorithm as ordinary Dijkstra, except that the label on a node (the current best value the algorithm has found for it) is the earliest arrival time, and an edge's travel time is evaluated at the moment you leave its tail node (the node the directed edge starts from; every edge runs from a tail node to a head node). This is correct when every edge satisfies the FIFO property (first in, first out: departing later never makes you arrive earlier), because then arriving at a node as early as possible is always best for everything after it. Complexity stays O((V + E) log V) priority-queue work, times the cost of evaluating one travel-time function.

Approach

• Model: each edge (u, v) has a function f_uv(t) giving travel time when departing u at time t. I use piecewise-linear functions (breakpoints of departure time and travel time, interpolated between them), which is how historical speed profiles are usually stored.
• FIFO condition: arrival time t + f(t) must be non-decreasing in t: if you leave one minute later you must not arrive earlier, so travel time f(t) can fall by at most one minute for every extra minute you wait, which for a piecewise-linear function means every segment's slope is at least -1.
• Why FIFO makes the greedy step safe: if you reach u at time t1 and t2 > t1, then leaving at t1 reaches every neighbour no later than leaving at t2. So the earliest arrival at u dominates any later one, exactly the property Dijkstra's "settle the smallest label" step relies on.
• Algorithm: priority queue keyed by arrival time; pop the earliest node; for each edge, candidate arrival = t + f(t); keep it if it improves the neighbour. Record the parent and the departure time on each improvement so the path, with a timestamp per leg, can be rebuilt, and printed as simple turn-by-turn instructions.

Code

import bisect, heapq, math

class TravelTime:
    """Piecewise-linear travel time (minutes) as a function of departure time
    (minutes after midnight). Constant before the first and after the last point."""
    def __init__(self, points):
        self.ts = [t for t, _ in points]
        self.ws = [w for _, w in points]

    def __call__(self, t):
        i = bisect.bisect_right(self.ts, t)
        if i == 0: return self.ws[0]
        if i == len(self.ts): return self.ws[-1]
        t0, t1, w0, w1 = self.ts[i-1], self.ts[i], self.ws[i-1], self.ws[i]
        return w0 + (w1 - w0) * (t - t0) / (t1 - t0)

    def is_fifo(self):
        # Arrival t + w(t) must never decrease: every slope dw/dt >= -1.
        return all((self.ws[i+1] - self.ws[i]) / (self.ts[i+1] - self.ts[i]) >= -1
                   for i in range(len(self.ts) - 1))

def earliest_arrival(graph, src, dst, depart):
    """Time-dependent Dijkstra. graph[u] = [(v, TravelTime, road_name)].
    Correct when every edge is FIFO (leaving later never arrives earlier)."""
    arrive = {src: depart}; parent = {}; done = set()
    pq = [(depart, src)]
    while pq:
        t, u = heapq.heappop(pq)
        if u in done: continue
        done.add(u)
        if u == dst: break
        for v, f, road in graph.get(u, []):
            tv = t + f(t)                      # evaluate weight at the time we LEAVE u
            if tv < arrive.get(v, math.inf):
                arrive[v] = tv; parent[v] = (u, road, t)
                heapq.heappush(pq, (tv, v))
    if dst not in done: return None
    legs, v = [], dst
    while v != src:
        u, road, t_leave = parent[v]
        legs.append((u, v, road, t_leave, arrive[v])); v = u
    return arrive[dst], legs[::-1]

def hhmm(m): return f"{int(m // 60):02d}:{int(round(m % 60)):02d}"

# Highway H is fast off-peak and jams 07:45-08:45; side streets are steady.
rush = TravelTime([(7*60+30, 12), (8*60, 40), (8*60+30, 40), (9*60, 12)])
graph = {
    "Depot":    [("H-on", TravelTime([(0, 4)]), "Main St"),
                 ("Elm/3rd", TravelTime([(0, 10)]), "Elm St")],
    "H-on":     [("H-off", rush, "Highway 101")],
    "Elm/3rd":  [("Oak/9th", TravelTime([(0, 14)]), "3rd Ave"),
                 ("H-off", TravelTime([(0, 30)]), "Bay Rd")],
    "Oak/9th":  [("H-off", TravelTime([(0, 8)]), "Oak St")],
    "H-off":    [("Customer", TravelTime([(0, 3)]), "Pine St")],
}
for depart in (7*60, 8*60):
    eta, legs = earliest_arrival(graph, "Depot", "Customer", depart)
    print(f"depart {hhmm(depart)} -> arrive {hhmm(eta)} ({eta - depart:.0f} min)")
    for u, v, road, t0, t1 in legs:
        print(f"   {hhmm(t0)} take {road} from {u} to {v}, arrive {hhmm(t1)}")

print("rush edge FIFO:", rush.is_fifo())
bad = TravelTime([(480, 40), (485, 10)])   # leave 5 min later, save 30 min
print("bad edge FIFO:", bad.is_fifo(), "| leave 08:00 ->", hhmm(480 + bad(480)),
      "| leave 08:05 ->", hhmm(485 + bad(485)))

Output:

depart 07:00 -> arrive 07:19 (19 min)
   07:00 take Main St from Depot to H-on, arrive 07:04
   07:04 take Highway 101 from H-on to H-off, arrive 07:16
   07:16 take Pine St from H-off to Customer, arrive 07:19
depart 08:00 -> arrive 08:35 (35 min)
   08:00 take Elm St from Depot to Elm/3rd, arrive 08:10
   08:10 take 3rd Ave from Elm/3rd to Oak/9th, arrive 08:24
   08:24 take Oak St from Oak/9th to H-off, arrive 08:32
   08:32 take Pine St from H-off to Customer, arrive 08:35
rush edge FIFO: True
bad edge FIFO: False | leave 08:00 -> 08:40 | leave 08:05 -> 08:15

Reading the result: at 07:00 the highway route takes 19 minutes. At 08:00 the highway is jammed (40 minutes on that edge alone, so 4 + 40 + 3 = 47 minutes), and the algorithm correctly picks the side streets for 35 minutes. The last two lines show a non-FIFO edge: leaving at 08:05 arrives at 08:15, but leaving at 08:00 arrives at 08:40, which is exactly the situation where settling the earliest arrival can be wrong.

Key points

• The weight is evaluated at departure time from u, which is the arrival label of u. Evaluating it at the trip's start time for every edge is the most common bug: it silently turns the problem back into a static one.
• Earliest arrival is a one-to-one or one-to-all query for a fixed departure time. "What is the best time to leave?" is a different problem: a profile query computes arrival as a function of departure time, so instead of one number the output is a small curve, for example "leave 07:00 to arrive 07:19, leave 07:15 to arrive 07:31, leave 07:30 to arrive 07:52", and it costs far more.
• A* extension (goal-directed search with changing weights): the heuristic must be a lower bound at every time of day, so compute it from each edge's minimum travel time over the day, never from the current one.
• If waiting at a node is allowed, a non-FIFO edge can be made FIFO by replacing f(t) with the best of "leave now or wait and leave later": for the bad edge above, instead of the raw values (leave 08:00, arrive 08:40; leave 08:05, arrive 08:15), the repaired function would return an 08:15 arrival for any departure at or before 08:05, since waiting until 08:05 is always at least as good; without waiting, the problem becomes NP-hard in general (Orda and Rom proved this in 1990, for travel-time functions with integer or rational breakpoints): with no waiting allowed and no FIFO guarantee, the earliest arrival at a node no longer dominates every later arrival there, because a later, worse arrival can unlock a much faster non-FIFO edge further on. A correct search has to keep every non-dominated arrival-time label at a node instead of settling on the smallest one, and the number of such labels can grow combinatorially with the length of the path, not run one clean settle-and-move-on pass like ordinary Dijkstra.

Complexity

• With a binary heap: O((V + E) log V) heap operations, as in Dijkstra. Lazy deletion (skipping already-settled pops) allows up to E heap entries, which is still O(E log V).
• Each edge evaluation is a binary search over its k breakpoints, O(log k), so the total is O(V log V + E(log V + log k)).
• Memory: O(V) for labels and parents, plus the graph with O(E k) breakpoints.

Edge cases

• Destination unreachable: returns None.
• Source equals destination: returns the departure time and an empty leg list.
• Departure outside the profile range: functions are held constant before the first and after the last breakpoint; for a daily profile you would wrap time modulo 24 h instead.
• Non-FIFO data: validate every edge at load (is_fifo) and either repair the profile (cap the downward slope at -1) or model waiting explicitly.
• Ties and floating-point arrival times: compare with a small tolerance if functions come from noisy data, or round to seconds.
• Very long trips across midnight or time zones: use absolute UTC times throughout and convert to each edge's local time only when looking up its profile.

Direct answer

From an SRE (site reliability engineering) seat, the graph-based system is a deterministic function of versioned data (map build, traffic snapshot, code), so it is reproducible, easy to roll back, and it fails loudly. An ML ETA (estimated time of arrival) predictor is a function of a model plus a live feature pipeline plus training data, so it can be fully "up" while wrong, its failures are silent and statistical, and reproducing a prediction requires logging inputs you would never log for the graph. I would run ML only as a bounded correction on top of the graph ETA with automatic fallback, and treat accuracy as a first-class SLI (service-level indicator: a measured signal of service health) alongside latency and availability.

Operational complexity

More owners and more independently changing artifacts means more ways for a change in one place to break another without anyone deploying the service itself.

Observability

• Graph system: standard golden signals (rate, errors, latency, saturation) plus input freshness (traffic metric age, map build version per region). A wrong ETA can almost always be traced to one input: a closed road, a stale speed.
• ML system: all of the above, plus
  • Feature freshness and null rate per feature: a feature silently defaulting to zero is the classic outage.
  • Prediction distribution drift: compare today's distribution of predicted ETAs, or of the residual, with a reference window (for example with PSI, the population stability index, which scores how much a distribution has shifted).
  • Training/serving skew: the same feature computed offline for training and online for serving should match; sample and compare.
  • Accuracy SLI against ground truth, which only arrives when the trip ends.

The ground-truth delay problem: if trips average 18 minutes, the accuracy signal for a bad model version lags by at least that long, plus aggregation time. Leading indicators (drift, freshness, residual distribution) are what page in the first minutes; accuracy confirms.

Rollback and reproducibility

• Graph: rollback is "load the previous map build or the previous traffic metric". Given the same versions and the same request, the answer is identical, so any logged ETA can be replayed exactly from (code version, map build, metric version, origin, destination, time).
• ML: rollback of the model is a pointer flip in a model registry (a catalog of trained model versions that tracks which one is currently live), which is fast, but rolling back the model does not fix a broken feature pipeline, which is the more common failure. Reproducing a prediction needs the exact feature vector (the specific list of numbers, such as time of day, area, recent speeds, that were fed into the model for that one prediction), since features are computed from live state that has since changed. So log, per prediction, the model version and either the full feature vector, or a cheaper hash of it (a short fingerprint that can later confirm whether the same inputs were used, even though a hash alone cannot be reversed to replay the prediction) plus a sampled full copy for a fraction of requests, since the sampled copy is what actually gets replayed.

Cost of that logging, computed: with roughly 25,000 trips concurrently active at peak, each re-scored about once per second as it progresses (an illustrative peak for this calculation, not a measured figure), that is about 25,000 predictions per second; at a 400-byte record, full logging is 25,000 x 400 B = 10 MB/s, which is 864 GB per day. Logging a 1% sample in full (8.64 GB per day) plus hashes and versions for everything is affordable; logging every full vector usually is not.

Failure modes

The design I would operate

1. Graph ETA is always computed and is the fallback.
2. ML predicts a residual, clamped to a fixed fraction of the graph ETA, with a hard deadline; on timeout or stale features, serve the graph ETA and count a fallback.
3. Model rollout: shadow mode (the new model scores every live request in parallel with the old one, but its answer is never shown to a user, only compared) for a day, then a canary on a few percent of traffic in a few cities, compared on accuracy after trips complete, then ramp. Automatic rollback triggers on leading indicators (drift, fallback rate) because accuracy is too slow.
4. SLOs (service-level objectives, the targets on those SLIs):
   • Availability: 99.9% of ETA requests succeed over 30 days, an error budget of 0.1% of requests, equivalent to 30 x 24 x 60 x 0.001 = 43.2 minutes of total outage per 30 days.
   • Latency: 99% under 150 ms.
   • Accuracy: for example, 90% of completed trips with absolute error within the larger of 2 minutes or 15% of trip time, measured daily per city.
5. Reproducibility contract: every served ETA carries the code version, map build, metric version and model version in the log.

What would change my view

If the ML model has proven stable for months, is retrained on a pinned, validated snapshot, and has feature pipelines with their own SLOs, it can take a larger share (predicting the full ETA for specific segments such as pickup time). Without that maturity, making ML the sole source moves the ETA service from failing loudly to failing silently, which is the worse trade for whoever is on call.