Airbnb Senior Solutions Architect Interview Preparation Guide

Direct answer

If the projects are independent and the money covers both, I would fund both, since each has a positive NPV. If instead the two are mutually exclusive (choosing one rules out the other, like two systems that do the same job) and capital is not scarce, I would fund Project A, because NPV (net present value, the value created in today's dollars after the required return) is the best single measure of value created: A creates $150,000 versus B's $110,000. I would fund B instead only if capital is rationed (a fixed budget cap, here $200,000, that cannot cover everything worthwhile) and the $80,000 left over can earn more than $0.50 of NPV per dollar elsewhere. B returns $0.92 of NPV per dollar invested against A's $0.75 and recovers its money faster, but that efficiency only pays if the saved money has a good use. Before deciding I would check what else the extra $80,000 could do, how reliable each forecast is, and whether the projects are really alternatives.

Why the metrics disagree

• NPV per dollar invested (the net profitability index; the plain profitability index is present value of inflows divided by outlay, which is 1.75 for A and 1.92 for B, and ranks them the same way): A = 150,000 / 200,000 = 0.75; B = 110,000 / 120,000 = 0.92. B is more efficient with capital.
• Payback (time to recover the outlay): B at 1.3 years is faster than A at 2.1 years. B has lower exposure and recovers cash sooner.
• Absolute NPV: A creates $40,000 more value, but it needs $80,000 more capital to do it.

The key comparison is the increment: choosing A over B means spending an extra $80,000 to gain an extra $40,000 of NPV, which is $0.50 of NPV per extra dollar. Whether that is worth it depends entirely on what $80,000 could do elsewhere.

The decision rule

At exactly $0.50 per dollar the leftover money ties A, so any better alternative makes B-plus-alternative the winner. That break-even is the number to put in front of the committee. Worked comparison with a $200,000 budget:

What else I would look at before deciding

1. Forecast reliability. A's NPV rests on more years of cash flows. If A's benefit is mostly in year 3 and B's is mostly in year 1, a 20% miss hurts A more.
2. Discount rate sensitivity. Re-run both at 6% and 12%. A project that is only better at one rate is a weaker choice.
3. Risk and reversibility. Can A be phased so the first $100,000 is the same as B's scope? Staged commitment lowers regret.
4. Dependencies and strategy. Does one unlock the other, or other work?
5. Delivery capacity. Do we have the people to run A without hurting B or other commitments?

Same logic on two vendor offers

Vendor 1 asks $100,000 now plus $30,000 a year for 5 years. Vendor 2 asks nothing now and $55,000 a year for 5 years. At a 10% discount rate (illustrative), cost is compared in present-value terms. The paid totals are $250,000 and $275,000, so Vendor 1 looks cheaper, but in present value Vendor 2 is cheaper ($208,493 versus $213,724), because Vendor 2 defers more of the money. A second case, where delivery speed matters as well as price: a fast vendor with an $80,000 licence that goes live in month 3 versus a slow vendor with a $50,000 licence that goes live in month 6, with $12,000 a month of benefit once live over 24 months at 10%. Because the licence fee and the benefit start dates are counted in months, the code converts the 10% yearly rate into the equivalent monthly rate, 1.10^(1/12) - 1 = 0.797% a month, which compounds back to exactly 10% over twelve months. Lost benefit during delivery goes into the monthly comparison, and the fast vendor wins (NPV $145,741 versus $141,142) even though it costs $30,000 more.

def npv(rate, flows):
    return sum(f / (1 + rate) ** t for t, f in enumerate(flows))

# NPV per dollar invested (net profitability index)
for name, outlay, v in (("A", 200_000, 150_000), ("B", 120_000, 110_000)):
    print(f"Project {name}: NPV per $ invested = {v / outlay:.2f}")
print(f"Extra NPV from choosing A over B: {150_000 - 110_000:,} USD for {200_000 - 120_000:,} USD more capital")

# Two vendor offers compared at 10% (costs, so lower is better)
v1 = [100_000] + [30_000] * 5      # 100k up front, 30k a year for 5 years
v2 = [0] + [55_000] * 5            # nothing up front, 55k a year for 5 years
print(f"Vendor 1: total paid {sum(v1):,}, PV cost {npv(0.10, v1):,.0f}")
print(f"Vendor 2: total paid {sum(v2):,}, PV cost {npv(0.10, v2):,.0f}")

# Delivery speed vs licence cost: monthly discounting, 24-month horizon, 12,000 USD/month benefit once live
monthly = 1.10 ** (1 / 12) - 1
def value(licence, live_month, per_month=12_000, horizon=24):
    return sum(per_month / (1 + monthly) ** m for m in range(live_month + 1, horizon + 1)) - licence
print(f"Fast vendor (80k licence, live month 3): NPV {value(80_000, 3):,.0f}")
print(f"Slow vendor (50k licence, live month 6): NPV {value(50_000, 6):,.0f}")

Output:

Project A: NPV per $ invested = 0.75
Project B: NPV per $ invested = 0.92
Extra NPV from choosing A over B: 40,000 USD for 80,000 USD more capital
Vendor 1: total paid 250,000, PV cost 213,724
Vendor 2: total paid 275,000, PV cost 208,493
Fast vendor (80k licence, live month 3): NPV 145,741
Slow vendor (50k licence, live month 6): NPV 141,142

Pitfalls

• Choosing by payback alone ignores everything after the money comes back.
• Choosing by profitability index alone can pick a small project and leave money unspent.
• The cheaper total price is not the cheaper deal once timing is counted.
• What would change my call: if A is the only project that builds a capability the business needs next year, or B's forecast is much less certain than A's.

Direct answer

Fund a mix that is chosen by value per unit of capacity, with two adjustments: anything required for compliance or a safe migration goes first, and anything that depends on being in the cloud goes after the move. I would quantify security by expected loss avoided and performance by revenue or cost gained, using ranges and stating assumptions.

Capacity: 3 teams x 26 weeks = 78 team-weeks; with a 15% buffer for surprises, about 66 are plannable.

Quantifying value (illustrative)

• Security risk = likelihood per year x impact if it happens. A fixable exposure with a 10% yearly chance of a $5M incident is worth $0.5M a year to remove.
• Performance value = revenue lifted (for example fewer abandoned sessions, meaning visitors who leave a slow page) or infrastructure cost saved.

Candidate work (value in $k per year, cost in team-weeks), with where each value comes from

Eight items total 102 weeks and $3,550k, more than 66 weeks.

What I fund: encryption is lowest by ratio but, in this illustration, I assume it is a compliance requirement (confirm with the client's auditors) and it is cheaper to design in before migration than to retrofit (add after the fact) later, so it is a gate (a must-pass requirement), not a candidate. Funded: identity (12), encryption (14), vulnerabilities (10), logging (10), database tuning (10), autoscaling (8) = 64 weeks, $2,450k of value. Deferred: caching and re-architecture, with a re-look once the cloud baseline is measured.

Order: months 1 to 3 identity, vulnerabilities, database tuning (all independent of the move); then encryption and logging as the landing zone (the pre-built, secured foundation in the cloud that workloads move into) is built; autoscaling after migration, when it can be tested.

Telling the client: show the table, the 2 deferred items and what each would add ($1,100k), and ask them to choose whether to add budget or accept the risk.

If encryption is not mandated: it competes on ratio (25.0, the lowest) and drops out, freeing 14 weeks. The caching layer (14 weeks, $400k) then fits: 50 + 14 = 64 weeks and $2,500k of value instead of $2,450k. That is why the compliance question is the first thing to confirm with the client.

Checking the ranking. Value per week is a heuristic, not an optimum. With the encryption gate in place, the funded set ($2,450k in 64 weeks) is the best combination of the eight items that fits 66 weeks. Without the gate, checking every combination finds a slightly better set than the ratio-ranked $2,500k: identity, vulnerabilities, database tuning, logging and the hot-path re-architecture, which is 66 weeks and $2,550k. I would still keep the ratio-ranked set, because the $50k gain is well inside the error of every estimate in the table and the re-architecture is a 24-week rebuild that would land before the migration. Also, the four security values all draw on the same $5M incident (12% + 10% + 6% + 7% = 35 points of yearly probability), and the fixes overlap, so treat their sum as an upper bound.

Direct answer: A CRDT (Conflict-free Replicated Data Type) is a data structure designed so that replicas can be updated independently, without coordinating with each other, and always converge to the same state once they've all seen the same set of updates, regardless of the order those updates arrived in. State-based CRDTs (CvRDTs) replicate by sending the WHOLE current state and merging it with a commutative, associative, idempotent merge function; operation-based CRDTs (CmRDTs) instead replicate the individual OPERATIONS, relying on the operations themselves being designed to commute (apply correctly regardless of order) when delivered at-least-once.

Structured elaboration

State-based (CvRDT). Each replica periodically sends its full local state to others; the receiving replica merges the incoming state with its own via a merge function that must be commutative (merge(A,B) == merge(B,A)), associative (merge(merge(A,B),C) == merge(A,merge(B,C))), and idempotent (merge(A,A) == A), these three properties together guarantee that no matter what order or how many times states are exchanged and merged, every replica converges to the same result. Simple to reason about (you don't need reliable, ordered delivery, since re-merging the same state twice is a no-op), but can be bandwidth-heavy if full state is large and changes frequently.

Operation-based (CmRDT). Each replica broadcasts individual OPERATIONS (e.g. "increment by 1", "add element X") rather than full state; correctness requires that concurrent operations commute (applying them in either order produces the same result) and typically that delivery is reliable (every replica eventually receives every operation, even if not in the same order), since a lost operation, unlike a lost state-sync, isn't automatically corrected by a later full-state exchange. More bandwidth-efficient for high-frequency small changes, but places a stronger requirement on the delivery layer.

Merge semantics, walked through with a concrete type. A G-Counter (grow-only counter, state-based) holds a vector of per-replica counts, e.g. replica A's local view {A: 3, B: 1} and replica B's local view {A: 2, B: 4} (B has incremented more locally, A's copy of B's count is stale). Merging: element-wise max, {A: max(3,2)=3, B: max(1,4)=4}. The TOTAL value is the sum of all entries, 7, and this merge is trivially commutative, associative, and idempotent (max has all three properties), so both replicas converge to the same {A:3, B:4} (total 7) regardless of how many times or in what order they exchange states.

An OR-Set (observed-remove set, supports add AND remove correctly) tags each added element with a unique ID; removing an element records the SPECIFIC IDs being removed (the ones the removing replica has observed), so a concurrent add of the "same" logical element (a new ID) from another replica isn't accidentally removed too, a naive set that just tracked "added" and "removed" element values (without unique IDs) would incorrectly let a remove suppress a concurrent, unrelated add of an equal value.

Two realistic use cases.

• A distributed like-counter that needs to keep incrementing correctly even while regions are partitioned from each other, a PN-Counter (positive-negative counter, supporting both increment and decrement) handles this without any coordination, each region increments its own local entry, and totals converge once partitions heal.
• A multi-device shopping cart or collaborative to-do list where items can be added from different devices while offline, an OR-Set lets each device add/remove items independently, converging correctly on reconnect without a central server arbitrating conflicts in real time.

Trade-offs and pitfalls. CRDTs are not a universal answer to distributed conflict, they only work for data types where a meaningful, automatic merge function exists; something like "the correct final price after two concurrent discount-code applications" often has no context-free merge rule (whether they should stack, or one should override, is a BUSINESS decision, not a data-structure property), and forcing that into a CRDT abstraction usually produces a mathematically well-defined but business-nonsensical result.

Direct answer

Sequence this as an incremental, strangler-fig style replacement (extracting one piece of the monolith at a time behind a routing layer, rather than rewriting it wholesale), and put discovery and telemetry before any code change, because a big-bang rewrite of a revenue-critical monolith is the single most common way this kind of project fails.

Structured elaboration

flowchart LR
    A[Discovery: map real call graph and data dependencies] --> B[Telemetry baseline: instrument before touching code]
    B --> C[Extract one low-risk module behind a facade]
    C --> D[Dual-run and diff against the baseline]
    D --> E{Matches baseline?}
    E -- yes --> F[Cut over, then repeat on next module]
    E -- no --> C
    F --> G[Decommission old path after a bake period]

Phases

• Discovery (2 to 4 weeks): map the monolith's actual call graph and data dependencies, not the outdated architecture diagram, and interview the on-call team for the tribal knowledge of which parts are actually scary in practice.
• Telemetry first, before any code changes: instrument distributed tracing, error rates, and latency percentiles (p95 and p99, the 95th and 99th percentile) so a "before" baseline exists; there is no way to prove a change did not make things worse without one.
• Strangler-fig extraction: carve out one well-bounded, low-risk module first, deliberately not the scariest one, route to it through a facade, and run it in shadow (dual-run) alongside the old path, diffing outputs before cutting traffic over.
• Iterate, prioritized by a risk-versus-effort score, always keeping the monolith itself deployable and the previous extraction's rollback path intact.
• Decommission the old code path only after the new path has run at 100 percent traffic for a defined bake period with no regression against the phase-one baseline.

Risk controls throughout

Feature flags for instant rollback, percentage-based canary traffic shifting, an explicit rollback trigger (for example, if p99 latency exceeds baseline plus 20 percent for 15 minutes, roll back automatically), and a change freeze calendar around peak business periods.

Worked example

An e-commerce monolith's checkout module processes roughly $2,000,000 a day in revenue. The telemetry baseline shows checkout at 800 milliseconds p99 latency and a 0.3 percent error rate. Rather than starting the strangler-fig pattern on checkout itself, the first extraction targets the lower-risk, read-only order-history module, validating the whole pattern (facade, dual-run, cutover, rollback) with a near-zero blast radius (blast radius: how much of the system would be affected if this extraction goes wrong). Only after two successful low-risk extractions does checkout get extracted, behind a 1 percent traffic canary for the first week before ramping up, so the riskiest module benefits from a pattern that has already been proven twice elsewhere.

Trade-offs and pitfalls

There is a real political pull to start with the scariest, highest-visibility module first, since that is what leadership actually asked about; the safer sequence, proving the pattern on a low-risk module first, trades slower initial visible progress for a dramatically smaller blast radius if something goes wrong, and that trade is worth naming out loud to stakeholders rather than making silently. Skipping the telemetry baseline is the second-most common failure, since a regression cannot be detected against a baseline that was never measured. And an open-ended decommission plan ("we will delete the old code eventually") almost never actually happens without a scheduled forcing function tied to a specific bake-period end date.

Direct answer

Managing a complex, multi-stakeholder initiative typically calls on a small toolkit, a stakeholder register, a RACI (Responsible, Accountable, Consulted, Informed) or similar decision-rights chart, a communication plan, and a decision log, and knowing which one to reach for comes down to what specific problem you're solving: who matters (register), who decides (RACI), how you'll keep people informed (communication plan), or why past choices were made (decision log).

Structured elaboration

• Stakeholder register: use when you need to know WHO is involved and how engaged they should be; it's your map of the people.
• RACI (or similar decision-rights chart): use when ambiguity about WHO DECIDES has already caused, or is likely to cause, friction; it's your map of authority, not of people generally.
• Communication plan: use to decide HOW and HOW OFTEN each stakeholder group hears from you; it's the operational layer that turns the register's classifications into an actual cadence.
• Decision log: use to preserve WHY past decisions were made, so the reasoning survives turnover and time; it's your institutional memory.
• Reaching for the right one, not all of them at once. A small, fast-moving initiative might only need a lightweight register and an informal communication habit; a large, multi-quarter, multi-team initiative typically needs all four, because the failure modes each tool prevents (losing track of who matters, decision-rights confusion, communication mismatches, re-litigated history) all become more likely as scale and duration grow.

Worked example

Early in a small initiative, a simple stakeholder register might be the only tool needed. As the initiative grows to span several teams and multiple approval gates, a RACI becomes necessary once the first instance of "wait, who was supposed to approve this" occurs. A communication plan becomes worth formalizing once informal, ad hoc updates start feeling inconsistent across stakeholder groups. A decision log becomes valuable once the initiative has been running long enough that a new stakeholder, or even an existing one, starts asking "why did we choose this approach" about something decided months earlier.

Trade-offs and pitfalls

Introducing all four tools at once for a small, simple initiative is overkill and will likely be abandoned as unnecessary overhead; introduce each tool reactively, in response to the specific friction it solves, rather than as a standard checklist applied uniformly regardless of the initiative's actual size and complexity.

Direct answer
Quantitative evidence (metrics, logs, traces) tells you what happened and when with precision but can miss context and intent; qualitative evidence (engineer interviews, notes, chat logs) fills in the why and the human decision-making, but is subject to memory bias and self-justification. Weigh them together, and when they conflict, treat the disagreement itself as a finding worth investigating rather than picking whichever is more convenient.

Structured elaboration

• Quantitative evidence is precise and timestamped, which makes it the backbone of any timeline, but it can be silent on intent and context: a metric shows latency spiked at 14:03, but not why an engineer chose to deploy at that specific moment or what they believed was true when they did.
• Qualitative evidence captures reasoning and context that logs can't ("I deployed because the dashboard looked fine and I didn't know about the downstream dependency"), but human memory reconstructs events after the fact, often unconsciously smoothing over uncertainty or minimizing one's own role, so it should never override hard timestamped data when the two genuinely conflict.
• Correlating them into one timeline: anchor the timeline on quantitative events (deploys, alerts, metric changes) first, since those are objective and timestamped, then layer qualitative context alongside each event (what the engineer believed, what they were looking at, why they made a given call) as annotation, not as competing facts.
• When they conflict: if an engineer recalls checking a dashboard that logs show wasn't accessed, that's not necessarily dishonesty, memory under stress is genuinely unreliable, but it IS worth investigating why the gap exists: was there a different dashboard, a misremembered timestamp, or a real gap in what was actually checked before the decision was made. The conflict itself, not just its resolution, is often informative about where the process broke down.

Worked example
An engineer recalls seeing a warning-level alert before deploying and deciding it looked minor enough to proceed. Logs show no alert fired until four minutes after the deploy. Rather than concluding the engineer is simply wrong or dismissing the recollection, the investigation digs further and finds the engineer was actually looking at a stale, cached view of the dashboard that hadn't refreshed in several minutes, itself a real and separately worth-fixing gap (a dashboard that can silently show stale data during exactly the moment it matters most). The quantitative record established what actually happened; the qualitative account, once reconciled rather than dismissed, revealed a genuine, previously-unknown contributing factor that the logs alone would never have surfaced.

Trade-offs and pitfalls
The most common mistake is treating quantitative data as always authoritative and qualitative accounts as merely decorative color, which misses genuine contributing factors that only surface through human context. The opposite mistake, treating a confident personal recollection as more reliable than the logs when they conflict, risks building the postmortem's conclusion on a memory distortion. The discipline is to anchor on timestamped data but take conflicting qualitative accounts seriously enough to investigate the gap, not dismiss either source reflexively.

Direct answer

I decide based on whether the new area only pays off for this one project or amortizes across future work, how risky it would be to look shallow in front of whoever's depending on the result, and whether the gap can be covered another way, a colleague, a partner, or an existing playbook, instead of by me personally learning it from zero.

Structured elaboration

Amortization: a skill that only matters for this one project is a weaker bet than one likely to keep paying off afterward, all else equal.

Risk of shallow competence: if the project puts me in front of a client or a senior stakeholder who already knows the adjacent area well, a thin understanding is more exposed and more costly to get caught out on than it would be internally.

Whether the gap can be covered another way: before assuming I personally have to close it, I check whether a colleague already has it, whether an existing playbook or reference implementation gets us most of the way, or whether the smarter move is to go deeper on what I already do and lean on someone else for the adjacent piece.

A realistic view of how far I can actually get in the window: a few weeks is enough to become dangerous-but-functional in most adjacent areas, not expert, so the plan has to assume that ceiling rather than an optimistic one.

The related case of several competing new areas on the same project, not just one: when that happens, the same logic applies but the constraint shifts from picking one over the other to splitting a fixed amount of learning time across more than one, deliberately, rather than defaulting to whichever is most comfortable.

Worked example

A project was coming up that would clearly lean on Kubernetes, an adjacent technical area I'd never worked in myself, and I had a few weeks before it started. I checked first whether someone else already covered it well; nobody did. I judged the risk of looking shallow as moderate, since the audience for this particular piece was mostly internal, not an expert client. I chose to spend the window on Kubernetes rather than going deeper on what I already knew, specifically because it looked likely to matter on projects after this one too, not just this one. By the time the project needed it, I was the one writing the deployment manifests and readiness probes for the new service, and the rollout held up on its own without needing to pull in the platform team. For the side I didn't pick, going deeper on my existing strength, I didn't just drop it: I lined up a teammate who already had strong depth there to review that part of the work, so the gap was covered by someone else's existing skill rather than by hoping it wouldn't come up.

Trade-offs and pitfalls

The common failure here is underestimating how visible shallow knowledge is to someone who already has real depth in the area, which shows up fastest in exactly the moments that matter most. The opposite failure is over-investing in depth on something that turns out to only matter for this one project, which doesn't amortize and would have been better spent elsewhere.

Direct answer

Deciding whether to replace or centralize an owned piece of technology means evaluating it against named technical and commercial criteria before building anything, proving the decision in a small, time-boxed proof of concept (PoC) on a representative case rather than the easiest or hardest one, involving the people who can actually speak to risk and impact, and then owning a transition that runs the old and new systems in parallel before ever decommissioning the original.

Structured elaboration

• Selection criteria, technical: capability fit (does the target platform genuinely support the transformations and schema the current job needs), migration complexity (how much rework is required), and integration effort with existing consumers of the output.
• Selection criteria, commercial: cost (current running cost plus the one-time migration effort, against the target platform's ongoing cost), support model (who's accountable when it breaks, the platform team or the current owner), risk (vendor lock-in and feature-parity gaps), and consumer impact (whether moving it changes output format, timing, or the SLA (service-level agreement) downstream consumers depend on).
• Structuring the proof of concept: pick one representative, medium-complexity case, not the simplest or the hardest, define explicit success criteria before starting (an output-agreement tolerance and a cost or performance target), and time-box it so it can't run open-ended.
• Who to involve: the platform team (who can speak honestly to true capability and the real support model), the downstream consumers of the output (to confirm the impact on them is genuinely acceptable), and a decision-maker who can approve the commercial trade-off, since cost and risk decisions are usually above what the individual owner can sign off on alone.
• Owning the transition: run the new version in parallel with the legacy one for a defined window, comparing output directly, cut consumers over one at a time rather than all at once, and keep the legacy path available as a rollback until the comparison has been clean for the full window.

Worked example

A legacy ETL (extract, transform, load) job costs roughly $1,200 a month in dedicated compute, plus an estimated 3 hours a week of ad hoc maintenance from its current owner.

• PoC selection: rather than the simplest job (3 source tables) or the most complex (over 40 tables with custom logic), the PoC migrates a mid-complexity job, a daily customer-aggregates pipeline with 15 source tables and moderate transform logic, onto the shared platform service.
• Success criteria, defined before starting: output must match the legacy job's daily results within a 0.1 percent tolerance on aggregate row counts across a 3-week comparison window, and the run cost on the shared platform must stay under $400 a month.
• Rule during the PoC: if any two consecutive comparison days exceed the 0.1 percent tolerance, the migration pauses for root-cause investigation before continuing, rather than being pushed through on a schedule.
• Who's involved: the platform team confirms the target service genuinely supports the transform logic used, the pipeline's downstream consumers confirm the output timing and format won't change from their perspective, and a decision-maker approves proceeding given the cost and vendor-risk trade-off.
• Transition plan once the PoC clears its bar: migrate the remaining jobs in ascending order of complexity, cutting each one's consumers over individually, and only decommission the corresponding piece of the legacy system's dedicated compute once each migrated job has run cleanly against the new source for a 30-day silence window with zero discrepancies.

Trade-offs and pitfalls

The most common mistake is choosing the simplest possible job for the PoC because it's the fastest to demonstrate success, which proves far less about whether the platform actually handles the complexity most of the real workload needs. A second mistake is skipping the parallel-run period to move faster, which means the first real discrepancy is discovered by a downstream consumer in production rather than caught safely during the comparison window. Watch also for treating the commercial and technical evaluations as separate conversations: a platform that's technically capable but commercially misaligned, for example locking the team into a support model with unclear escalation, is still the wrong choice even if its capability checks all pass.

Direct answer

Don't optimize the layer that looks slow, instrument the request path end to end first. Get a latency budget broken into per-hop numbers (network, serialization, database, business logic) that actually sum to the 200 ms observed, then attack the hop with the best ratio of milliseconds saved to effort required, re-measuring after every change rather than assuming which layer is guilty before the data says so.

Structured elaboration

Method, in order:

1. Baseline with distributed tracing across the full request path, capturing per-hop timing, not just a total.
2. Form one hypothesis per layer (network/TLS overhead, serialization cost, database query time, business logic compute) and check it against the trace data rather than intuition.
3. Rank candidate fixes by (milliseconds likely saved) divided by (implementation effort and risk), not by which one is technically most interesting.
4. Ship the highest-ranked fix, re-measure the full trace, and repeat, because fixing the biggest hop changes which hop is now biggest.

Isolation checks, when tracing alone doesn't localize it: compare with keep-alive/connection pooling on versus off to isolate network/TLS overhead, compare payload size before and after trimming to isolate serialization cost, and compare with and without a query cache or added index to isolate the database's contribution.

Worked example

Assume tracing on the current 200 ms path yields this breakdown (illustrative numbers, chosen to sum to the measured total):

Reproducing the arithmetic: current total $40+15+100+45=200\text{ms}$, matching the measured baseline. Target total $10+5+25+10=50\text{ms}$, matching the 50 ms goal, and the sum of savings $30+10+75+35=150\text{ms}$ accounts for exactly the gap ($200-50=150$). The database hop is the largest single lever (75 ms, half the total savings) and gets prioritized first for that reason, not because it's assumed to be the culprit before measuring.

Trade-offs & pitfalls

• Jumping straight to rewriting business logic when tracing shows the database is half the budget is solving the wrong problem first, always rank by measured contribution, not by which layer is the most familiar to fix.
• Not re-measuring after each change stacks unverified assumptions, a fix that looked good in isolation can interact badly with the next one.
• Chasing 90% of the theoretical win on the hardest 10% of the effort (a protocol rewrite) before taking the cheap 30 ms keep-alive win first wastes the easiest gains.
• Caching for latency introduces a correctness trade-off (staleness) that needs an explicit owner and time-to-live (TTL), "just add a cache" without that ownership is a common wrong turn.
• Reserve architectural changes (removing a network hop entirely, changing the protocol) for after the low-risk, high-yield fixes are exhausted, they carry more deployment and compatibility risk and should be justified by the remaining gap, not reached for first.

Direct answer

Mitigate first, diagnose second, fix third. In the first minutes: declare an incident, check for a recent deploy or config change and roll it back, and apply safe generic relief (widen the search radius, switch from batched to greedy matching, shift load off any hot partition). Then locate the failure with data: was supply actually missing, or were drivers present but not found or not offered? Scope it by partition and by pipeline stage (index freshness, matcher CPU and queues, downstream call timeouts, offer delivery and acceptance). The shape of the scope points at the cause. Long-term fixes follow from the cause, plus the detection and fallback gaps that let it run for 30 minutes.

Terms

• Unassigned request: a ride request that ended (timed out or was cancelled) without a driver accepting.
• Index lag: how old the positions in the matcher's location index are (now minus the timestamp of the last update applied).
• IC (incident commander): the one person coordinating the response, so others can investigate.
• Consumer lag: how far a stream consumer is behind the newest message.
• Partition / tile / cell: three names used for the same idea below. The city's matching workload is split into geographic slices; a partition is the underlying stream and processing shard that owns one slice, a tile is the map grid square that slice covers, and cell is used the same way when talking about one slice informally. "Moving a tile off a partition" means reassigning that geographic slice's traffic to a different, less-loaded shard.

Step 1: stabilise (first 10 minutes)

1. Declare and assign roles: IC, one investigator per subsystem, one person on communications (riders, drivers, city operations).
2. Change log first: any deploy, config change, feature-flag flip or infrastructure change in the hour before onset? Roll it back. Most incidents like this are self-inflicted, and a rollback is fast and reversible.
3. Generic relief that is safe to apply blind:
   • Widen the search radius and relax filters (for example allow a slightly longer pickup ETA).
   • Switch batched matching to greedy immediate matching if the batching job is slow.
   • Scale out the matcher and move tiles off any partition at high CPU.
   • Retry unassigned requests automatically rather than letting riders give up.
4. Communicate: in-app banner for riders in the city, status to operations.

Step 2: collect data (keep it; the post-mortem needs it)

Step 3: probable causes and their signatures

The last-but-one row is the organic rush-hour trigger: no-matches and long waits with every system healthy. The triage checklist is then: demand-to-supply ratio by zone, whether surge pricing engaged and how fast, whether the search radius and dispatch filters were too tight for the moment, driver acceptance rates (drivers may be declining long pickups), and whether drivers were stranded in the wrong part of the city. That is a marketplace-balance problem, and the fixes are pricing, repositioning incentives and radius policy rather than engineering.

Worked example (a hypothetical incident, to show the reasoning)

• Normal unassigned rate: 3%. During the 30 minutes: 21% city-wide.
• Online idle drivers: 1,100 before, 1,080 during. Supply was there, so it is not an organic surge.
• By partition: 3 of 16 partitions at 95% unassigned; the other 13 at 4%. If demand is roughly equal per partition, (3 x 0.95 + 13 x 0.04) / 16 = (2.85 + 0.52) / 16 = 0.21, which fully explains the 21%. So the problem is partition-scoped.
• On those 3 partitions: index lag p99 went from about 1 s to 45 s at the onset; CPU was normal. The other 13 were normal.
• The change log shows a location-consumer deploy that triggered a consumer-group rebalance (when the set of consumer instances reading a topic changes, for example because one restarts, the group redistributes which instance reads which partitions), which assigned those 3 partitions to one consumer instance that had crashed during startup, and the group waited for its session timeout (the time the group waits without a heartbeat from a member before deciding it is dead and reassigning its partitions) before reassigning.

Root cause: stale index on 3 partitions. Contributing factors: no alert on index lag, the matcher had no rule to refuse candidates with old positions, and the rebalance took minutes to notice a dead consumer.

Step 4: root-cause analysis and long-term fixes

1. Timeline from the data, then ask "why" until reaching a fixable cause, and separately list why it was not detected for 30 minutes and why it was not mitigated sooner.
2. Fixes by category:

Terms in the fixes table: a circuit breaker stops calling a failing dependency for a cooldown period instead of piling up timeouts on it; load shedding deliberately drops or delays lower-priority requests so the rest of the system stays up; a canary deploy rolls a change out to a small slice of traffic first and watches its metrics before rolling out to everyone; a game day is a scheduled rehearsal where the team deliberately injects a failure to test detection and response before a real one happens; a feature flag is a runtime switch that turns a code path on or off without a new deploy.

3. Blameless review with owners and dates on each action item, and a follow-up that verifies the alert would have fired.

Pitfalls

• Debugging before mitigating while riders wait.
• Looking at city-wide averages, which dilute a 3-partition failure into a modest bump.
• Assuming "no match" means "no drivers" without checking idle supply.
• Closing the incident without fixing detection, so the next one also runs for 30 minutes.