Airbnb Solutions Architect - Entry Level Interview Preparation Guide

Situation: A team submits an ADR that prescribes a different authentication mechanism than an existing platform ADR. As the solutions architect I’d lead a structured resolution to minimize risk, preserve governance, and enable a clear decision.

1. Impact analysis (step-by-step)

• Inventory scope: list services, components, and teams affected by both ADRs.
• Technical comparison: map differences (protocols, tokens, session management, key rotation, SSO integration, backward compatibility).
• Risk assessment: security, compliance, operational cost, performance, migration effort, and time-to-market.
• Dependency analysis: downstream clients, SDKs, monitoring, CI/CD, and rollout timeline.
• Proof-of-concept (if needed): short spike to validate claims and quantify migration cost.

2. Stakeholder negotiation plan

• Convene stakeholders: platform architects, submitting team lead, security, SRE, product owner, and relevant business reps.
• Pre-read package: share impact analysis and PoC results 48 hours prior.
• Facilitated meeting: use decision criteria matrix (security, user experience, cost, vendor lock-in, timeline).
• Options to present: adopt platform ADR, adopt new ADR, hybrid (adapter/compatibility layer), or postpone with mitigations.
• Seek consensus; if contested, escalate to decision authority with documented trade-offs.

3. Decision authority and sign-off

• Define sign-off hierarchy in advance: Platform Architecture Board / Enterprise Architecture Council signs platform-level ADRs; Product/program owner signs product-scoped ADRs. For cross-cutting conflicts, Platform Architecture Board has final authority after stakeholder review.
• Record approvers and sign-off timestamps.

4. Documentation updates

• Update both ADRs: link each other, summarize conflict, rationale for chosen path, migration plan, and deprecation schedule if replacing.
• Create follow-up artifacts: migration runbook, API compatibility contracts, test plans, and communication plan for dependent teams.
• Publish to central architecture repository and notify impacted teams.

5. Preserving audit trail

• Keep all artifacts in version-controlled system (Git) under ADRs: add PRs for changes, review comments, meeting notes, and recorded votes.
• Tag commits and ADR versions; attach PoC results and risk assessments.
• Log decision in governance tracker with approver signatures (digital approvals), timestamps, and a small appendix listing alternatives considered and why rejected.
• Retain an open issue/ticket for migration tracking until completion and include post-mortem after rollout.

Result: This structured process ensures decisions are reproducible, defensible, and minimize disruption—balancing platform consistency, security, and product agility.

Direct answer

Put a dedicated adapter service (not a shared library scattered across callers) between the legacy protocol and your new consumers, responsible for protocol translation, authentication bridging, and error-shape normalization, and treat it as a temporary, shrinking piece of infrastructure with an explicit plan to retire it, not a permanent integration point. The design decisions that matter most are where it lives (sidecar versus a shared gateway), how it handles the legacy system's throughput and latency ceiling, and how you keep it from quietly becoming a second system nobody wants to touch.

Structured elaboration

The core responsibilities of the adapter:

1. Protocol translation: converting between the legacy wire format (SOAP, a proprietary queue protocol, fixed-width records) and whatever your new services speak (REST, gRPC, JSON).
2. Schema and semantic mapping: legacy field names, units, and enums translated to the new domain model, the same discipline as an anti-corruption layer, because this adapter usually needs to be one.
3. Authentication bridging: legacy systems often use older auth mechanisms (SAML, an older XML-based single sign-on standard; mutual certificates; static API keys); the adapter is where that gets exchanged for whatever your new services use, most commonly OAuth 2.0 tokens, with mTLS (mutual TLS) reserved for machine-to-machine cases rather than being an equally likely default.
4. Backpressure and resilience: legacy systems frequently cannot handle the request volume a modern service mesh can generate. The adapter needs connection pooling, request batching, and its own rate limiting so it does not accidentally take down the legacy system it is protecting new consumers from.

On placement: a sidecar (deployed alongside each consuming service) minimizes added network hops and keeps the blast radius of a failure small, at the cost of deploying and versioning the adapter logic N times. A shared gateway centralizes the translation logic in one place, which is easier to evolve and monitor, at the cost of being a single point of failure and a potential throughput bottleneck if the legacy system's guaranteed-once processing requirements mean requests cannot simply be load balanced across replicas without care. For a small number of consumers with strict latency budgets, a sidecar is usually right; for many heterogeneous consumers, a shared gateway with careful capacity planning is usually right.

To avoid the adapter becoming permanent: track what fraction of the legacy system's capabilities still route through it, treat new special cases added to the adapter as a signal that a capability just got more entangled rather than less, and set an actual target date to revisit whether the adapter can start shrinking.

Worked example

A legacy mainframe payments system exposes a proprietary message-queue protocol with strict guaranteed-once processing semantics: the mainframe cannot handle duplicate submissions, and it cannot tell a modern REST client "I got that, don't worry." The adapter needs to:

• Accept REST requests from new services and assign each one an idempotency key before submitting to the mainframe queue, so a retried REST call does not become a duplicate mainframe transaction.
• Maintain a connection pool to the mainframe queue sized to its actual capacity, not the capacity of whatever load-balanced modern service is calling it, and queue or reject excess requests with a clear error rather than letting them silently pile up.
• Translate the mainframe's fixed-format response codes into REST status codes and structured error bodies the new services can actually branch on.
• Emit metrics on translation errors and latency added, since "how much is this adapter costing us" is exactly the number that later justifies (or delays) retiring it.

For the case where a third-party vendor keeps owning part of the flow (say, payment processing) for a bounded transition period, the same adapter pattern applies but with an explicit compatibility window: the adapter needs graceful-degradation fallback flows if the vendor's system is slow or down, SLA monitoring against the vendor's contracted latency, and a version-compatibility check so a vendor-side change does not silently break translation.

Trade-offs and pitfalls

The biggest pitfall is under-provisioning the adapter for the legacy system's real limits: a modern service tier can generate far more concurrent load than a mainframe queue was ever designed for, and the adapter's job is partly to be a deliberate throttle, not just a translator. The second is scope creep: once an adapter exists, it is tempting to route every new integration through it "since it's already there," which is exactly how a temporary migration aid becomes a permanent, poorly-owned piece of critical infrastructure that outlives the legacy system it was built to retire.

Role: On-device analytics reduces raw telemetry, preserves privacy, and enables low-latency personalization and local quality metrics without centralized raw data transfer.
Signals best processed on-device:

• Sensitive personal signals (listening history, keystrokes, health metrics): aggregate locally and send only anonymized or differential-private summaries.
• Low-latency personalization features (recommendation embeddings, caching decisions): compute locally to reduce latency and bandwidth.
• Quality telemetry tied to UX (local crash logs, sensor diagnostics): pre-processed on-device to filter noise and redact PII before upload.
  Signals better centralized:
• Cross-user aggregates (global popularity trends, training data for models), heavy-weight model training, and cross-device deduplication requiring many users' data.
  Why: on-device processing minimizes privacy risk and bandwidth, reduces structural latency, and enables personalization without exposing raw data. Central servers are necessary for global learning, model consolidation, and business analytics that require population-level views.

Direct answer
Split a service into two when at least two of these hold at once: the two parts change at very different frequencies, they're owned (or should be owned) by different teams, they need to scale differently, and a failure in one part shouldn't take down the other. A single one of these signals firing isn't usually enough justification on its own; the process should default to NOT splitting until the signals stack up.

Structured elaboration
Change frequency: if one part of a service is touched every day and another part hasn't changed in a year, bundling them together means every deploy of the fast-changing part carries risk for the stable part too, even though nothing about the stable part actually needed to change. Team ownership: if two different teams are regularly making changes to the same service and stepping on each other's release schedule, that's a coordination cost a split would remove, but only if the teams' work genuinely doesn't need to be coordinated (if they're deeply interdependent, splitting just moves the coordination cost from code review to an API contract). Scaling differences: if one part of a service needs to handle 100x the traffic of another (a hot read path bundled with a rarely-called admin endpoint, for example), scaling the whole service to serve the hot path wastes resources on the cold path. Failure blast radius: if a bug or outage in one part of a service currently takes down an unrelated part, and that unrelated part has a stricter uptime requirement, that's a real argument for isolating them.

The process to avoid premature decomposition: track these signals over time rather than reacting to the first instance of any one of them (one slow deploy or one minor incident isn't a trend), and require at least two signals to agree before committing to a split, since a split that turns out to be unnecessary is expensive to undo (merging two services back into one is rarely done and usually more painful than the original split).

Worked example
A concrete example: an Order service that also handles admin-only bulk data exports. The export feature is used rarely, by a different (internal tooling) team, runs long batch queries that occasionally degrade the Order service's regular request latency, and doesn't need the same uptime service-level agreement (SLA) as live order processing. That's change frequency, team ownership, scaling, and blast radius all pointing the same direction, which is a strong case to split Export into its own service reading from a replica rather than the Order service's primary database.

Trade-offs and pitfalls
The most common overcorrection is splitting on a single weak signal ("these two files feel conceptually different") without checking whether the other three actually agree, producing a service boundary that adds deployment and monitoring overhead without removing any real coordination cost. The opposite mistake, waiting until every signal is screaming before acting, lets a service accumulate enough tangled responsibility that the eventual split becomes a much bigger, riskier project than it would have been if addressed early; the discipline of tracking the four signals over time, rather than reacting to a single incident, is what keeps the timing right in both directions.

Direct answer

When a stakeholder chooses gut over your recommendation, the first job is to figure out whether that's stubbornness or a legitimate competing priority you haven't accounted for, like protecting a release timeline, and then reframe the case around what they're actually protecting, rather than simply repeating the data louder or overriding the objection because you believe you're right.

Structured elaboration

Step 1: diagnose before you reframe. "Going with my gut" usually means one of two things: they don't trust the data, or they trust it fine but are weighing it against something you haven't priced in, like a release date, a relationship, or a risk you don't see. These require different responses. Reframing only works on the second case; on the first, you need to rebuild trust in the data before framing matters.

Step 2: distinguish reframing from overriding. If the resistance turns out to be a legitimate competing priority, for example a PM protecting a release timeline that a delay would blow up, the senior move is not to win the argument and get your way anyway. It's to treat the timeline as a real constraint to negotiate against, not an objection to defeat. Overriding a reasonable objection with a stronger-sounding data point isn't persuasion, it's just louder; it also tends to win the room and lose the relationship.

Step 3: the reframe, in practice.

1. Listen and validate: ask what's driving the instinct and what they're weighing, specifically. This often surfaces the real constraint (a deadline, a prior bad experience, a political consideration) that the data alone never addressed.
2. Restate the shared goal: get explicit agreement on the metric that actually matters, so the conversation isn't "my data vs. your gut" but "how do we both hit the same target."
3. Present evidence against that shared goal, briefly, including where it's uncertain, not just where it's favorable.
4. If the blocker is a legitimate priority like a release timeline, negotiate against it directly: propose a version of your recommendation that doesn't threaten the thing they're protecting, for example a smaller pilot that fits inside the existing timeline rather than a change that would slip it.
5. Offer a low-risk test with a clear decision gate, so the disagreement gets resolved by a result instead of by who argued better.

Worked example

Situation: a product manager wants to launch a promotional push on gut instinct; the leading indicators (early signals, like click-throughs and signups, that show up well before the final conversion numbers do) suggest low conversion probability, and the recommendation is to wait for more signal.

In the room: instead of restating the data more forcefully, the first move is a clarifying question: "is the concern that the data's wrong, or that waiting costs us the launch window?" The PM's answer reveals it's the second: the campaign is tied to a release date that can't move without a real cost. That reframes the whole conversation, this isn't stubbornness, it's a legitimate competing priority.

The reframe: instead of "wait until we have better signal," the proposal becomes a scoped, two-week pilot that launches inside the existing window on a smaller segment, with clear success criteria, so the PM's timeline is protected and the analyst's concern about weak signal gets tested rather than ignored.

Resolution: the PM agrees to the pilot because it doesn't cost them the thing they were actually protecting. The disagreement gets resolved by what the pilot shows, not by whoever had the stronger-sounding argument in the room.

Trade-offs & pitfalls

• Treating every "gut" objection as stubbornness to be argued down is the most common miscalibration here; a good chunk of the time it's a real constraint you simply hadn't modeled.
• Overriding a stakeholder because your data is defensible can win the individual decision and still damage the relationship, making the next disagreement harder.
• Not every gut call is protecting something legitimate; if the "priority" turns out to be unfounded once probed, the reframe should say so directly rather than inventing a compromise that doesn't need to exist.
• A pilot or compromise that doesn't actually test the disagreement (a token concession) just defers the same argument to a later date.

Requirements & constraints:

• 100M monthly users, sub-minute dashboards (near‑real‑time), heavy ad‑hoc analytics, must not impact transactional DBs, cost & scalability.

High-level approach

• Use a hybrid streaming + batch OLAP architecture (Kappa-style): ingest events via Kafka; route to a low-latency OLAP store for dashboards and to a data lake/warehouse for ad‑hoc analytics and historical queries.

Ingestion & CDC

• Transactional DBs -> CDC (Debezium) -> Kafka topics (events, rides, payments).
• Client/mobile telemetry -> Kafka directly (protobuf/Avro).

Low-latency OLAP for dashboards (sub‑minute)

• Use a real‑time analytical engine optimized for fast ingest & low-latency reads (Apache Pinot, Druid, ClickHouse).
• Design materialized real‑time tables focused on dashboard KPIs (active rides, ETA heatmaps, matching latency).
• Partition/segment by event date (hour) + region (city/zone) and shard by ride_id hash for parallelism.
• Maintain pre-aggregations (rolling windows, tumbling windows) and rolling materialized views updated via stream processing (Flink/Beam).
• Keep retention short on raw detail in OLAP (e.g., 7–30 days) and longer for aggregates.

Data lake + OLAP for ad‑hoc analytics

• Sink Kafka to a data lake (S3/ADLS) using compacted Parquet/ORC/Delta Lake files partitioned by date, country, city.
• Partition strategy: date/hour -> region -> event_type; use hashing/bucketing on ride_id for large partitions.
• Use a columnar cloud DW (Snowflake/BigQuery/Redshift Spectrum) or Trino on top of lake for interactive ad‑hoc queries.
• Maintain a logical OLAP model (wide fact tables + dimension tables) and zone separation: raw, curated, aggregated.

ETL / Streaming cadence

• Real‑time: stream processors produce/upsert to OLAP and update materialized aggregations continuously (sub-10s).
• Micro-batch: every 1–5 minutes produce hourly incremental Parquet files for data lake (small-file compaction job runs hourly).
• Batch compaction & partition optimization: daily compaction and Z-ordering/optimize (Delta) to keep query performance.
• Full aggregations: nightly heavy aggregations (DAG via Airflow) to populate pre-computed cubes for complex ad‑hoc queries.

Query strategies & workload isolation

• Separate query routes: dashboards -> real‑time OLAP; ad‑hoc analysts -> data warehouse / lake + cached aggregates.
• Use query federation & caching (materialized views, Redis, CDN for static charts).
• Limit ad‑hoc concurrency; enforce resource classes / workload isolation in DW (slots, warehouses).
• Use cost-based query routing: small-time-range interactive queries hit lake+Trino; heavy historical scans use scheduled ETL or pushdown to Spark.

Partitioning & compaction best practices

• Keep partitions granular enough for pruning (day/hour) but avoid too many small files — use compaction.
• Use partition and clustering (Z-order, sort-by) on high-cardinality columns used in filters (region, driver_id).
• TTL policies for raw detail in OLAP; archive cold partitions to cheaper storage.

Trade-offs & TL;DR

• Real‑time OLAP systems give sub‑minute dashboards with low-latency reads at the cost of duplication and shorter retention.
• Data lake + cloud DW supports deep, cost-efficient ad‑hoc analytics but has higher latency — mitigated by frequent micro-batches and precomputed aggregates.
• Stream-first ingestion with CDC ensures transactional DBs are unaffected while providing consistent, near‑real‑time analytics.

Direct answer

Decide per field with a simple test: what does a user or the business lose if this field is read stale for a few seconds, and does that loss involve authorization, money, or identity? Email, password, and account status gate who can act as whom, so they get a linearizable (single, globally agreed order) read/write path even at a latency cost. Display name and avatar are cosmetic: a stale value for a few seconds costs nothing but a visual blip, so they get eventual, region-local, low-latency writes and reads. Defending the split means showing what making everything strong actually costs on the read path, not just asserting that it is safer.

Structured elaboration

Per-field decision table

Mechanism

This paragraph is implementation detail, useful to know by name but not required to follow the field-by-field argument made above it. Two logical stores per user: a small, strongly-consistent store (consensus-replicated, for example a Raft-based database, where Raft is an algorithm that gets a cluster of replicas to agree on the same order of writes, or a globally-consistent database) for the identity-critical fields, and a multi-region, eventually-consistent store (Dynamo-style or similar) for everything else. Reads compose a single user object from both stores, so only the strong-store portion pays the cross-region latency cost. Read-after-write for the strong fields comes from routing that specific read to the writer's region or the current leader; monotonic reads (once a client has seen a value, a later read never shows it an older one) for the weak fields come from a session token, not from the strong store.

Extending the framework: billing correctness without going fully strong

Billing state is the case that tempts people into "just make everything strong." Resist it: instead of a synchronous global commit for every billing event, use compensating transactions, an idempotent charge (safe to run the same charge request twice, say after a retry, without actually billing the customer twice) plus a defined reversal or refund path if a downstream step (fraud check, inventory hold) fails after the charge already happened. This gets you correctness (the ledger is right once reconciliation finishes) without paying the linearizable-everything latency tax on a field written far less often than it is read.

Defending the split with a number, not an opinion

The strongest defense against "why not just make it all strong" is quantifying what "all strong" costs on the read path, since profile reads vastly outnumber profile writes.

Worked example

Assume a region-local cache read costs 5 ms, and a linearizable read from the strong store (contacting a majority of replicas across 3 regions, with an illustrative one-way inter-region round-trip time (RTT) of 100 ms) costs roughly two one-way trips:

If, say, 95% of profile reads only ever touch display-name or avatar fields (illustrative traffic mix, would come from real access logs), forcing all of them through the strong store means 95% of read traffic pays a 40x latency tax for a guarantee only the remaining 5% of fields ever needed. That is the number to put in front of someone asking why you didn't make everything strongly consistent.

Now the revenue-risk quantification (the second absorbed angle): the case for still investing in correctness on the billing fields, even though they don't get the fully linearizable treatment either.

Ten thousand dollars a day of exposure (illustrative; in practice pulled from real incident and error-rate data) is what justifies spending engineering time on compensating transactions for billing.

Trade-offs & pitfalls

• The strong store becomes a small, high-value target: shard it narrowly (identity fields only) so its lower throughput ceiling never becomes the bottleneck.
• Session tokens that carry the last-seen strong-store commit are what give read-your-own-writes on the critical fields without every read hitting the leader; skipping this is a common miss that reintroduces stale-password bugs.
• Pitfall: treating "eventual consistency" as a synonym for "no correctness work needed." The weak store still needs a conflict-resolution rule (last-writer-wins or a merge function), or two concurrent display-name edits silently lose one.
• Pitfall: treating billing as either fully strong or fully eventual instead of reaching for the third option, compensating transactions, which is usually the right cost and correctness balance for money-adjacent but not identity-adjacent fields.

Managed cloud services vs self-managed on IaaS is a classic trade-off across operational overhead, reliability, performance control, cost predictability, and data portability.

Operational overhead:

• Managed: provider handles provisioning, patching, backups, monitoring—teams focus on application logic. Example: Amazon RDS automates backups and failover.
• Self-managed: you run OS, DB, HA, backups—requires ops expertise and runbook investment.

Reliability:

• Managed: SLAs, built-in replication, automated failover — higher out-of-box resilience.
• Self-managed: can match or exceed reliability if you design and operate robust clusters, but costs and risk are higher.

Performance control:

• Managed: limited tunability and noisy-neighbor risk; good for typical workloads.
• Self-managed: full control over instance types, kernel, I/O tuning — preferable for latency-sensitive, high-throughput systems (e.g., custom-tuned Cassandra on NVMe).

Cost predictability:

• Managed: predictable monthly pricing, less ops labor but can be pricier at scale.
• Self-managed: lower unit costs possible (spot/commit), but operational headcount and overhead make TCO higher or variable.

Data portability:

• Managed: vendor-specific features and backup formats can lock you in; extraction can be effortful.
• Self-managed: using open-source setups simplifies migration and tooling portability.

When to self-manage:

• Need extreme performance tuning, custom extensions, or open-source compatibility (e.g., custom Kafka connectors, specific filesystem drivers).
• Large-scale deployments where per-unit savings offset ops costs.

When managed is better:

• Short time-to-market, limited ops staff, standard workloads (RDS, managed Kafka) or where SLA and simplified operations matter.

Recommendation: evaluate TCO including ops headcount, required SLAs, migration risk, and expected scale. For clients with small teams or rapid delivery needs, favor managed; for large, performance-critical, or highly specialized stacks, consider self-managed with automation and clear runbooks.

Direct Answer

Eventual consistency means that if a piece of data stops changing, every copy of it, spread across different machines, will eventually show the same value, but there's no promise about how quickly that happens. Right after something changes, different copies can briefly disagree, so different people, or even the same person on different devices, can see different things for a short window.

Three Concrete Things Users Will Notice

1. A change that looks like it disappeared or reverted. You update something, say your profile bio, and it saves fine, but a moment later, on a different device or after a refresh, you briefly see the old version again. This happens because that device happened to read from a copy of the data that hadn't caught up yet, not because your change was lost. The same effect shows up in less obviously social products too: right after a recommendation or personalization model is updated, some requests can still be served by a copy of the system using the old values for a short window, so two people who do the exact same thing a minute apart can get visibly different recommendations, purely because of which copy answered them.

2. Actions that look duplicated. If a user doesn't get quick feedback that their action went through (a like, a form submission), they often retry it. If the retry and the original attempt both eventually land, the user can end up seeing what looks like two of the same action. This isn't really an eventual-consistency artifact on its own; it becomes a real duplicate unless the system also deduplicates the underlying writes, not just the on-screen display.

3. Optimistic updates that hide the delay, until they don't. Many products make the delay invisible to the person taking the action by updating their own screen immediately, before the write has actually finished spreading to other copies. For example, when you post a comment, it appears in your own feed the instant you hit submit, even though the write is still propagating to the copies that other users' feeds are reading from. This makes the product feel instant for the person who acted, but it means other people may not see that comment for a moment, and if the underlying write ultimately fails, the app has to quietly roll back the comment it optimistically showed you.

A Concrete Trace

Say a comment-posting service has two copies of the feed data, one near user A and one near user B. User A posts "Great point!". Step 1: A's client shows the comment in A's own feed immediately, the optimistic update, while the actual write is sent to A's nearby copy. Step 2: User B, served by their own nearby copy, refreshes their feed before the write has replicated over to B's copy; B does not see the comment yet. Step 3: once the write has replicated to B's copy, B's next refresh does show the comment. Nothing was lost; B was simply reading from a copy that hadn't caught up at step 2.

Trade-offs and What to Plan For

• Eventual consistency is a deliberate trade for availability and responsiveness, not a bug, but it is the wrong choice for data where a stale answer is actively harmful, such as an account balance, the last unit of inventory, or a security permission change. Those flows are usually worth paying for stronger consistency even if it's slower.
• A common and cheap mitigation for the "did my own change disappear" complaint is guaranteeing read-your-writes (RYW): making sure the person who just made a change always sees their own latest write, typically by routing their own subsequent reads back to the copy that has it, even while other users' view of that same data is still catching up.
• A common wrong turn is treating optimistic UI as if it solves eventual consistency; it only hides the delay from the person who acted. It doesn't change how long the write actually takes to reach everyone else, and it adds its own failure case, rolling back a shown-then-failed action, that the product needs to handle gracefully.

This architecture has three single points of failure once you look past the app tier: the load balancer, and the primary database, are both singletons that the whole request path depends on; the app-server tier looks redundant on paper (three instances) but is only actually redundant if those three instances sit in different fault domains, so it's worth confirming rather than assuming.

Walking the diagram

flowchart LR
  U[Users] --> LB[Load Balancer\nSINGLE instance]
  LB --> A1[App Server 1]
  LB --> A2[App Server 2]
  LB --> A3[App Server 3]
  A1 --> DB[(Primary DB\nSINGLE writer)]
  A2 --> DB
  A3 --> DB

Load balancer (single instance). Every request passes through it, so its failure is a total outage regardless of how healthy the three app servers are behind it. Mitigation: run an active-active pair of LB nodes behind a floating IP or DNS-based failover, or use a managed cloud load balancer where the provider owns that redundancy. Cost: a small amount of extra infrastructure and configuration; the bigger cost is usually operational (health-check tuning, avoiding split traffic during LB failover), not dollars.

App servers (three instances, conditionally redundant). If all three run in the same rack, same availability zone, or share an underlying host, they aren't actually independent, a single power or network event takes out all three at once. Mitigation: spread them across at least two, ideally three, availability zones and confirm the LB health-checks each independently and routes around a dead one automatically. Cost: cross-AZ data transfer costs and slightly higher latency on some requests; this is usually the cheapest SPOF to fix since it's mostly a placement decision, not new infrastructure.

Primary database (single writer, no replica). This is the highest-blast-radius SPOF: if it fails, every write path is down and, depending on the failure mode, recent unreplicated data can be at risk. Mitigation: add at least one synchronous or semi-synchronous replica in another AZ with automated failover (promote-on-failure), plus continuous backups for protection against logical corruption that replication alone wouldn't catch. Cost: this is the most expensive fix of the three, both in infrastructure (a standing replica) and in write latency if replication is synchronous.

Worked example: quantifying the SPOFs

Assume, for illustration, per-component annual availability of 99.95% for the load balancer, 99.9% for each app-server instance, and 99.9% for the database, all figures pinned as inputs for this calculation, not measured facts about any real vendor.

The three-server app tier, if truly independent across fault domains, only fails when all three fail simultaneously, so its unavailability multiplies:

That's an app-tier availability of essentially 99.9999999%, negligible. But the LB and DB are each in series with the whole request path (either one being down takes the whole system down), so their unavailabilities add through multiplication of the availabilities:

That's about $525{,}600 \times (1 - 0.998500) \approx 788.4$ minutes of downtime per year, roughly the sum of the LB's own downtime (about 262.8 min/yr at 99.95%) and the DB's own downtime (about 525.6 min/yr at 99.9%), because the redundant app tier contributes essentially nothing to the failure budget while the two singletons dominate it completely. This is the concrete version of "fix the SPOFs first": no amount of extra app-server redundancy moves that 788-minute number until the LB and DB are addressed.

Trade-offs and pitfalls

The most common wrong turn is stopping at "add more app servers," which is the SPOF that's already effectively solved in this diagram and contributes the least to the real number above; teams do this because it's the cheapest, least disruptive change, not because it's the highest-leverage one. A second pitfall is fixing the database with synchronous cross-region replication by default: it does reduce RPO to near zero, but the added write latency (and reduced availability during a partition, since a strict quorum, requiring a majority of replicas to agree before a write is accepted, can block writes when too few replicas are reachable) is often the wrong trade for a service that would have been fine with an in-region synchronous replica and async cross-region for disaster recovery only. The same "look for the singleton" walk generalizes past this exact diagram: in a streaming ingestion pipeline the SPOF is usually a single partition leader or a schema registry with no standby; in an ML-serving stack it's a lone model server or a feature store with no fallback; and at a more abstract level, any shared control plane (service discovery, config store, secrets manager) or shared cache that every downstream service depends on is a SPOF even when nobody draws it on the diagram.