System Design Methodology and Trade-off Analysis Questions

Direct answer

Relational (SQL) databases enforce a fixed schema and strong transactional guarantees, ACID: atomicity, consistency, isolation, durability, across related tables, making them the right default whenever the correctness of interrelated data matters more than raw write throughput or schema flexibility. Non-relational (NoSQL) databases trade some of that structure and transactional strength for a flexible schema and horizontal scalability, making them the right choice when data is naturally document-shaped, key-value-shaped, or high-volume enough that spreading it across many nodes matters more than joining it in a single query.

Structured elaboration

Worked example

An e-commerce platform rarely picks one store for everything; it's a polyglot-persistence decision made per data shape.

• Orders and payments go in a relational database, because completing an order touches multiple related rows, the order, the payment, the inventory decrement, that must succeed or fail together, and the business needs ad hoc reporting across them.
• The product catalog goes in a document store, because different product categories genuinely have different attribute sets (a book has an author and page count, a t-shirt has a size and color), and forcing that into a fixed relational schema means either a table full of mostly-null columns or a constant stream of migrations.
• Session state or an in-progress shopping cart goes in a key-value store, because it's accessed by a single key, the session id, needs to be fast, and doesn't need to be joined with anything else.

None of these is "the database for this company"; they're three separate, defensible answers to three differently shaped access patterns on the same platform.

Trade-offs & pitfalls

• Choosing NoSQL for anticipated scale the product doesn't have yet, paying for lost transactional guarantees and application-level join logic before there's any actual scale benefit to show for it.
• Forgetting that "NoSQL" is not one thing: a key-value store, a document store, and a graph database solve different problems, and citing "NoSQL" as a single technology choice is a sign the trade-off hasn't actually been thought through.
• Treating the choice as permanent and binary rather than per-data-shape, polyglot persistence, which is how most real systems at scale are actually built.
• What separates a senior answer: naming the specific access pattern, single-key lookup, multi-row transaction, flexible attributes, graph traversal, that drives each choice, rather than reciting "SQL is for structured data, NoSQL is for unstructured data," which is imprecise enough to be nearly meaningless.

Direct answer

A monolith is a single deployable unit; microservices split a system into independently deployable services that communicate over the network. The monolith wins on development velocity and simplicity while the team and codebase are small; microservices win on independent scaling, fault isolation, and team autonomy once the organization and traffic have grown enough to need them, but they add real operational cost that a small team pays for even before it needs the benefits.

Structured elaboration

Worked example

A five-person startup with 200 daily active users splits its checkout flow into a separate payments service, an inventory service, and a notifications service on day one. In practice: three CI/CD pipelines to maintain instead of one, a network call, with its own latency and failure modes, added to every checkout in place of a function call, and the same five engineers now also debugging cross-service request tracing for a system with barely any real traffic. None of the microservices benefits, independent team ownership or independent scaling under real load, apply yet, because there is one team and no bottleneck to isolate. A modular monolith, meaning a single deployable codebase with clean internal module boundaries and clear ownership per module, gets the same code-organization benefit without the network and operational cost, and it can be decomposed later once an actual bottleneck, not a hypothetical future one, justifies the split.

flowchart TB
  subgraph MONO["Monolith: one deployable unit"]
    direction TB
    W[Web layer]
    B[Business logic]
    D[Data access]
  end
  subgraph MICRO["Microservices: independently deployable, network-connected"]
    direction TB
    PaySvc[Payments service]
    InvSvc[Inventory service]
    NotifSvc[Notifications service]
    PaySvc <--> InvSvc
    InvSvc <--> NotifSvc
  end

Trade-offs & pitfalls

• Adopting microservices for resume-driven or "best practice" reasons rather than a named bottleneck.
• Splitting along technical layers (a services layer, a database layer) instead of business capability boundaries, which just moves tight coupling onto the network instead of removing it; this is the organizational mirror named Conway's Law (a system's structure tends to mirror the communication structure of the organization that built it, so splitting along technical layers just recreates the same coordination problems on the network instead of removing them), worth knowing by name without needing to re-derive it here.
• Treating "the codebase feels big" as the signal a split is overdue, instead of a concrete one: one team's deploy regularly breaks or is blocked by another team's unrelated changes.

Direct answer

Strong consistency means every read after a write returns that write, as if there were only one copy of the data. Eventual consistency means replicas are allowed to disagree temporarily after a write, and are only guaranteed to converge if writes stop arriving. The trade-off: strong consistency gives simpler application logic at the cost of higher latency and lower availability during network trouble, while eventual consistency gives lower latency and higher availability at the cost of the application having to tolerate, or actively resolve, temporarily stale or conflicting data.

Structured elaboration

The consistency spectrum, not just two points

Why "eventual" has no built-in deadline
"Eventually" is not a duration; it is a promise that convergence happens once writes stop. The actual staleness window at any moment depends on the replication pipeline's replay capacity versus the current write rate (worked example below), which is a capacity problem, not a fixed property of the word "eventual."

Worked example

Suppose a replication pipeline can apply writes at 5,000 writes/sec sustained, and normal traffic sends 3,000 writes/sec. The replica keeps up, and staleness stays bounded by pure network propagation delay (milliseconds). If a promotional event spikes incoming writes to 6,000 writes/sec, the pipeline falls behind by:

$6{,}000 - 5{,}000 = 1{,}000 \text{ writes/sec of backlog}$

so after 60 seconds of that spike, the replica is:

$1{,}000 \times 60 = 60{,}000 \text{ writes behind}$

an ever-growing staleness window until the spike ends or the pipeline gets more replay capacity. This is the concrete question an eventual-consistency design has to answer: not "how long is eventually," but "what happens to staleness when write rate exceeds replay capacity, and what backpressure or scaling kicks in when it does."

Two scenarios where eventual consistency is acceptable

1. Social feed likes and comment counts: a like missing for a few seconds is invisible to the user experience and self-corrects on the next refresh.
2. Near real-time analytics dashboards: aggregates lagging by seconds is normal and expected, and nobody is making a financial decision off the exact current second.

Two where it is not

1. Financial account balances and transfers: a stale read can let a user believe funds are available when they aren't, or let two operations both proceed against the same balance.
2. Inventory reservation at checkout: a stale stock count can oversell an item, which is a customer-facing failure and an operational one (an order that can't actually be fulfilled).

The same distinction shows up at the feature level in machine learning. An online feature store computing a rolling 7-day purchase count can safely be eventually consistent; a few seconds of lag rarely changes a recommendation. But if the offline training pipeline computes that same rolling count from a different, out-of-sync snapshot than the one used at serving time, the model trains on a systematically different feature distribution than it sees in production. That training-serving skew is a strong-consistency requirement in disguise: the question isn't "is the count right," it's "was the count computed from the same consistent point-in-time snapshot on both sides."

Trade-offs & pitfalls

• Assuming "eventual" implies a short, informal default delay; there is no bound unless the team designs one, backed by a replay-capacity-versus-write-rate analysis like the one above.
• Reaching for causal consistency without realizing it requires tracking machinery (commonly version vectors: a small per-replica counter attached to each write that lets the system tell whether one write happened before, after, or concurrently with another, named here only) to know which writes are causally related; it is not a free upgrade over plain eventual consistency.
• Applying strong consistency everywhere "to be safe" and paying its latency and availability cost even for the majority of operations, like a social feed's like count, that never needed it.

Direct answer

Payment processing should favor consistency during a network partition, and product catalog browsing should favor availability, because the two operations have opposite costs when they go wrong: an inconsistent payment can create a real financial loss or a double charge, while a stale catalog page is a minor, self-correcting annoyance. The right lens is not "which service is more important" but "what does staleness or unavailability actually cost for this specific data," which is exactly why the same platform can, and should, make opposite choices for its two services.

Structured elaboration

Decision criteria, side by side

Metrics that would defend the split, if challenged

• Payment: commit latency (P95/P99, 95th/99th percentile), and abort/retry rate. A rising abort rate under partition is the system correctly refusing to guess; a rising rate of duplicate-charge incidents would mean the consistency posture failed.
• Catalog: replica lag (a staleness window measured in seconds) and cache hit rate. A growing staleness window is the visible cost of the availability-first choice, and it should have an agreed ceiling (a service-level objective, SLO) rather than being left open-ended.

Mechanism, named but not re-derived
Payment typically uses a majority-quorum write (a quorum is the minimum number of replicas that must agree before a read or write counts as successful) against a small number of strongly consistent replicas (or a single-leader transactional database); catalog typically uses asynchronous replication with read replicas and edge caching. The internals of quorum protocols and cache invalidation are their own topics; what matters here is that these are two different, deliberate consistency configurations applied to the same platform.

Worked example

Take a five-node deployment (N = 5) split across three data centers, and a partition that isolates 2 nodes from the other 3. Two pieces of notation carry the arithmetic below: W is how many replicas must acknowledge a write before it counts as done, and R is how many must respond to a read before it is returned to the caller; AP and CP name the two postures, AP meaning the system favors Availability over Consistency when the network Partitions, CP meaning it favors Consistency over Availability instead.

Catalog (AP): W = 1, R = 1. Either side can serve any single reachable node.

$\text{majority side: } 3 \ge 1, \qquad \text{minority side: } 2 \ge 1$

Both sides stay available. The risk: the two sides may accept conflicting updates to the same catalog item (say, a price change), which gets reconciled (for example, by last-write-wins on a timestamp) once the partition heals.

Payment (CP): majority-quorum writes, requiring W = ⌈(N+1)/2⌉ = 3 acknowledgments.

$W_{maj} = \left\lceil \frac{N+1}{2} \right\rceil = \left\lceil \frac{5+1}{2} \right\rceil = 3$

$\text{majority side: } 3 \ge 3 \; \Rightarrow \; \text{quorum reachable, writes continue}$

$\text{minority side: } 2 < 3 \; \Rightarrow \; \text{quorum unreachable, writes must be refused}$

The same partition event produces two different outcomes on purpose: the catalog stays available everywhere and quietly reconciles later; payment processing keeps working on the majority side and explicitly refuses new authorizations on the minority side, rather than risk two systems each thinking they alone authorized the same order.

The same reasoning generalizes to other service pairs on a platform. A shopping cart during a partition usually leans AP too: accepting an item add on whichever side is reachable and merging any duplicate or conflicting cart state once the partition heals costs less (in lost conversions) than blocking the add. An ML feature store splits the same way payments and catalog do: the online serving path leans AP (serve the last known feature value within a freshness window), while the offline training-data snapshot leans CP (a training run built from a partially-written snapshot silently corrupts the model, so it waits for a consistent point-in-time view).

Trade-offs & pitfalls

• Defending the split with an opinion ("payments feel important") instead of naming a concrete cost of staleness or downtime and a metric that would catch a violation of the chosen posture.
• Assuming the whole platform must share one CAP posture; a mature platform is a portfolio of per-service, sometimes per-operation, decisions.
• Choosing CP for payment but forgetting the user-facing failure path: what checkout shows when the minority side can't reach quorum matters as much as the backend behavior. A clear "please try again" beats a silent hang.
• Naming, without re-deriving, that idempotency keys (a unique identifier attached to a request so that retrying it after a timeout or failure cannot accidentally apply the same charge twice) and compensating transactions let a team take a calculated availability risk on payment writes without producing duplicate charges; that mechanism belongs to a different topic, but knowing it exists is part of a complete answer here.

Direct answer

Latency and throughput are not opposites by nature, they trade off through queueing: pushing more concurrent work through a fixed amount of processing capacity increases the time each request waits behind others, and holding latency low means keeping spare capacity in reserve rather than running it flat out. The right balance comes from setting an explicit target for both (a throughput floor and a tail-latency ceiling), then using queueing math plus load testing to find the utilization level where more throughput starts costing more latency than the business can absorb. What to measure at each load level: the full latency distribution, not just the average, including the 95th and 99th percentile (P95/P99), alongside the downstream business metric (conversion rate, task completion time) the latency target exists to protect.

Structured elaboration

Why the tension exists. Little's Law ties the three quantities together:

$L = \lambda W$

where $L$ is the average number of requests in the system (concurrency), $\lambda$ is the arrival rate (throughput), and $W$ is the average time a request spends in the system (latency). For a fixed amount of concurrency capacity $L$, pushing $\lambda$ up forces $W$ up. Throughput and latency are linked by whatever capacity sits between them, they only look independent at low load.

Decision criteria to walk through, in order:

• Is there a hard external constraint (a contractual service-level agreement, or SLA) versus a soft internal preference? Hard constraints bound the feasible region before you optimize anything.
• Is the load steady or bursty? A bursty workload needs headroom sized for the peak, not the average, or tail latency spikes during every burst.
• What is the true cost of extra capacity relative to the revenue or reliability cost of extra latency? If compute is cheap relative to the business impact of latency, buy headroom instead of accepting queueing.
• Which metric does the product actually care about, median latency almost never predicts user-visible pain, the tail does.

Process: baseline the current latency distribution and throughput, ramp load in steps while recording the full distribution at each step, locate the point where the P95 or P99 curve bends upward sharply (the "knee"), then correlate that knee to the business metric to decide whether operating past it is acceptable.

Worked example

Assume, for illustration, a single worker with an average service time of 10 ms per request ($S = 0.01\text{s}$), so its theoretical maximum throughput is $1/S = 100$ requests per second (RPS). Using the M/M/1 queueing approximation (a standard model for one server handling one request at a time, with randomly arriving requests and randomly varying service times, a common simplification for a single queue), the average wait time in queue at utilization $\rho = \lambda S$ is:

$W_q = \frac{\rho \cdot S}{1-\rho}$

Reproducing the middle row: $W_q = \dfrac{0.90 \times 0.01}{1-0.90} = \dfrac{0.009}{0.10} = 0.09\text{s} = 90\text{ms}$, so $W = 90 + 10 = 100\text{ms}$. Going from 70 to 90 RPS (a 29% throughput increase) roughly triples latency; the next 5.6% of throughput (90 to 95 RPS) roughly doubles it again. This is the shape of the trade-off: throughput gains near saturation cost latency disproportionately.

The same law sizes capacity to hit both targets at once. To sustain 5,000 RPS at an average latency target of 15 ms, the required in-flight concurrency is $L = \lambda W = 5000 \times 0.015 = 75$ concurrent request slots. If each server instance can hold 25 concurrent requests (its thread or connection budget), raw sizing needs $75 / 25 = 3$ instances, but running at 100% utilization guarantees queueing, so target roughly 65% utilization for headroom: $3 / 0.65 \approx 4.6$, round up to 5 instances.

Trade-offs & pitfalls

• Treating the median as the target metric hides exactly the users experiencing queueing delay, always instrument and alert on the tail, not the average.
• Adding raw compute capacity fixes queueing-induced latency but does nothing for latency caused by serialization cost or an inefficient algorithm, these are different bottleneck classes and need different fixes (see bottleneck-identification questions for the diagnostic process).
• Batching or coalescing requests can raise both average throughput and average latency-per-request while making the tail worse for whichever request lands first in a batch, batching trades individual completion time for aggregate efficiency and needs a separate tail-latency check.
• Autoscaling on CPU utilization alone can under-react to a pure queueing problem, alerting or scaling on the latency percentile itself, or on queue depth, catches the tension directly.
• Always tie the chosen operating point back to the business metric with real data (an A/B test or canary), a default like "P95 under 300 ms" is only correct if it is where the business metric actually degrades.