System Design Methodology and Trade-off Analysis Questions

Direct answer

Latency and throughput are not opposites by nature, they trade off through queueing: pushing more concurrent work through a fixed amount of processing capacity increases the time each request waits behind others, and holding latency low means keeping spare capacity in reserve rather than running it flat out. The right balance comes from setting an explicit target for both (a throughput floor and a tail-latency ceiling), then using queueing math plus load testing to find the utilization level where more throughput starts costing more latency than the business can absorb. What to measure at each load level: the full latency distribution, not just the average, including the 95th and 99th percentile (P95/P99), alongside the downstream business metric (conversion rate, task completion time) the latency target exists to protect.

Structured elaboration

Why the tension exists. Little's Law ties the three quantities together:

$L = \lambda W$

where $L$ is the average number of requests in the system (concurrency), $\lambda$ is the arrival rate (throughput), and $W$ is the average time a request spends in the system (latency). For a fixed amount of concurrency capacity $L$, pushing $\lambda$ up forces $W$ up. Throughput and latency are linked by whatever capacity sits between them, they only look independent at low load.

Decision criteria to walk through, in order:

• Is there a hard external constraint (a contractual service-level agreement, or SLA) versus a soft internal preference? Hard constraints bound the feasible region before you optimize anything.
• Is the load steady or bursty? A bursty workload needs headroom sized for the peak, not the average, or tail latency spikes during every burst.
• What is the true cost of extra capacity relative to the revenue or reliability cost of extra latency? If compute is cheap relative to the business impact of latency, buy headroom instead of accepting queueing.
• Which metric does the product actually care about, median latency almost never predicts user-visible pain, the tail does.

Process: baseline the current latency distribution and throughput, ramp load in steps while recording the full distribution at each step, locate the point where the P95 or P99 curve bends upward sharply (the "knee"), then correlate that knee to the business metric to decide whether operating past it is acceptable.

Worked example

Assume, for illustration, a single worker with an average service time of 10 ms per request ($S = 0.01\text{s}$), so its theoretical maximum throughput is $1/S = 100$ requests per second (RPS). Using the M/M/1 queueing approximation (a standard model for one server handling one request at a time, with randomly arriving requests and randomly varying service times, a common simplification for a single queue), the average wait time in queue at utilization $\rho = \lambda S$ is:

$W_q = \frac{\rho \cdot S}{1-\rho}$

Reproducing the middle row: $W_q = \dfrac{0.90 \times 0.01}{1-0.90} = \dfrac{0.009}{0.10} = 0.09\text{s} = 90\text{ms}$, so $W = 90 + 10 = 100\text{ms}$. Going from 70 to 90 RPS (a 29% throughput increase) roughly triples latency; the next 5.6% of throughput (90 to 95 RPS) roughly doubles it again. This is the shape of the trade-off: throughput gains near saturation cost latency disproportionately.

The same law sizes capacity to hit both targets at once. To sustain 5,000 RPS at an average latency target of 15 ms, the required in-flight concurrency is $L = \lambda W = 5000 \times 0.015 = 75$ concurrent request slots. If each server instance can hold 25 concurrent requests (its thread or connection budget), raw sizing needs $75 / 25 = 3$ instances, but running at 100% utilization guarantees queueing, so target roughly 65% utilization for headroom: $3 / 0.65 \approx 4.6$, round up to 5 instances.

Trade-offs & pitfalls

• Treating the median as the target metric hides exactly the users experiencing queueing delay, always instrument and alert on the tail, not the average.
• Adding raw compute capacity fixes queueing-induced latency but does nothing for latency caused by serialization cost or an inefficient algorithm, these are different bottleneck classes and need different fixes (see bottleneck-identification questions for the diagnostic process).
• Batching or coalescing requests can raise both average throughput and average latency-per-request while making the tail worse for whichever request lands first in a batch, batching trades individual completion time for aggregate efficiency and needs a separate tail-latency check.
• Autoscaling on CPU utilization alone can under-react to a pure queueing problem, alerting or scaling on the latency percentile itself, or on queue depth, catches the tension directly.
• Always tie the chosen operating point back to the business metric with real data (an A/B test or canary), a default like "P95 under 300 ms" is only correct if it is where the business metric actually degrades.

Direct answer

Relational (SQL) databases enforce a fixed schema and strong transactional guarantees, ACID: atomicity, consistency, isolation, durability, across related tables, making them the right default whenever the correctness of interrelated data matters more than raw write throughput or schema flexibility. Non-relational (NoSQL) databases trade some of that structure and transactional strength for a flexible schema and horizontal scalability, making them the right choice when data is naturally document-shaped, key-value-shaped, or high-volume enough that spreading it across many nodes matters more than joining it in a single query.

Structured elaboration

Worked example

An e-commerce platform rarely picks one store for everything; it's a polyglot-persistence decision made per data shape.

• Orders and payments go in a relational database, because completing an order touches multiple related rows, the order, the payment, the inventory decrement, that must succeed or fail together, and the business needs ad hoc reporting across them.
• The product catalog goes in a document store, because different product categories genuinely have different attribute sets (a book has an author and page count, a t-shirt has a size and color), and forcing that into a fixed relational schema means either a table full of mostly-null columns or a constant stream of migrations.
• Session state or an in-progress shopping cart goes in a key-value store, because it's accessed by a single key, the session id, needs to be fast, and doesn't need to be joined with anything else.

None of these is "the database for this company"; they're three separate, defensible answers to three differently shaped access patterns on the same platform.

Trade-offs & pitfalls

• Choosing NoSQL for anticipated scale the product doesn't have yet, paying for lost transactional guarantees and application-level join logic before there's any actual scale benefit to show for it.
• Forgetting that "NoSQL" is not one thing: a key-value store, a document store, and a graph database solve different problems, and citing "NoSQL" as a single technology choice is a sign the trade-off hasn't actually been thought through.
• Treating the choice as permanent and binary rather than per-data-shape, polyglot persistence, which is how most real systems at scale are actually built.
• What separates a senior answer: naming the specific access pattern, single-key lookup, multi-row transaction, flexible attributes, graph traversal, that drives each choice, rather than reciting "SQL is for structured data, NoSQL is for unstructured data," which is imprecise enough to be nearly meaningless.

Direct answer

CAP theorem says a distributed system cannot simultaneously guarantee all three of Consistency (every read returns the most recent write), Availability (every request receives a non-error response), and Partition tolerance (the system keeps working when some nodes cannot reach others). Because real networks drop packets and links fail, partition tolerance is not optional, so the actual choice during a partition is between consistency and availability. Systems like Google Cloud Spanner lean CP (some requests are blocked or rejected to stay correct); systems like Cassandra or Riak in their common tuning lean AP (every request gets an answer immediately, even a possibly stale one).

Structured elaboration

The three letters, precisely

• Consistency: every node that answers a read returns the same, most-recent value; there is no "which replica did you happen to hit" effect.
• Availability: every request that reaches a non-failed node gets a response, in bounded time, even if that response can't be proven up to date.
• Partition tolerance: the system continues operating even though some messages between nodes are lost or delayed.

Why "CA" is not a real option
A system that isn't partition tolerant is only correct as long as the network never drops a message, which no real, multi-node network guarantees. So "CA" only describes a single-node system, or a distributed system that hasn't yet been tested by a real partition; it is not a deployable choice for a genuinely distributed service.

CP vs AP

PACELC, the extension worth naming
PACELC (if Partitioned, choose Availability or Consistency; Else, choose Latency or Consistency) points out that CAP only describes what happens during a partition. Even with no partition at all, a system still has to decide, on every write, how synchronously to replicate: wait for every replica to acknowledge (favors consistency, costs latency) or acknowledge after one replica and replicate the rest in the background (favors latency, costs consistency). An AP system's day-to-day behavior is really governed by its PACELC answer, not its CAP answer, since partitions are rare and normal operation is not.

Quorum tuning as the middle ground
Many stores let you tune how many replicas must acknowledge a write (W) and how many must respond to a read (R) out of N total replicas, sliding the same system between CP-like and AP-like behavior per operation rather than committing the whole deployment to one label. The mechanics of quorum protocols are their own topic; the point for this answer is that the CP/AP choice is a dial, not a single global switch.

Worked example

Take three replicas (N = 3) with a write quorum W = 2 and a read quorum R = 2, so that any write and any read are guaranteed to overlap on at least one replica:

$W + R > N \quad\Rightarrow\quad 2 + 2 > 3$

Now a partition splits the three replicas into a group of 2 and a group of 1.

• The group of 2 can still assemble a quorum (2 out of the required 2), so it keeps serving reads and writes: consistent and available, on that side.
• The lone replica cannot assemble a quorum (1 is short of 2), so it must refuse requests rather than answer from possibly-stale local data: it sacrifices availability to avoid sacrificing consistency.

If the same store were instead configured for W = 1, R = 1 (an AP-leaning setting), both sides could keep accepting writes independently during the same partition, and the two sides would need to reconcile any conflicting writes once the partition heals; nothing here changes the network, only the dial.

Two alternate ways to see the same trade, folded in from adjacent scenarios: to a non-engineer stakeholder, the choice can be framed without the theorem's name at all: "during an outage, would you rather the app show a slightly out-of-date number and stay up, or freeze and show an error until we're sure the number is right?" That question is CAP in plain language. The same trade also shows up in machine learning infrastructure: an online feature store serving features to a live model at inference time typically leans AP (serve the last known values within a freshness window rather than block the request), while the offline pipeline that snapshots those same features for a training run typically leans CP (a training set built from a partially-written snapshot silently corrupts the model, so the pipeline waits for a consistent point-in-time snapshot instead).

Trade-offs & pitfalls

• Treating "pick 2 of 3" as one permanent, whole-system architecture decision, instead of a per-operation or even per-data-type decision a single platform can make differently in different places.
• Conflating CAP's "C" with the "C" in ACID (atomicity, consistency, isolation, durability): CAP consistency is about replicas agreeing with each other; ACID consistency is about the database enforcing its own integrity constraints. They answer different questions and can both be true or false independently.
• Forgetting PACELC: an AP system still owes an answer to "how synchronously do you replicate during normal operation," since that decision affects latency and staleness every day, not just during rare partitions.
• Citing "CA" as if it were a legitimate label for a real distributed deployment, rather than a description of a single-node system or an untested assumption.

Direct answer

Match the pricing model to how predictable and interruptible the workload is: pay full price (on-demand) for unpredictable or short-lived work, commit to a discount (a reserved instance or savings plan) for the steady baseline you know you'll run for a year or more, and reach for the deep spot or preemptible discount only when the work can be killed and resumed without harm. Real workloads mix all three: a steady floor on committed pricing, a growth buffer on on-demand, and interruption-tolerant batch work on spot.

Structured elaboration

When each fits

Mixing for one workload

Split the workload into a steady floor, a variable middle, and interruption-tolerant batch, and price each piece with the model that fits it, rather than choosing one model for the whole workload.

Worked example (illustrative unit prices, not vendor-quoted rates)

Assume on-demand costs $0.10 per instance-hour, a committed baseline (reserved instance or savings plan) costs $0.06 per instance-hour (40% off), and spot averages $0.03 per instance-hour (70% off, an illustrative average across reclaim risk).

Baseline: 20 instances running 24/7 (730 hours/month):

Variable batch work: 10 additional instances running 8 hours/day on average (240 hours/month) on spot:

Blended monthly cost with the mix versus all on-demand:

Trade-offs & pitfalls

• Never put a single-master, stateful database on spot; there is no fallback for the primary if it is reclaimed mid-write.
• Overcommitting reserved or savings capacity beyond your actual steady floor turns a discount into waste; size the commitment to your measured baseline, not your hoped-for growth.
• Spot needs operational work to be safe: checkpointing, diversified instance types and zones so a single reclaim wave doesn't take out your whole batch fleet, and an automatic fallback to on-demand.
• Managing multiple pricing commitments has its own administrative overhead (tracking utilization, renewal timing); factor that into whether the discount is worth it at small scale.

Direct answer

A monolith is a single deployable unit; microservices split a system into independently deployable services that communicate over the network. The monolith wins on development velocity and simplicity while the team and codebase are small; microservices win on independent scaling, fault isolation, and team autonomy once the organization and traffic have grown enough to need them, but they add real operational cost that a small team pays for even before it needs the benefits.

Structured elaboration

Worked example

A five-person startup with 200 daily active users splits its checkout flow into a separate payments service, an inventory service, and a notifications service on day one. In practice: three CI/CD pipelines to maintain instead of one, a network call, with its own latency and failure modes, added to every checkout in place of a function call, and the same five engineers now also debugging cross-service request tracing for a system with barely any real traffic. None of the microservices benefits, independent team ownership or independent scaling under real load, apply yet, because there is one team and no bottleneck to isolate. A modular monolith, meaning a single deployable codebase with clean internal module boundaries and clear ownership per module, gets the same code-organization benefit without the network and operational cost, and it can be decomposed later once an actual bottleneck, not a hypothetical future one, justifies the split.

flowchart TB
  subgraph MONO["Monolith: one deployable unit"]
    direction TB
    W[Web layer]
    B[Business logic]
    D[Data access]
  end
  subgraph MICRO["Microservices: independently deployable, network-connected"]
    direction TB
    PaySvc[Payments service]
    InvSvc[Inventory service]
    NotifSvc[Notifications service]
    PaySvc <--> InvSvc
    InvSvc <--> NotifSvc
  end

Trade-offs & pitfalls

• Adopting microservices for resume-driven or "best practice" reasons rather than a named bottleneck.
• Splitting along technical layers (a services layer, a database layer) instead of business capability boundaries, which just moves tight coupling onto the network instead of removing it; this is the organizational mirror named Conway's Law (a system's structure tends to mirror the communication structure of the organization that built it, so splitting along technical layers just recreates the same coordination problems on the network instead of removing them), worth knowing by name without needing to re-derive it here.
• Treating "the codebase feels big" as the signal a split is overdue, instead of a concrete one: one team's deploy regularly breaks or is blocked by another team's unrelated changes.