System Design Methodology and Trade-off Analysis Questions

Direct answer

Mutual TLS (mTLS, where both client and server present and verify certificates, not just the server) everywhere assumes the internal network is not trustworthy, so a compromised service or a misconfigured firewall rule cannot be used to eavesdrop on or impersonate another service. Perimeter-only encryption assumes the internal network is a trusted zone once past the edge, cheaper to run, but means one breached internal host has broad access to plaintext traffic between every other internal service.

Structured elaboration

mTLS everywhere: contains lateral movement, since a compromised pod cannot silently sniff or spoof traffic between two other services; costs certificate issuance and rotation infrastructure (usually a service mesh sidecar, a small helper process deployed alongside each service instance that handles the mTLS handshake and certificate rotation for it so the application code doesn't have to), added CPU for handshakes and encryption on every hop, and new per-hop latency; certificate expiry becomes a new outage class if rotation automation breaks.

Perimeter-only: no security gain past the edge, everything inside the perimeter is implicitly trusted; much lower CPU and latency overhead internally, no per-service certificate management; a single compromised internal host has plaintext access to everything else inside the perimeter.

Worked example

A request chain touches 5 internal services, each handshake plus encryption overhead adding 2ms per hop, a stated assumption for this exercise:

$\text{added latency} = 5 \times 2\text{ms} = 10\text{ms}$

Against a 200ms end-to-end SLA:

$\frac{10\text{ms}}{200\text{ms}} = 5\%$

a cost worth paying for a payments or healthcare system handling regulated data, and possibly not worth paying for an internal analytics dashboard with no sensitive data in the path.

Trade-offs and pitfalls

The most common failure is not the crypto overhead, it is operational: certificate rotation automation breaking silently until certificates expire and take down the whole mesh at once. Teams that adopt mTLS everywhere without investing in automated rotation and monitoring often experience their first real outage from the mTLS layer itself, not from an attacker.

What the interviewer probes next

Expect questions on rolling this out incrementally without a big-bang cutover, monitoring that catches certificate rotation failure before it becomes an incident, and whether you would carve out exceptions for latency-critical hot paths.

Direct answer

Decide per field with a simple test: what does a user or the business lose if this field is read stale for a few seconds, and does that loss involve authorization, money, or identity? Email, password, and account status gate who can act as whom, so they get a linearizable (single, globally agreed order) read/write path even at a latency cost. Display name and avatar are cosmetic: a stale value for a few seconds costs nothing but a visual blip, so they get eventual, region-local, low-latency writes and reads. Defending the split means showing what making everything strong actually costs on the read path, not just asserting that it is safer.

Structured elaboration

Per-field decision table

Mechanism

This paragraph is implementation detail, useful to know by name but not required to follow the field-by-field argument made above it. Two logical stores per user: a small, strongly-consistent store (consensus-replicated, for example a Raft-based database, where Raft is an algorithm that gets a cluster of replicas to agree on the same order of writes, or a globally-consistent database) for the identity-critical fields, and a multi-region, eventually-consistent store (Dynamo-style or similar) for everything else. Reads compose a single user object from both stores, so only the strong-store portion pays the cross-region latency cost. Read-after-write for the strong fields comes from routing that specific read to the writer's region or the current leader; monotonic reads (once a client has seen a value, a later read never shows it an older one) for the weak fields come from a session token, not from the strong store.

Extending the framework: billing correctness without going fully strong

Billing state is the case that tempts people into "just make everything strong." Resist it: instead of a synchronous global commit for every billing event, use compensating transactions, an idempotent charge (safe to run the same charge request twice, say after a retry, without actually billing the customer twice) plus a defined reversal or refund path if a downstream step (fraud check, inventory hold) fails after the charge already happened. This gets you correctness (the ledger is right once reconciliation finishes) without paying the linearizable-everything latency tax on a field written far less often than it is read.

Defending the split with a number, not an opinion

The strongest defense against "why not just make it all strong" is quantifying what "all strong" costs on the read path, since profile reads vastly outnumber profile writes.

Worked example

Assume a region-local cache read costs 5 ms, and a linearizable read from the strong store (contacting a majority of replicas across 3 regions, with an illustrative one-way inter-region round-trip time (RTT) of 100 ms) costs roughly two one-way trips:

$$\text{strong-store read latency} \approx 2 \times 100 \text{ ms} = 200 \text{ ms}$$ $$\text{latency multiplier if every read used the strong path} = \frac{200 \text{ ms}}{5 \text{ ms}} = 40\times$$

If, say, 95% of profile reads only ever touch display-name or avatar fields (illustrative traffic mix, would come from real access logs), forcing all of them through the strong store means 95% of read traffic pays a 40x latency tax for a guarantee only the remaining 5% of fields ever needed. That is the number to put in front of someone asking why you didn't make everything strongly consistent.

Now the revenue-risk quantification (the second absorbed angle): the case for still investing in correctness on the billing fields, even though they don't get the fully linearizable treatment either.

$$\text{assumed error rate on a race-prone billing path} = 0.1\% = 0.001$$ $$\text{assumed volume} = 200{,}000 \text{ billing transactions/day at average value } \$50$$ $$\text{expected daily exposure} = 200{,}000 \times 0.001 \times 50 = \$10{,}000/\text{day}$$

Ten thousand dollars a day of exposure (illustrative; in practice pulled from real incident and error-rate data) is what justifies spending engineering time on compensating transactions for billing.

Trade-offs & pitfalls

• The strong store becomes a small, high-value target: shard it narrowly (identity fields only) so its lower throughput ceiling never becomes the bottleneck.
• Session tokens that carry the last-seen strong-store commit are what give read-your-own-writes on the critical fields without every read hitting the leader; skipping this is a common miss that reintroduces stale-password bugs.
• Pitfall: treating "eventual consistency" as a synonym for "no correctness work needed." The weak store still needs a conflict-resolution rule (last-writer-wins or a merge function), or two concurrent display-name edits silently lose one.
• Pitfall: treating billing as either fully strong or fully eventual instead of reaching for the third option, compensating transactions, which is usually the right cost and correctness balance for money-adjacent but not identity-adjacent fields.

Direct answer

Treat pushback as signal, not an attack: restate the alternative back to the interviewer to confirm you understood it, name the assumption your original choice actually depends on, and check whether the pushback introduces a genuinely new constraint or is just testing your conviction. If it changes a load-bearing assumption, revise the design and say so plainly. If it does not, hold the decision and explain why the alternative loses on the axis that matters here, without getting defensive or repeating yourself louder.

Structured elaboration

Separate what kind of decision is being challenged

A useful first move, often invisible to the interviewer but doing real work for you, is classifying the decision itself:

• A reversible decision (a cache eviction policy, an index choice, a queue's retry backoff) can be tried, measured, and changed later at low cost. It is fine to say "I'd start with X, and revisit once we have real traffic data" and mean it.
• A largely irreversible decision (the primary datastore for a dataset that will grow to hold years of production data, a data-residency architecture with legal constraints attached) is expensive to unwind once built. These deserve a firmer defense, because "we'll just change it later" is not actually true for them.
  A candidate who signals which category their choice falls into is showing exactly the judgment this kind of pushback is designed to probe.

The actual steps, in order

1. Paraphrase the alternative back ("so the question is why not do X instead of what I proposed"). This confirms you understood the objection rather than reacting to a version of it you invented, and buys you a beat to think.
2. State the assumption or constraint your original choice depended on, out loud. This is the load-bearing piece: if that assumption is still true, your choice still holds; if the interviewer's follow-up just knocked it down, you now know exactly what to revise.
3. Ask, explicitly if needed, whether the pushback is introducing new information (a constraint you did not have, or did not weight correctly) or is testing whether you actually understand your own trade-off. Those call for different responses.
4. Decide: hold, revise, or partially revise (keep the core choice, adjust a parameter). Say which one you are doing and why, in one sentence.
5. Move on. Do not keep re-litigating a decision you already reopened and closed; that reads as insecurity, not thoroughness.

A worked dialogue skeleton

Interviewer: "Why would you use a queue here instead of just calling the downstream service directly?"

Candidate: "So the question is whether the extra moving part, the queue, is worth it compared to a direct synchronous call. My choice assumes the downstream service is slower and less reliable than the caller can afford to block on, so decoupling protects the caller's own latency and gives us a retry point if the downstream service is briefly unavailable."

Interviewer: "What if that downstream service is actually one of the most reliable and fast services we operate?"

Candidate: "That changes the assumption I was leaning on. If it is genuinely fast and reliable, the resilience argument for a queue weakens a lot, and a direct call with a short timeout and a couple of retries might be simpler and just as safe. I would want to know its actual latency and error behavior before committing either way, but I would not stubbornly keep the queue just because that is what I said first."

Interviewer: "And if it were the flakiest service in the system instead?"

Candidate: "Then I would hold the original call. A flaky downstream dependency is exactly the case the queue protects against, buffering the caller from its failures and giving us retry and backpressure without cascading the failure upstream."

Notice the candidate did not fold immediately in the second exchange, and did not dig in reflexively in the third; the answer changed only where the underlying assumption actually changed.

Trade-offs & pitfalls

• Caving on every objection is the most common failure mode: treating any pushback as proof you were wrong signals you did not have real conviction in the first place, and an interviewer who sees you reverse instantly on a restated version of your own design will keep pushing to find the floor.
• Stonewalling is the opposite failure and just as damaging: repeating your original justification louder, or refusing to update even when the interviewer has handed you a genuinely new constraint, reads as an inability to incorporate new information, which is the exact skill system-design interviews are trying to probe.
• Relitigating from scratch instead of anchoring on the specific new point wastes time and often talks yourself into a worse answer than the one you started with; stay anchored to the one assumption that was actually challenged.
• Treating every decision as equally reversible is a subtler pitfall: defending a cache TTL choice and defending your core datastore choice with the same intensity misses that one of them is cheap to revisit later and one is not. Senior candidates spend their conviction where it is actually load-bearing.
• The strongest signal is not being right on the first guess, it is showing a clear, repeatable process for deciding whether to hold or revise, and being transparent in the moment about which one you are doing.

Direct answer

Removing multi-factor authentication (MFA: requiring a second proof of identity beyond a password) outright trades away real protection against credential-stuffing and phishing for a conversion gain. The better move is usually risk-based step-up authentication: ask for the second factor only when a login looks unusual (new device, new location, impossible travel), not on every login.

Structured elaboration

Always-on MFA: strongest protection against stolen-password takeovers, but friction on every login costs some drop-off.

No MFA: zero friction, but any leaked or guessed password is a full takeover.

Risk-based step-up: most takeover attempts come from a new device, location, or IP, so triggering the second factor only there catches most of the risk while leaving most legitimate logins frictionless, at the cost of building and maintaining a risk-scoring signal.

Worked example

2,000,000 logins a month, 3% historically from a new device or location, a stated assumption for this exercise:

$\text{risky logins} = 2{,}000{,}000 \times 0.03 = 60{,}000$

Always-on MFA adds friction to all 2,000,000 logins; risk-based step-up adds it to roughly 60,000 (3%), while still covering where most takeover attempts land, since an attacker is by definition logging in from a device the real user hasn't used before.

Trade-offs and pitfalls

Risk-based MFA is only as good as its signal: an attacker who steals a session, or reuses the victim's usual IP, evades the trigger entirely. If the fallback for a failed second factor is a weak recovery flow, the vulnerability just moved.

What the interviewer probes next

What signals you'd use to score login risk, how you'd measure whether step-up MFA reduces takeover incidents rather than just complaints, and how you'd keep account recovery from becoming a weaker back door.

Direct answer

Structure the mapping as a matrix: one row per requirement, columns for the target metric, the component(s) that satisfy it, and what you gave up to get there. Walking three rows for this payment subsystem: 99.99% availability drives multi-availability-zone (multi-AZ) redundancy at the cost of doubled infrastructure and failover complexity; sub-200ms p95 (95th-percentile) authorize latency drives a token cache and dedicated crypto hardware at the cost of extra compute spend; and PCI-DSS (Payment Card Industry Data Security Standard) plus 7-year retention drives tokenization and immutable long-term storage at the cost of losing raw-card analytics fidelity and paying for years of storage.

Structured elaboration

Use a table with these columns for every requirement in the list:

This format forces every requirement to land on a concrete component and a concrete cost, rather than staying as an aspiration in a requirements document. It also makes conflicts visible: if two rows both compete for the same fixed budget, that surfaces in the table instead of being discovered mid-build.

Worked example

Three rows from the matrix, with the underlying arithmetic shown:

Row 1: 99.99% availability. A 99.99% target permits:
$\text{allowed downtime/year} = (1 - 0.9999) \times 365 \times 24 \times 60 \text{ min} = 52.56 \text{ min/year}$
Component: the authorize API runs multi-AZ with automated failover rather than a single instance. Gave up: roughly double the compute footprint (active-active or hot-standby) plus the operational cost of regularly testing failover, in exchange for that 52.56-minute annual downtime budget instead of the far larger downtime a single-AZ deployment would risk.

Row 2: sub-200ms p95 authorize latency. An illustrative latency budget that sums to the target:
$20\text{ms (network)} + 30\text{ms (tokenize/HSM)} + 50\text{ms (fraud rules)} + 20\text{ms (cache read)} + 60\text{ms (network to processor)} + 20\text{ms (buffer)} = 200\text{ms}$
Component: an in-memory cache for token lookups and a hardware security module (HSM) colocated with the authorize path, rather than a network round trip to a shared crypto service. Gave up: dedicated cache and HSM capacity that sits idle outside peak hours, which is more expensive per request than a shared pool would be.

Row 3: PCI-DSS plus 7-year retention. Assume, as illustrative pinned inputs, 1 million transactions/day and a 2 KB (kilobyte) retained metadata record per transaction (tokenized, not raw card data):
$\text{bytes/day} = 1{,}000{,}000 \times 2\text{KB} = 2{,}048{,}000{,}000 \text{ bytes} \approx 2.05\text{ GB/day}$
$\text{total (7yr)} = 2.05\text{ GB/day} \times 365.25 \times 7 \text{ days} \approx 5{,}236\text{ GB} \approx 5.2\text{ TB}$
Component: a tokenization service so raw card numbers never enter long-term storage, plus write-once immutable object storage for the 5.2 TB of retained metadata. Gave up: the ability to run ad hoc analytics on raw card attributes, since only tokens and derived fields are retained.

Trade-offs & pitfalls

• The fixed monthly budget row is where the other three collide: if multi-AZ plus dedicated cache/HSM plus 7 years of immutable storage exceeds the budget, something has to re-scope, not silently degrade in production.
• A weak answer lists components without naming what was given up; the "what you gave up" column is the actual trade-off-analysis signal, not the component list itself.
• Treat compliance requirements (PCI-DSS, retention) as filters applied before cost optimization, not something to negotiate down after the architecture is built.
• Revisit the matrix at each design review; a requirement's target or its owning component can shift as the system evolves, and a stale matrix gives false confidence.