System Design Methodology and Trade-off Analysis Questions

Direct answer

Before designing, find out how the 5,000 requests per second (RPS) actually splits by region and time of day, whether traffic is single-tenant or multi-tenant with isolation requirements, and what the payment provider's own latency and reliability really are, because each answer changes whether you build one active-active deployment or several regional ones with different capacity and failover needs.

Structured elaboration

Clarifying questions and what they change

Traffic distribution changes the architecture directly

Assume, once asked, the answer comes back as 40% North America, 30% Europe, 20% Asia-Pacific, 10% elsewhere:

Assume each service instance handles 50 RPS at the target P95, with 2x headroom for burst and failover:

Without the regional split, you would size one global pool for 5,000 RPS in one place, which is both the wrong shape (traffic is not colocated) and misses the actual failover unit, which is a region, not the global total.

Latency-budget arithmetic once the provider's numbers are known

Assume component P95s: 40 ms edge/network, 30 ms auth, 40 ms inventory reservation, 50 ms fraud check, 250 ms payment provider call, 20 ms response serialization:

Against a 500 ms budget, that leaves 70 ms (14%) of margin. That margin is the number that tells you how much slower the payment provider is allowed to get before the flow must switch to an async-confirm pattern (accept the order, confirm payment out of band) rather than blowing the SLO on every request.

Trade-offs & pitfalls

• Pitfall: sizing to the global RPS total instead of the regional split; you either over-provision the small regions or under-provision the busy one.
• Pitfall: assuming the payment provider's advertised latency holds under your own peak; treat its P95 as a variable you monitor, not a constant you designed around once.
• Multi-tenant isolation is easy to forget when the ask is phrased purely in terms of aggregate RPS; a single large tenant's flash sale can consume capacity meant for everyone else unless per-tenant quotas exist.
• Choosing an async-confirm path for payment buys latency headroom but costs the user a pending state, and costs you a reconciliation or webhook path instead of a single synchronous answer.

flowchart TB
    Client --> Router[Global traffic router]
    Router --> NA[NA region: checkout service]
    Router --> EU[EU region: checkout service]
    Router --> APAC[APAC region: checkout service]
    NA --> PayNA[Payment adapter + regional store]
    EU --> PayEU[Payment adapter + regional store]
    APAC --> PayAPAC[Payment adapter + regional store]

Direct answer

Score each non-functional requirement (NFR, a quality attribute like security, maintainability, or observability rather than a feature) by the risk of skipping it, not by how important it sounds in the abstract, then fund the highest-scoring ones first and consciously document what you are deferring. In this scenario that usually means security and enough observability to see when something breaks get funded first, while maintainability work (broad refactors, exhaustive test coverage) is the one to accept debt on, because a small team can still move fast without it in the short term, while an invisible security or reliability gap can end the project.

Structured elaboration

A repeatable scoring rule

Score each candidate NFR on impact, likelihood, and effort:

where impact and likelihood are rated on a small scale, say 1 to 5 (illustrative severity ratings calibrated with the team) and effort is the cost to address it now. Rank by score, fund top-down until the budget runs out, and document what falls below the line and why.

Worked example (the three from the question)

Assume illustrative ratings for a client project on a tight timeline:

By this scoring, observability actually ranks first here, cheap and high odds you'll need it fast when something breaks. Security ranks second, highest impact and worth the extra effort. Maintainability ranks last, which is the one to consciously under-invest in: ship with a thinner test suite and postpone larger refactors, but only after writing down that decision so it is a choice, not an accident.

Defending the deferred one

Under-investing in maintainability is defensible specifically because its failure mode is slow (code gets harder to change over months) rather than sudden (unlike a security breach or a blind outage), and because a small team on a tight timeline has not yet hit the coordination cost that makes poor maintainability expensive. Conway's Law (a system's structure tends to mirror the communication structure of the team that built it) means that cost shows up later, once more people touch the same code, which is exactly when the decision should be revisited.

Extension (absorbed angle): the same rubric on six NFRs under a revenue constraint

Given six candidate NFRs for a new API (availability, latency, security, observability, maintainability, scalability) and a fixed budget, weight impact by revenue at risk instead of a generic scale, then rank the same way:

The mechanics are identical to the three-NFR case: rank by risk per unit of effort, fund down the list, write down what was deferred and why.

Trade-offs & pitfalls

• Pitfall: treating this as "pick two of three" instead of a continuous funding line; you can partially fund all three (a minimal security baseline plus basic dashboards plus a lighter test suite) rather than fully skipping one.
• Pitfall: scoring by gut feeling instead of writing the numbers down; the value of the rubric is that it survives being questioned by a stakeholder later.
• What changes the ranking: a prior incident (raises likelihood), a compliance requirement (raises impact on security specifically), or a known team-scaling event on the horizon (raises maintainability's score because the Conway's Law cost is about to arrive).
• Under-investing is not the same as ignoring: document the gap, set a revisit trigger (a metric or a milestone), and make sure whoever inherits the debt knows it exists.

Direct answer

Choose a relational database with a flexible-attribute column, for example PostgreSQL with a JSONB (binary JSON) column, rather than a pure document store. The workload's defining features, frequent reads, moderate writes, and occasional complex queries with joins across related entities, are exactly what a relational engine with atomicity-consistency-isolation-durability (ACID) transactions and a real query planner are built for. A document store would force those occasional joins to be rebuilt in application code, which is a worse trade than tolerating a bit more schema rigidity for the flexible fields.

Structured elaboration

Decision criteria to walk through: how often do "occasional" joins actually occur in practice (if frequent, this favors relational strongly); how correctness-sensitive is the data (account state favors strong transactional guarantees); how much of the schema is genuinely unpredictable versus a fixed set of optional fields (JSONB handles the latter well without needing a schema-less engine).

Worked example

A concrete schema: core relational columns, user_id (primary key), email, status, created_at, with foreign-key relationships to organizations and permissions tables to support the join-heavy queries (for example, "list all users in an organization with a given permission"). A JSONB attributes column holds optional or evolving profile fields, indexed with a generalized inverted index (GIN) for filtering on specific attribute keys without requiring a migration every time a new optional field is added.

Decision branch: if the write volume for this profile service later grows to a level one primary node can no longer sustain, that crosses into write-heavy datastore territory (partitioned, write-optimized storage engines) and would call for revisiting this choice; "moderate writes" as stated in this scenario doesn't cross that line, so the relational-plus-JSONB design holds.

Trade-offs & pitfalls

• Choosing a document database by default because "user profile" sounds document-shaped, then discovering the "occasional" joins aren't so occasional in practice, and rebuilding relational logic in application code, is a common wrong turn.
• Over-normalizing the flexible attributes into their own relational tables, when a JSONB column with a targeted index would have been simpler and equally queryable for the actual filter patterns, adds unnecessary schema churn.
• If writes were instead high-concurrency across many independent keys with no cross-record transactions needed, a document or wide-column store would flip this recommendation, this decision is shaped by the workload, not a permanent rule.
• A field that becomes a genuine business invariant (something the system must enforce, not just store) should graduate from the flexible column into a real, constrained relational column, leaving it in JSONB indefinitely trades away the very guarantees the relational choice was made for.

Direct answer

Accept the trade only if the fallback path is already known, tested, and cheap enough to run continuously, in other words, you already know exactly what happens when the new dependency is unavailable and it's within budget. If losing that dependency would blow through the feature's latency or reliability target with no tested fallback, the "free" latency win isn't free, it's borrowed against an incident that hasn't been priced yet. What changes the answer: the dependency's real availability track record and blast radius, whether a cheap compensating control exists (a local cache or replica, a circuit breaker with a real fallback), and how much of the total latency budget the dependency actually buys versus how much of the reliability budget it spends.

Structured elaboration

Architecture options that keep the latency win without an unconditional single point of failure (SPOF):

• Local caching (edge or in-process): serve from a local cache with an acceptable time-to-live (TTL) on cache miss or dependency outage, calling the fast dependency asynchronously to refresh it.
• Replicated fallback: keep a periodically synced local snapshot of the critical data (for example, in an embedded store), and switch to it deterministically if the dependency is slow or down.
• Graceful degradation: return a best-effort or slightly stale result with the feature marked degraded rather than failing the whole request.
• Circuit breaker with a real fallback: short client-side timeouts well under the caller's own budget, bounded retries, and a breaker that opens on sustained errors or latency and routes to one of the above instead of continuing to hammer a failing dependency.

Budgeting the trade-off as an explicit SLO stack, not a single number: treat the end-to-end service-level objective (SLO) as a budget allocated across every hop, so each dependency's slice, and its failure mode, is visible and owned.

Worked example

Assume a user-facing latency SLO for the 99th percentile (P99) of 200 ms. Allocate the budget across the request path:

The budget sums to $20+50+50+30+50=200\text{ms}$, matching the P99 target exactly, so the new dependency is allowed 50 ms of the budget, and 30 ms is explicitly reserved to cover a retry or fallback path if that dependency times out, this reserved slice is what makes "accept the trade" different from silently accepting an unmitigated SPOF. If the caller enforces a timeout on the dependency at, say, 40 ms and falls back to cache within the remaining 10 ms of the retry buffer, the end-to-end P99 target still holds even when the dependency fails, provided the fallback path itself has been measured to fit inside that slice.

Each dependency should also carry its own error budget: if the new dependency's failures start consuming a disproportionate share of the overall transaction's error budget, that is the signal to invest more in the fallback path or reconsider the dependency, before the composed SLO breaches.

Trade-offs & pitfalls

• Building a fallback path but never testing it under a real failure (a chaos experiment that actually kills the dependency) is a common and costly gap, an untested fallback is often broken exactly when it's needed.
• Treating the SLO as one number instead of a budgeted stack hides which hop is actually responsible when the composed latency degrades, budget every hop explicitly.
• Accepting an unconditional SPOF is far more defensible for a non-critical, optional feature (personalization, a "you might also like" widget) than for anything on the authentication or payment path, the acceptable risk is workload-specific, not universal.
• A dependency with a strong availability track record and a cheap, tested fallback is a good trade even with no changes; a dependency with an unproven track record and no fallback is not, regardless of how much latency it saves.

flowchart LR
Client --> Service
Service --> FastDep[New Low-Latency Dependency]
Service --> Fallback[Local Cache or Replica Fallback]
FastDep --> Breaker{Circuit Breaker}
Breaker --> Fallback
Breaker --> Response
Fallback --> Response

Direct answer

Mutual TLS (mTLS, where both client and server present and verify certificates, not just the server) everywhere assumes the internal network is not trustworthy, so a compromised service or a misconfigured firewall rule cannot be used to eavesdrop on or impersonate another service. Perimeter-only encryption assumes the internal network is a trusted zone once past the edge, cheaper to run, but means one breached internal host has broad access to plaintext traffic between every other internal service.

Structured elaboration

mTLS everywhere: contains lateral movement, since a compromised pod cannot silently sniff or spoof traffic between two other services; costs certificate issuance and rotation infrastructure (usually a service mesh sidecar, a small helper process deployed alongside each service instance that handles the mTLS handshake and certificate rotation for it so the application code doesn't have to), added CPU for handshakes and encryption on every hop, and new per-hop latency; certificate expiry becomes a new outage class if rotation automation breaks.

Perimeter-only: no security gain past the edge, everything inside the perimeter is implicitly trusted; much lower CPU and latency overhead internally, no per-service certificate management; a single compromised internal host has plaintext access to everything else inside the perimeter.

Worked example

A request chain touches 5 internal services, each handshake plus encryption overhead adding 2ms per hop, a stated assumption for this exercise:

$\text{added latency} = 5 \times 2\text{ms} = 10\text{ms}$

Against a 200ms end-to-end SLA:

$\frac{10\text{ms}}{200\text{ms}} = 5\%$

a cost worth paying for a payments or healthcare system handling regulated data, and possibly not worth paying for an internal analytics dashboard with no sensitive data in the path.

Trade-offs and pitfalls

The most common failure is not the crypto overhead, it is operational: certificate rotation automation breaking silently until certificates expire and take down the whole mesh at once. Teams that adopt mTLS everywhere without investing in automated rotation and monitoring often experience their first real outage from the mTLS layer itself, not from an attacker.

What the interviewer probes next

Expect questions on rolling this out incrementally without a big-bang cutover, monitoring that catches certificate rotation failure before it becomes an incident, and whether you would carve out exceptions for latency-critical hot paths.