System Design Methodology and Trade-off Analysis Questions

Direct answer

Match the pricing model to how predictable and interruptible the workload is: pay full price (on-demand) for unpredictable or short-lived work, commit to a discount (a reserved instance or savings plan) for the steady baseline you know you'll run for a year or more, and reach for the deep spot or preemptible discount only when the work can be killed and resumed without harm. Real workloads mix all three: a steady floor on committed pricing, a growth buffer on on-demand, and interruption-tolerant batch work on spot.

Structured elaboration

When each fits

Mixing for one workload

Split the workload into a steady floor, a variable middle, and interruption-tolerant batch, and price each piece with the model that fits it, rather than choosing one model for the whole workload.

Worked example (illustrative unit prices, not vendor-quoted rates)

Assume on-demand costs $0.10 per instance-hour, a committed baseline (reserved instance or savings plan) costs $0.06 per instance-hour (40% off), and spot averages $0.03 per instance-hour (70% off, an illustrative average across reclaim risk).

Baseline: 20 instances running 24/7 (730 hours/month):

Variable batch work: 10 additional instances running 8 hours/day on average (240 hours/month) on spot:

Blended monthly cost with the mix versus all on-demand:

Trade-offs & pitfalls

• Never put a single-master, stateful database on spot; there is no fallback for the primary if it is reclaimed mid-write.
• Overcommitting reserved or savings capacity beyond your actual steady floor turns a discount into waste; size the commitment to your measured baseline, not your hoped-for growth.
• Spot needs operational work to be safe: checkpointing, diversified instance types and zones so a single reclaim wave doesn't take out your whole batch fleet, and an automatic fallback to on-demand.
• Managing multiple pricing commitments has its own administrative overhead (tracking utilization, renewal timing); factor that into whether the discount is worth it at small scale.

Direct answer

Mutual TLS (mTLS, where both client and server present and verify certificates, not just the server) everywhere assumes the internal network is not trustworthy, so a compromised service or a misconfigured firewall rule cannot be used to eavesdrop on or impersonate another service. Perimeter-only encryption assumes the internal network is a trusted zone once past the edge, cheaper to run, but means one breached internal host has broad access to plaintext traffic between every other internal service.

Structured elaboration

mTLS everywhere: contains lateral movement, since a compromised pod cannot silently sniff or spoof traffic between two other services; costs certificate issuance and rotation infrastructure (usually a service mesh sidecar, a small helper process deployed alongside each service instance that handles the mTLS handshake and certificate rotation for it so the application code doesn't have to), added CPU for handshakes and encryption on every hop, and new per-hop latency; certificate expiry becomes a new outage class if rotation automation breaks.

Perimeter-only: no security gain past the edge, everything inside the perimeter is implicitly trusted; much lower CPU and latency overhead internally, no per-service certificate management; a single compromised internal host has plaintext access to everything else inside the perimeter.

Worked example

A request chain touches 5 internal services, each handshake plus encryption overhead adding 2ms per hop, a stated assumption for this exercise:

$\text{added latency} = 5 \times 2\text{ms} = 10\text{ms}$

Against a 200ms end-to-end SLA:

$\frac{10\text{ms}}{200\text{ms}} = 5\%$

a cost worth paying for a payments or healthcare system handling regulated data, and possibly not worth paying for an internal analytics dashboard with no sensitive data in the path.

Trade-offs and pitfalls

The most common failure is not the crypto overhead, it is operational: certificate rotation automation breaking silently until certificates expire and take down the whole mesh at once. Teams that adopt mTLS everywhere without investing in automated rotation and monitoring often experience their first real outage from the mTLS layer itself, not from an attacker.

What the interviewer probes next

Expect questions on rolling this out incrementally without a big-bang cutover, monitoring that catches certificate rotation failure before it becomes an incident, and whether you would carve out exceptions for latency-critical hot paths.

Direct answer

Normalize when write correctness and storage efficiency matter most: each fact lives in exactly one place, so an update touches one row and there is no duplicate copy to drift out of sync. Denormalize when read speed matters most: copying a value into the table that needs it removes a join at read time, at the cost of extra storage and extra write work to keep every copy consistent. The decision is really about where you are willing to pay a cost: on the write path (normalized) or on the read path (denormalized).

Structured elaboration

What normalization buys you

• A single source of truth for each fact (a customer's name lives in one row in the customers table). Rename a customer once, and every order referencing that customer's ID sees the new name immediately, because nothing else stored a copy.
• No update anomalies: you cannot end up with two rows disagreeing about the same customer's e-mail address, because there is only one row.
• Smaller row sizes and less redundant storage, since each attribute is stored once.

What it costs

• Reads that need a full picture (an order plus the customer's name and the product's title) require joining across multiple tables. As the number of tables in the join grows, so does read latency and database load per request.

What denormalization buys you

• Fast reads: a single table scan or index lookup returns everything the page needs, no join required. This matters most for read-heavy, latency-sensitive paths (a product listing page, an order-history feed).
• Fewer round-trips and less join computation on the database, which matters at high read volume.

What it costs

• Duplicated data: the same fact (a product's name, a customer's e-mail) now lives in more than one row.
• Write amplification and staleness risk: change the source fact once, and every duplicate copy must also be updated, or the duplicates drift and become wrong. If you skip updating one copy, you now have silently inconsistent data.
• More total storage, since the same bytes are stored multiple times.

How to actually decide

1. Estimate the read:write ratio on the specific table or field in question, not the system as a whole. A field read a thousand times for every write is a strong denormalization candidate; a field written as often as it is read is not.
2. Ask how often the would-be-duplicated value actually changes. A product's category ID rarely changes; a live inventory count changes constantly. Denormalizing something that changes constantly multiplies your write cost and your staleness risk.
3. Ask how expensive staleness is if a duplicate briefly lags. A denormalized display name that is a few seconds stale is usually fine; a denormalized account balance is usually not.
4. Consider partial solutions before going fully one way: a materialized view or a cached read model gives you denormalized-shaped reads without hand-maintaining duplicate columns in the source tables, at the cost of a refresh lag you must define and tolerate.

Worked example

Take an orders schema. Normalized (third normal form): an orders table (order ID, customer ID, timestamp), an order_items table (order ID, product ID, quantity, unit price), a customers table, and a products table. Rendering an order-detail page means joining order_items to products (for the product name and image) and joining orders to customers (for the customer's name), a three- to four-way join.

Suppose the system processes 1,000,000 orders a month, averaging 3 line items per order, so 3,000,000 order_items rows are written per month. A normalized order_items row (order ID, product ID, quantity, unit price as fixed-width fields) is roughly 28 bytes. A denormalized version that also copies in the product name (about 24 bytes), product category (about 12 bytes), customer name (about 20 bytes), and customer e-mail (about 24 bytes) adds about 80 bytes per row:

At 3,000,000 rows a month, that is:

of pure duplicate data added every month, before counting index overhead or replication. That is the storage side of the cost. The write side shows up when a product gets renamed: if that product already appears in 50,000 historical order_items rows, a normalized schema needs a single row updated in products; a denormalized schema that copied the product name into order_items needs all 50,000 rows updated (or accepts that historical order rows show the old name, which is a legitimate choice for orders specifically, since an order should arguably show the name as it was at purchase time, not the current name).

That last point is the real lesson: denormalizing an order line item's product name is often correct, not just a performance hack, because an order is a historical record and should not silently change when a product is renamed later. Denormalizing a customer's current e-mail address into the same row would be the wrong call, because you want that field to always reflect the customer's latest value, and a copy will drift.

Trade-offs & pitfalls

• Over-normalizing a read-heavy path (a product catalog page hit thousands of times a second) forces the database to redo the same multi-table join on every request, which is real, measurable load that a single denormalized read model would remove.
• Over-denormalizing a field that changes often multiplies write cost for a marginal read benefit, and creates a data-integrity bug class (stale duplicates) that is easy to miss in testing and expensive to debug in production.
• A common pitfall is denormalizing before measuring the actual read:write ratio, based on an assumption that reads are always dominant. Analytics and reporting schemas intentionally denormalize heavily (star-schema fact and dimension tables in an online analytical processing, OLAP, warehouse), because they are overwhelmingly read-heavy and batch-loaded; the live transactional path behind an online transaction processing (OLTP) system usually should not copy that pattern wholesale.
• The strongest senior answer treats this as a per-field decision, not a whole-schema philosophy: a single table can normalize some columns and denormalize others based on how each specific column is actually read and written.

Direct answer

Treat this as a constrained optimization, not a wishlist: list every cost lever, estimate each one's savings and its latency/service-level-objective (SLO) risk independently, combine the savings correctly (multiplicatively, since each lever applies to whatever cost remains after the prior ones, not additively), and sequence the lowest-risk, highest-confidence levers first so you are validating architecture changes only if the safe levers don't already close the gap.

Structured elaboration

Categorize levers by risk to latency and SLOs, not just by savings size:

• Commitment-based (reserved capacity, savings plans on predictable baseline usage): near-zero runtime risk, same infrastructure, different billing.
• Right-sizing and off-peak scheduling: low risk if headroom and monitoring are retained, touches capacity, not request-path logic.
• Caching improvements: moderate risk, changes the request path and introduces a staleness trade-off, needs a pilot.
• Consolidation or replacing a managed service: highest risk, changes topology or introduces new operational surface, needs a staged rollout with a rollback path.

Execution plan: run the low-risk levers first and measure actual savings against current spend, only reach for a higher-risk lever if the low-risk set doesn't clear the target, and size that higher-risk lever to close exactly the remaining gap rather than over-applying it.

Worked example

Assume four levers, sequenced from lowest to higher risk, each estimated independently:

Combined savings are multiplicative on remaining cost, not additive, because each lever's percentage applies to whatever spend is left after the prior levers:

$\text{remaining fraction} = (1-0.15)(1-0.10)(1-0.08)(1-0.05)$

Computing stepwise: $0.85 \times 0.90 = 0.765$; $0.765 \times 0.92 = 0.7038$; $0.7038 \times 0.95 = 0.66861$.

Remaining fraction $\approx 0.6686$, so total reduction $\approx 1 - 0.6686 = 0.3314 = 33.1\%$, clearing the 30% target with roughly 3 percentage points of margin for estimation error, using only levers with low-to-moderate individual latency risk and none requiring the highest-risk consolidation lever.

If these four levers had instead totaled, say, 24%, that is the point to reach for a higher-risk lever (service consolidation or replacing a managed component), sized with the same multiplicative method to close exactly the remaining gap, and gated behind a canary rollout given its higher risk to latency and SLOs.

Trade-offs & pitfalls

• Adding percentages linearly ($15+10+8+5=38\%$) overstates the true combined savings (33.1% here) and can make a plan look like it clears the ceiling when it doesn't, always combine sequential percentage savings multiplicatively.
• Reaching for the single biggest-percentage lever first, even when it's also the highest-risk one, instead of exhausting low-risk levers first, front-loads risk unnecessarily when a safer combination might already hit the target.
• Measuring "savings" against a stale baseline instead of current spend produces accounting surprises when finance reconciles the actual bill.
• Latency and SLO risk aren't uniform across levers, track a risk budget alongside the dollar target, a plan that hits 30% savings but blows through 15% latency increase on one lever has still failed the actual constraint.

Direct answer

A sustained 10,000 writes-per-second, low-latency, stateful workload pushes you away from a design tuned for reads (a single write primary, heavy indexing, read replicas) and toward one built for write scaling: a storage engine optimized for sequential writes, a partitioning scheme that spreads writes across many nodes, and a replication model with an explicit, tunable durability-versus-latency trade-off rather than a single write bottleneck.

Structured elaboration

Why a read-optimized design breaks down here. Traditional B-tree storage engines perform random-access writes and update every index on every insert, each additional index roughly adds another write per record. A single-writer relational primary caps total write throughput at whatever one node's disk and CPU can sustain, and read replicas do nothing for write capacity, they only copy the primary's write stream.

What changes for write-heavy:

• Storage engine: log-structured merge (LSM) tree engines (used by databases like Cassandra, HBase, and the storage layer behind DynamoDB-style stores) append writes sequentially and merge them in the background, trading some read amplification (a single logical read may have to check several separate on-disk files before it can answer, since recent and older writes land in different segments) for much higher sustained write throughput than a B-tree.
• Partitioning: writes are sharded across many nodes by a partition key. The key must be chosen for even cardinality, a monotonically increasing key (like a timestamp or auto-increment ID) concentrates all new writes on one shard regardless of how many nodes exist.
• Replication and durability: instead of one primary with no built-in fan-out, use a quorum-based replication scheme, writes are acknowledged once a majority of replicas confirm, giving a tunable point between "acknowledge on one node" (fast, risks data loss) and "acknowledge on all nodes" (safest, slowest).
• Indexing discipline: keep secondary indexes to the minimum the write path can afford, every index is a write, this is the opposite instinct from a read-heavy design where more indexes are usually free wins.

Worked example

Assume, illustratively, that a single write-optimized node sustains 2,000 writes per second at the target latency.

Nodes needed for raw throughput: $10{,}000 / 2{,}000 = 5$ shards.

For durability, replicate each shard three ways (tolerate one node failure without data loss): $5 \times 3 = 15$ total storage nodes.

A quorum write with $N=3$ replicas and a write quorum of $W=2$ means the client waits only for the second-fastest replica to acknowledge, not the slowest, bounding tail write latency while still guaranteeing the write survives a single node failure.

Cost contrast, provisioned versus per-operation pricing. At an illustrative $0.00001 per write operation under a consumption-priced managed service:

$\text{ops/day} = 10{,}000 \times 86{,}400 = 864{,}000{,}000 \text{ writes/day}$

$\text{daily cost} = 864{,}000{,}000 \times \$0.00001 = \$8{,}640/\text{day} \approx \$259{,}200/\text{month}$

Against 15 provisioned nodes at an illustrative $400/node/month: $15 \times \$400 = \$6{,}000/\text{month}$. At this sustained write rate the per-operation model costs roughly 40 times more, which is why sustained high-volume writes usually favor provisioned or self-managed clusters, and why consumption pricing fits bursty, low-average workloads instead.

Trade-offs & pitfalls

• Carrying over every index from a read-heavy design roughly multiplies write cost by the number of indexes, audit which indexes the write path can actually afford.
• A low-cardinality or monotonically increasing partition key creates a hot shard that caps total throughput no matter how many nodes you add, this is the single most common write-scaling mistake.
• Waiting for all replicas ($W=N$) is the safest durability setting but the slowest; a majority quorum balances safety and latency, the exact quorum size is itself a trade-off decision, not a default.
• High write concurrency needs connection pooling and write batching, naive one-connection-per-request patterns hit connection limits long before they hit the storage engine's real capacity.

flowchart LR
Client --> Router[Write Router]
Router --> ShardA[Shard A Leader]
Router --> ShardB[Shard B Leader]
Router --> ShardC[Shard C Leader]
ShardA --> ShardARep[Shard A Replicas x2]
ShardB --> ShardBRep[Shard B Replicas x2]
ShardC --> ShardCRep[Shard C Replicas x2]