System Design Methodology and Trade-off Analysis Questions

Direct answer

Use two different datastores, because the two workloads sit at opposite ends of nearly every axis that matters: the session store needs low-latency point lookups by key with automatic expiry, while the audit log needs durable, ordered, append-only writes with long retention and replay. Forcing one engine to serve both means either paying the audit log's durability and retention cost on every fast session read, or giving up the audit log's append-only, replayable guarantees.

Structured elaboration

Why this generalizes beyond sessions and audit logs: the absorbed framing of a machine-learning (ML) metadata store shows the same underlying shape. Continuous integration (CI) pipelines generate a high write rate logging experiment runs, while dashboards and monitoring run read-heavy, scan-heavy queries over historical runs. That's structurally the identical pattern, a write-optimized append path pulling against a read-optimized query path, which is why the "one store or two" question should be re-asked per component rather than defaulting to whatever store is already in the stack.

Worked example

Primary scenario (session + audit). Use Redis for sessions, with native per-key TTL handling expiry automatically, no cleanup job required. Use an append-only log or a partitioned table for the audit trail.

Storage growth check on the audit side: assume, illustratively, 500 audit events per second at 1 KB each.

$500 \text{ events/s} \times 1\text{KB} = 500 \text{KB/s}$
$500 \text{KB/s} \times 86{,}400 \text{s/day} = 43{,}200{,}000 \text{KB/day} = 43.2 \text{GB/day (decimal GB, 1 GB = 1{,}000{,}000 KB)}$
$43.2 \text{GB/day} \times 730 \text{days (2 years)} = 31{,}536 \text{GB} \approx 31.5 \text{TB}$

Thirty-one and a half terabytes of append-only history is exactly the kind of cheap, sequential, long-retention storage a log-oriented system is built for, and exactly the kind of volume that would be prohibitively expensive to hold in an in-memory session store's provisioned capacity.

Second scenario (ML metadata, the absorbed framing). The same reasoning applies with a relational or document store for structured experiment metadata (read-heavy, queried by dashboards across many attributes) paired with an append-only log or object storage for the raw high-volume run logs written continuously by CI, again two engines, chosen for two genuinely different access patterns, not one engine stretched across both.

Trade-offs & pitfalls

• Storing audit events in the session store "for convenience" either evicts old sessions to make room or silently drops the durability guarantee the audit log actually needs, neither is acceptable for an audit trail.
• Using an append-only log system for session storage means reimplementing per-key expiry logic that a key-value store like Redis provides natively, most log/stream systems have topic-level retention, not per-key TTL.
• Two datastores are two systems to operate, monitor, and back up, that overhead is worth paying here because the access patterns are genuinely incompatible; it would not be worth it if the two workloads were similar enough to share one engine.
• Deletion requirements, for example a regulatory "right to be forgotten" request, are harder to satisfy in an append-only log than in a key-value store, this needs a tombstone (a marker record saying "treat this key as deleted") plus a compaction strategy (the background process that rewrites and shrinks the log over time, actually purging tombstoned data) decided up front, not retrofitted after the first request arrives.

flowchart LR
Client --> SessionAPI[Session Read/Write]
Client --> AuditWriter[Audit Event Writer]
SessionAPI --> Redis[(Redis Session Store)]
AuditWriter --> Log[(Append-only Log)]
Log --> ColdStorage[(Long-term Storage)]

Direct answer

Split along business capability and data ownership, not by technical layer, and treat coupling and cohesion as the actual test: a service boundary is right when it groups things that change together and separates things that don't, and when one team can own its full lifecycle (build, deploy, operate) without waiting on another team to also deploy. Team size and deployment cadence usually decide the timing more than the theory does: a well-modularized monolith can run comfortably until the coordination cost of shared deploys and shared blast radius starts to exceed the operational cost of running the same code as separate services.

Structured elaboration

The criteria, applied together

• Bounded context or business capability: one service per coherent business concept (Orders, Inventory, Billing), not per database table.
• Data ownership: the service that owns a piece of data is its only writer; everyone else goes through its API or its events, never a shared schema.
• Deployment independence: if two "services" cannot be deployed on separate schedules without breaking each other, they are one service wearing two names, a distributed monolith.
• Team boundaries (Conway's Law: a system's structure tends to mirror the structure of the team that builds it): align a service to a team that can own it end to end, so ownership and org chart don't fight each other.
• Transaction boundary: keep operations that need a real ACID (atomicity, consistency, isolation, durability) transaction inside one service; cross-service consistency should default to eventual consistency plus an explicit compensating action, not a distributed transaction.
• Chattiness: if two components exchange many synchronous calls per user request, the network hop between them is pure overhead with no ownership benefit; merge them.

The team-size-driven worked example (absorbed angle)

Consider an org at 200 people, organized as roughly 20 teams, running a well-modularized monolith with clear internal module boundaries (a modular monolith). Model the shared deploy pipeline as a single server processing one deploy at a time, 30 minutes each, across a 16-hour working day (960 minutes):

At 62.5% utilization there is queueing delay, but the pipeline is stable. Now grow to 500 people, roughly 50 teams, same one-deploy-at-a-time pipeline:

Demand exceeding capacity on a single-server queue means the queue is unstable: it does not just get slower, it grows without bound. That crossing point, not a stylistic preference for microservices, is the concrete signal to start extracting services along the module boundaries the modular monolith already has, so teams stop sharing one serialized deploy pipeline and one shared blast radius.

Anti-patterns that signal you split wrong (or didn't split at all)

• Shared database schema across "separate" services: the clearest sign of a distributed monolith with extra network hops.
• Splitting by technical layer (a UI service, an API service, a database-access service) instead of by capability: nothing can deploy alone, because every user-facing change touches all three.
• A "god" service or shared library that every team depends on for routine changes: it recreates the same coordination bottleneck a monolith had, with worse debugging.
• Over-splitting a capability that still needs real ACID guarantees just because a diagram looks tidier with more boxes.

Trade-offs & pitfalls

• Splitting too early, before the coordination cost above actually bites, buys distributed-systems complexity (network calls, partial failure, eventual consistency) for a coordination problem you didn't have yet.
• Splitting too late means the deploy-pipeline math above turns into a real, measured queue of waiting teams, not a hypothetical.
• The bounded-context choice is the expensive one to get wrong: correcting a wrong service boundary later means a data migration, not just a configuration change.
• Watch for teams treating microservices as a goal instead of a response to a specific coupling problem; the checklist above should produce the boundary, not the other way around.

Direct answer

Normalize when write correctness and storage efficiency matter most: each fact lives in exactly one place, so an update touches one row and there is no duplicate copy to drift out of sync. Denormalize when read speed matters most: copying a value into the table that needs it removes a join at read time, at the cost of extra storage and extra write work to keep every copy consistent. The decision is really about where you are willing to pay a cost: on the write path (normalized) or on the read path (denormalized).

Structured elaboration

What normalization buys you

• A single source of truth for each fact (a customer's name lives in one row in the customers table). Rename a customer once, and every order referencing that customer's ID sees the new name immediately, because nothing else stored a copy.
• No update anomalies: you cannot end up with two rows disagreeing about the same customer's e-mail address, because there is only one row.
• Smaller row sizes and less redundant storage, since each attribute is stored once.

What it costs

• Reads that need a full picture (an order plus the customer's name and the product's title) require joining across multiple tables. As the number of tables in the join grows, so does read latency and database load per request.

What denormalization buys you

• Fast reads: a single table scan or index lookup returns everything the page needs, no join required. This matters most for read-heavy, latency-sensitive paths (a product listing page, an order-history feed).
• Fewer round-trips and less join computation on the database, which matters at high read volume.

What it costs

• Duplicated data: the same fact (a product's name, a customer's e-mail) now lives in more than one row.
• Write amplification and staleness risk: change the source fact once, and every duplicate copy must also be updated, or the duplicates drift and become wrong. If you skip updating one copy, you now have silently inconsistent data.
• More total storage, since the same bytes are stored multiple times.

How to actually decide

1. Estimate the read:write ratio on the specific table or field in question, not the system as a whole. A field read a thousand times for every write is a strong denormalization candidate; a field written as often as it is read is not.
2. Ask how often the would-be-duplicated value actually changes. A product's category ID rarely changes; a live inventory count changes constantly. Denormalizing something that changes constantly multiplies your write cost and your staleness risk.
3. Ask how expensive staleness is if a duplicate briefly lags. A denormalized display name that is a few seconds stale is usually fine; a denormalized account balance is usually not.
4. Consider partial solutions before going fully one way: a materialized view or a cached read model gives you denormalized-shaped reads without hand-maintaining duplicate columns in the source tables, at the cost of a refresh lag you must define and tolerate.

Worked example

Take an orders schema. Normalized (third normal form): an orders table (order ID, customer ID, timestamp), an order_items table (order ID, product ID, quantity, unit price), a customers table, and a products table. Rendering an order-detail page means joining order_items to products (for the product name and image) and joining orders to customers (for the customer's name), a three- to four-way join.

Suppose the system processes 1,000,000 orders a month, averaging 3 line items per order, so 3,000,000 order_items rows are written per month. A normalized order_items row (order ID, product ID, quantity, unit price as fixed-width fields) is roughly 28 bytes. A denormalized version that also copies in the product name (about 24 bytes), product category (about 12 bytes), customer name (about 20 bytes), and customer e-mail (about 24 bytes) adds about 80 bytes per row:

At 3,000,000 rows a month, that is:

of pure duplicate data added every month, before counting index overhead or replication. That is the storage side of the cost. The write side shows up when a product gets renamed: if that product already appears in 50,000 historical order_items rows, a normalized schema needs a single row updated in products; a denormalized schema that copied the product name into order_items needs all 50,000 rows updated (or accepts that historical order rows show the old name, which is a legitimate choice for orders specifically, since an order should arguably show the name as it was at purchase time, not the current name).

That last point is the real lesson: denormalizing an order line item's product name is often correct, not just a performance hack, because an order is a historical record and should not silently change when a product is renamed later. Denormalizing a customer's current e-mail address into the same row would be the wrong call, because you want that field to always reflect the customer's latest value, and a copy will drift.

Trade-offs & pitfalls

• Over-normalizing a read-heavy path (a product catalog page hit thousands of times a second) forces the database to redo the same multi-table join on every request, which is real, measurable load that a single denormalized read model would remove.
• Over-denormalizing a field that changes often multiplies write cost for a marginal read benefit, and creates a data-integrity bug class (stale duplicates) that is easy to miss in testing and expensive to debug in production.
• A common pitfall is denormalizing before measuring the actual read:write ratio, based on an assumption that reads are always dominant. Analytics and reporting schemas intentionally denormalize heavily (star-schema fact and dimension tables in an online analytical processing, OLAP, warehouse), because they are overwhelmingly read-heavy and batch-loaded; the live transactional path behind an online transaction processing (OLTP) system usually should not copy that pattern wholesale.
• The strongest senior answer treats this as a per-field decision, not a whole-schema philosophy: a single table can normalize some columns and denormalize others based on how each specific column is actually read and written.

Direct answer

For a read-heavy workload with moderate writes, for example a 90% read / 10% write split, the default lean should be read replicas, because they scale read capacity without adding a second consistency model to reason about. Add a cache on top only for a narrow, measured set of hot keys that replicas still cannot serve cheaply or quickly enough. The decision comes down to three questions: can the application tolerate replication lag or cache staleness, is the read traffic skewed enough that a small cache absorbs most of it, and is there enough engineering capacity to build correct cache invalidation.

Structured elaboration

Decision rule: if managed read replicas are available with a replication lag the application tolerates, start there. Add a cache only where a specific, measured hot-key or hot-query pattern needs sub-database latency or needs to shed load the replicas can't absorb cheaply.

The absorbed framing of a 90% read / 10% write split is the same decision restated: the write share matters because every write still has to land on the primary and propagate down to every replica. At 10% writes this is a non-issue; if the write share climbed toward 40-50%, the datastore choice itself would need revisiting (see write-heavy architecture reasoning), not just the cache-versus-replica question.

Worked example

Assume a baseline read load of 10,000 requests per second (RPS) and, illustratively, that each database read replica sustainably serves 2,000 RPS at acceptable latency.

Without a cache: replicas needed $= 10{,}000 / 2{,}000 = 5$ replica nodes (plus the primary handling writes).

With a cache in front, assume an 80/20 access skew (a common real-world pattern: 20% of keys account for 80% of reads) and a 90% cache hit rate on that hot 20%:

$\text{DB-bound reads} = (0.8 \times 10{,}000 \times (1-0.9)) + (0.2 \times 10{,}000) = (8{,}000 \times 0.1) + 2{,}000 = 800 + 2{,}000 = 2{,}800 \text{ RPS}$

Replicas needed with the cache in place: $\lceil 2{,}800 / 2{,}000 \rceil = 2$ replicas.

That's a drop from 5 replica nodes to 2 from caching just the hot 20% of keys, which is why a targeted cache is usually layered on top of replicas rather than chosen instead of them: it earns its operational cost only where the skew is large enough to matter.

Trade-offs & pitfalls

• Adding a cache first because it "feels faster," without first measuring read skew, risks solving an already-adequate problem while introducing invalidation bugs for no real gain.
• Replicas trade consistency for scale: if a user reads immediately after writing in the same session, that read must be routed to the primary or to a lag-aware router, or the user will see stale data from their own write.
• Cache stampede on a mass invalidation event can hit the primary at exactly the worst moment, right after the thing that made the cache go stale in the first place; request coalescing and staggered TTLs guard against this.
• A self-run cache cluster is a second system to operate, patch, and monitor; a managed database's built-in replicas usually cost less operational attention than they save, which is why replicas are the default and the cache is the exception.

Direct answer

Treat it as an investment decision: translate the latency improvement into a dollar figure, usually via conversion or engagement lift, compare it to the extra cost plus the risk the added complexity introduces, and ship it only if the net benefit is positive with a margin that survives your uncertainty about the conversion assumption. What changes the answer is the size of that margin: a break-even case should prompt more evidence (a real experiment) before committing, not a coin flip.

Structured elaboration

The decision framework

1. Translate tail latency into revenue using your own historical relationship between latency and conversion, not an assumption invented for this decision alone.
2. Compare that revenue gain to the extra infrastructure cost plus the risk-adjusted cost of the added complexity (a higher chance and cost of an incident, a longer time to diagnose an outage).
3. Validate with a phased rollout (a small canary first) before committing the full 3x spend, so the confirmed number is what's being paid for, not the projected one.
4. Set a rollback trigger up front: if the reliability cost shows up (more incidents, longer mean time to recover) before the revenue gain is confirmed, pull back.

Worked example (illustrative assumptions; would come from your own A/B data in practice)

Assume current infrastructure costs $50,000/month and the proposed change triples it to $150,000/month:

Assume 99th-percentile (P99) tail latency drops from 2,000 ms to 1,000 ms, a 1,000 ms reduction, and assume (illustrative, pulled from historical experiments in a real decision) a 0.5% relative conversion lift per 100 ms of P99 reduction:

Against baseline monthly revenue of $2,000,000:

That is a break-even case at these assumptions, exactly the situation that should prompt a real experiment (canary the change to a fraction of traffic, measure the actual conversion delta) rather than a decision made purely on the spreadsheet. If the elasticity assumption were even slightly optimistic, this ships negative.

What would change the answer

• A larger baseline revenue (the same 5% lift is worth more on a bigger base) tips it positive without changing anything else.
• A confirmed, measured elasticity from a canary experiment replaces the illustrative 0.5% per 100 ms figure with a real one.
• A materially lower or higher risk-adjusted reliability cost (the 3x infrastructure is also more than 3x more complex to operate) shifts the true cost side of the equation.
• A non-revenue reason, such as a contractual service-level agreement (SLA) or a strategic customer who explicitly asked for this, can justify shipping even at a break-even or slightly negative revenue case.

Trade-offs & pitfalls

• Pitfall: treating the latency-to-revenue elasticity as a known constant instead of an assumption to validate; shipping a 3x cost change on an invented number is the actual failure mode this question is testing for.
• Pitfall: ignoring the complexity side of the cost. Three times the infrastructure is usually more than three times the operational surface area (more failure modes, longer incident diagnosis), and that risk has a cost even when nothing has broken yet.
• A break-even or narrowly-positive case is a signal to run a smaller, reversible experiment, not to commit fully in either direction.
• Don't ignore alternatives: a targeted optimization for only the highest-value request paths, or a hybrid where only certain traffic gets the expensive treatment, sometimes captures most of the benefit at a fraction of the cost.