Architectural Patterns and Anti-Patterns Questions

Direct answer

A three-tier architecture splits a web system into a presentation tier (what renders the user interface), an application tier (stateless business logic and APIs) and a data tier (databases, caches, object storage, anything that holds state). As an SRE I design each tier as its own isolation, scaling and failure domain: only the edge is reachable from the internet, each tier accepts traffic only from the tier directly in front of it, the first two tiers are stateless so they scale horizontally and can be replaced freely, and all shared state lives in the data tier. The data tier is the one that is hard to scale and hard to fail over, so most of the design effort goes into protecting it.

Ingress routing mapped to tiers

flowchart LR
  U[Users] --> DNS[DNS] --> CDN[CDN + WAF]
  CDN -->|/static, SPA bundle| OBJ[(Object storage)]
  CDN -->|/ and /api| ALB[Public L7 load balancer]
  ALB -->|/ server-rendered pages| WEB[Presentation tier: web servers]
  ALB -->|/api| APP[Application tier]
  WEB -->|internal LB| APP
  APP --> PGB[Connection pooler] --> DB[(Primary DB + replicas)]
  APP --> RED[(Redis: sessions, rate limits)]

• DNS to CDN (content delivery network) with a WAF (web application firewall, which filters malicious requests) at the edge. Static assets and the single-page-app bundle are served from object storage through the CDN and never touch a server.
• One public layer-7 load balancer (it routes on HTTP paths and hosts) is the only internet-facing component. Path rules map to tiers: / goes to the presentation tier's target group (the load balancer's list of healthy backend instances it is allowed to route a request to), /api goes to the application tier's target group.
• I commit to letting /api go straight from the public load balancer to the application tier, with the application instances still in private subnets (network segments with no route from the internet). Forcing API calls through a web tier that only proxies them adds a hop and a failure point for no isolation gain, because the load balancer is already the controlled entry point. If the presentation tier does server-side rendering, it calls the application tier through an internal load balancer.
• The data tier has no ingress from the load balancers at all.

Isolation

Scaling each tier

• Presentation and application tiers: stateless, horizontally autoscaled on requests per instance or CPU.
• Data tier: scales vertically first, then with read replicas for read traffic, caching for hot keys (a small number of records or cache entries that receive disproportionately more reads or writes than the rest), and only much later sharding (splitting the data itself across multiple databases so no single one holds all of it). Its limit that bites first is usually connections, not CPU.

Worked example, capacity with zone failure. Peak is 6,000 RPS (requests per second) and one application instance handles 300 RPS at its target utilisation, so peak needs 6,000 / 300 = 20 instances. To survive losing one of three zones, the two surviving zones must carry the full 20, so I run 20 / (2/3) = 30 instances, 10 per zone, which means about 67% utilisation of that target in normal operation.

Worked example, connection budget. 40 application pods (the deployable unit in Kubernetes: one or a few containers scheduled and scaled together) with an ORM connection pool of 20 each could open 40 x 20 = 800 connections, while Postgres ships with max_connections = 100 by default and each connection is a separate server process with its own memory. Autoscaling the application tier would therefore take the database down. Fix: a connection pooler (a proxy that sits between the application and the database and multiplexes many client connections onto a much smaller number of real database connections) such as PgBouncer in transaction mode (it hands a client a real database connection only for the duration of one transaction, then returns it to the shared pool immediately afterward, instead of holding one real connection open per client for the whole session) between the tiers, so 800 client connections share a few dozen server connections, and a cap on application autoscaling tied to that budget.

What belongs in which tier

Rule of thumb: if losing an instance would lose something a user cares about, that thing is in the wrong tier.

Security boundaries

• TLS (encryption in transit) terminates at the load balancer; traffic is re-encrypted to the tiers behind it, or uses mTLS (mutual TLS, where both sides present certificates) if a service mesh (a dedicated network layer, usually a fleet of per-service proxies plus a shared control plane, that handles encryption, retries and routing between services without changes to application code) is in place.
• Authentication checks at the application tier on every request, even from the presentation tier, so a compromised web server cannot act as an arbitrary user.
• Database accounts are least privilege: the application role can read and write tables but not alter schema.
• Blast radius: a compromised presentation-tier host can reach only the application tier's API port, not the database.

Trade-offs and pitfalls

• Sticky sessions (routing every request from one user's session to the same backend instance) as a scaling shortcut: they make scale-in (removing instances as load drops) and deploys disruptive and hide the fact that state is in the wrong tier.
• Treating the database as infinitely elastic: the application tier can multiply its instance count in minutes; the database cannot follow.
• Tier count as dogma: a separate presentation tier is worth it for server-side rendering; for a pure single-page app, the CDN effectively is the presentation tier.
• Shared cache as hidden coupling: once several services read the same Redis keys, the cache has become a shared database, with the same coupling problems.

Direct answer

A distributed monolith is a system that is split into separately deployed services but still behaves like one program: the services cannot be changed, tested or released independently. You pay every cost of a distributed system (network latency, partial failures, harder debugging) and get none of the main benefit, which is independent teams shipping independently. Two strong signs are lockstep deployments (a feature routinely requires several services to release together, in a specific order) and long synchronous call chains (one user request cannot complete unless many services are all up at the same moment). A concrete first mitigation is to make deploys independent with versioned, backward-compatible APIs checked by consumer-driven contract tests, so each service can ship without waiting for the others.

What it looks like

Think of an online store split into Cart, Pricing, Inventory, Orders and Payments. On paper these are five microservices. In practice:

• Adding a "gift wrap" option means changing all five, and the release plan is a spreadsheet saying "deploy Pricing first, then Orders, then Cart, within 30 minutes, or checkout breaks".
• Placing an order calls Cart → Pricing → Inventory → Orders → Payments, one after another, synchronously.
• Orders and Inventory read and write the same products table.

Sign 1: lockstep deployment

What you observe: releases are coordinated across services; there is a shared release train (a fixed, recurring release schedule that several services are bundled onto and ship together, rather than each shipping on its own whenever its change is ready) or a "deploy order" document; rolling back one service requires rolling back others.

Why it happens: services share a data model or API that changes without versioning. If Pricing renames a field in its response, Cart breaks until Cart is updated, so they must deploy together.

How to detect it: from the deploy log, compute the fraction of releases in which a service was deployed within the same short window as another service for the same ticket. If Cart ships with Pricing in most of its releases, those two are one deployable in disguise.

Sign 2: synchronous call chains that multiply failure

What you observe: a distributed trace (a record, stitched together from the timestamped spans each service emits for the same request, showing every service that request touched and how long each one took) of one request shows a deep chain of blocking calls, and an outage in any one service takes down the whole user flow.

Why it matters, with numbers: if a request needs 5 services in series and each is independently available 99.9% of the time, the chain is available only when all 5 are up:

• 0.999 × 0.999 × 0.999 × 0.999 × 0.999 = 0.999^5 ≈ 0.99501, about 99.50%.
• Over a 30-day month (30 × 24 × 60 = 43,200 minutes), one service at 99.9% is down about 0.001 × 43,200 = 43.2 minutes.
• The 5-service chain is down about (1 − 0.99501) × 43,200 ≈ 215.6 minutes, roughly five times as much.

A monolith would have had one component's worth of downtime; the distributed monolith has five, with the same features. (This simple model assumes failures are independent; correlated failures change the number but not the lesson.)

Other signs worth naming

• Services sharing a database schema, so a column change needs coordination.
• A shared library of domain objects that every service must upgrade together.
• End-to-end tests that need every service running before anyone can merge.

The mitigation: break deploy coupling first

Why start here: lockstep deploys are the most expensive symptom (they slow every team, every release) and fixing them is incremental. Steps:

1. Version the contract. API changes become additive: add a new field, never rename or remove one in place. Consumers use a tolerant reader (ignore fields you do not recognise, do not fail on missing optional ones).
2. Add consumer-driven contract tests. Each consumer (Cart) publishes the exact requests and response fields it depends on; the provider (Pricing) runs those contracts in its own CI. Tools such as Pact implement this. Now Pricing learns in its own build, before deploying, whether it would break Cart.
3. Deploy in expand-then-contract order. To rename price to unit_price: Pricing ships a version returning both; Cart switches to unit_price whenever it likes; once contracts show no consumer reads price, Pricing removes it. Three independent deploys, no coordination window.
4. Measure the result. Track the lockstep share from the deploy log. If Cart and Pricing co-deployed in 8 of 10 releases before, the target is near zero within a couple of quarters.

If the contracts reveal that two services cannot evolve separately no matter what (they change together on every feature), the honest fix is to merge them back into one service. A boundary that always moves together was never a real boundary.

Pitfalls

• Fixing the symptom with tooling. A better release-orchestration tool makes lockstep deploys easier to run, which hides the coupling instead of removing it.
• Adding more services. Splitting further to "fix" coupling usually adds more edges.
• Replacing sync calls with async messages carrying the same shared data model. The deploy coupling survives; it just moved into the message schema.

Direct answer

Those three symptoms point to six suspects, including one that is worth checking even though it does not map cleanly to a single symptom below. Lockstep releases suggest a distributed monolith: services that are packaged and deployed separately, so they look like microservices, but are coupled tightly enough that they still have to change and ship together, the coordination cost of a monolith without the deployment independence a microservice split is supposed to buy. Cascading pages (one service's trouble tripping alerts through several other services it calls, or that call it, in a chain) suggest chatty services and long synchronous call chains, possibly with cyclic dependencies. A god service, one service that has absorbed so many responsibilities that it shows up in almost every trace and almost every incident, is that additional suspect: it can force lockstep releases because every dependent team waits on the team that owns it, and it can make pages cascade because it sits on the path of most requests. A schema change breaking two other services points to shared-database coupling or a leaky abstraction (an interface meant to hide a service's internal details, that lets them show through anyway) that exposes one service's internal schema to others. Confirm each with evidence from traces, deploy history and database access logs before acting, then fix incrementally, starting with the one that causes the most pages. Do not propose a rewrite.

Symptom to anti-pattern, detection signal and first remediation step

Span: one timed operation in a distributed trace, such as one HTTP call. Trace: the tree of spans produced by one user request.

Worked example: putting numbers on the cascade

A checkout trace shows one user request producing 23 spans (a span is one timed operation in the trace, such as one HTTP call; 23 means some services are called more than once) across 7 services, with 6 of the calls made one after another, which is what makes the request slow. Separately from that latency picture, checkout's correctness depends on all 7 services succeeding, whether a given call sits on the sequential chain or not: if any one of them fails, the checkout fails. Suppose each of the 7 services is available 99.9% of the time and checkout needs all of them to succeed.

• Chained availability: 0.999 raised to the 7th power (one factor per required service) = 0.99302, about 99.30%. That looks like a small drop from 99.9%, because both numbers are close to 100%.
• Over a 30-day month (43,200 minutes), that is (1 - 0.99302) x 43,200, about 301 minutes of failed checkouts, versus 0.001 x 43,200 = 43.2 minutes for a single 99.9% service. 301 / 43.2 is about 7: it is the expected downtime, not the availability percentage, that scales roughly with the number of required services.

So a design where every request synchronously depends on seven services has about seven times the expected downtime of any one of them (301 minutes a month versus 43.2), before even counting the extra latency the 6 sequential calls add on top, even though the availability percentages themselves look close (99.30% versus 99.9%). The 23-span, 6-sequential-call shape is why the request is also slow; the 7-services-required shape is why it is fragile. That is the architectural reason on-call keeps getting paged: the dependency structure multiplies failure, and no single team's service looks broken.

For deploy coupling, pull the last quarter's release records. If, for example, 31 of 50 releases needed a coordinated deploy, that 62% is the number to drive down, and the co-change data shows which service pairs to fix first.

Sequencing the remediation

The aim is to reduce risk and restore delivery speed without a big-bang rewrite:

1. Measure first. Record the baseline: deploy-coupling rate, pages per week by root-cause service, median spans per key request, number of services with write access to each shared table.
2. Stop the bleeding on pages. Fix the worst chatty path or break the cycle on the most-paged flow; these are local changes with fast payback.
3. Stop new coupling. Add contract tests in CI, and a rule that no new service gets credentials to another service's tables. This keeps the smell from recurring while you pay down the old debt.
4. Pay down the shared database table by table, starting with the tables whose migrations caused incidents.
5. Report progress in delivery terms: lead time for changes (the time from a commit to it running in production) and change failure rate (the share of production changes that cause an incident), two of the DORA metrics (the software delivery measures from the DevOps Research and Assessment programme), plus the deploy-coupling rate.

Pitfalls

• Armouring instead of fixing. Timeouts and circuit breakers (a guard that stops calling a failing dependency for a while, instead of piling retries onto a call that keeps failing) limit the blast radius (how much of the system one failure can drag down) of a bad dependency but leave the dependency in place; they are not a remediation of the architecture.
• Merging everything back. Sometimes two services that always change together should become one, and that is a legitimate fix, but decide it per pair from co-change evidence, not as a reaction to pain.
• Guessing from symptoms alone. The same "cascading pages" symptom can come from chatty calls or from a cycle; the trace graph tells you which.
• Fixing everything at once. Coordinated remediation across all teams recreates the coordination problem you are trying to remove.

Direct answer

Event sourcing means the system stores every state change as an immutable event ("OrderPlaced", "PaymentCaptured") in an append-only log, and the current state of anything is computed by replaying, or folding (combining events one by one, in order, into a running total), its events. CQRS (Command Query Responsibility Segregation) means writes and reads use different models: commands append events on the write side, and one or more read models (also called projections: tables or indexes built by consuming the events) serve queries.

Operationally that changes three things. Ordering becomes a per-stream guarantee rather than a global one. Reads become eventually consistent, so the lag between an event being written and a read model reflecting it becomes the most important thing I monitor. And the log only grows, which gives me a perfect replayable record for debugging but also a storage bill and a rebuild time that grow forever unless I manage them.

flowchart LR
  C[Client command] --> W[Write model / command handler]
  W -->|append with expected version| ES[(Event store)]
  ES --> P1[Order-list projector]
  ES --> P2[Search projector]
  ES --> R[Email reactor]
  P1 --> RM1[(Read DB)]
  P2 --> RM2[(Search index)]
  Q[Client query] --> RM1

1. Ordering guarantees

• Within one stream (usually one aggregate: a cluster of related data and rules, such as one order and its line items, that is kept consistent as a single unit) order is strict. The write side appends with an expected version: "append event 7 to order-123 only if the stream is currently at version 6". If two commands race, one gets a concurrency conflict and retries. This is optimistic concurrency, and it is what keeps a single order's history coherent.
• Across streams there is usually no total order. If events are published to a partitioned log such as Kafka (a message log split into independent, ordered segments called partitions, so throughput scales by spreading writes and reads across them, though order is only guaranteed within one partition) keyed by aggregate ID, order-123 and order-456 land on different partitions and a projector can see them in either order. Anything that needs "A happened before B" across two aggregates has to carry that relationship explicitly (a causation ID, or a timestamp it is prepared to distrust).
• Projectors track a checkpoint (the last event position they applied). After a crash they resume from the checkpoint, so they will see some events twice. Projection code must therefore be idempotent: applying the same event twice leaves the same result, typically by storing the last applied stream version per row and ignoring anything at or below it.

What this means for an SRE: a spike in concurrency conflicts is a real signal (a hot aggregate: one entity taking far more concurrent writes than typical, so its version check fails repeatedly; or a client retry storm: a burst of automatic retries from a client that amplifies load instead of backing off), and "the read model skipped or double-counted something" is almost always a checkpoint or idempotency bug, not a database bug.

2. Replayability

Because the events are the source of truth, any read model can be dropped and rebuilt from position zero. That is the superpower: a buggy projection is fixed by fixing the code and replaying, not by hand-editing rows.

The operational costs are real, though:

• Rebuild time. Worked example: an event store holding 2 billion events and a projector that applies 5,000 events per second needs 2,000,000,000 / 5,000 = 400,000 seconds, which is about 4.6 days. If the projection is partitioned by aggregate and 32 projector workers run in parallel, the same rebuild is 400,000 / 32 = 12,500 seconds, about 3.5 hours. That number belongs in the runbook, because it is effectively the recovery time for a corrupted read model.
• Side effects must not replay. A projector that updates a table is safe to replay; a reactor that sends an email or calls a payment provider is not. Keep them as separate consumers with separate checkpoints, and never reset a reactor's checkpoint as part of a rebuild.
• Rebuild without downtime. Build the new projection into a new table (a blue/green switch: run the new version alongside the old one and only redirect reads once it has caught up), let it catch up to the head of the log (the most recently written position, i.e. the newest event), compare it against the old one, then switch reads over.

3. Storage growth

Nothing is ever deleted, so growth is linear in write volume. At an average of 1,000 events per second and 1 KB per event: 1,000 x 86,400 = 86.4 million events per day, about 86.4 GB per day, or roughly 31.5 TB per year before replication and before the read models, which are extra copies. Controls:

• Snapshots: periodically store an aggregate's folded state so loading it does not require replaying its whole history.
• Tiering: move old, closed streams to cheaper storage while keeping them replayable.
• Personal data: "delete this user" collides with "never delete events". A common answer is crypto-shredding (encrypt each user's personal fields with a per-user key and delete the key), which should be designed in on day one, not retrofitted.

Monitor the growth rate against the forecast; a sudden jump usually means a new event type is far larger than expected or a client is emitting duplicate commands.

4. What I monitor

A key point: request metrics on the API can be perfectly green while a projection is stalled, because commands still succeed. Lag is the metric that catches that.

5. Debugging an incident

Worked example: support reports "customer paid, but the order page still says AWAITING_PAYMENT".

1. Is the event there? Query the order's stream. If PaymentCaptured is missing, the problem is on the write side (the command failed or was never sent). Follow its correlation ID into the tracing system.
2. If the event exists, is the projection behind? Compare the projector's checkpoint with the event's position. If the checkpoint is earlier, this is lag; check whether the projector is stalled on a poison event.
3. If the checkpoint is past the event, the projection logic is wrong. Fold the stream in a scratch environment with the current projection code and diff the result against the stored row. Now the bug is reproducible from real data, which is the biggest debugging advantage event sourcing gives you.
4. Fix forward, never edit history. If the events themselves are wrong (a bug emitted a bad event), append a compensating or correcting event; do not rewrite the log. Then fix the projection and replay the affected streams.

Every event should carry an event ID, stream ID and version, a timestamp, a correlation ID (the request that started the chain) and a causation ID (the event or command that directly caused it). Without those, step 1 is guesswork.

Trade-offs and pitfalls

• Treating eventual consistency as a bug. Users will read their own writes from a stale projection. Decide per screen whether that is acceptable, and where it is not, read from the write side or wait for the projection to reach the version the command returned.
• Skipping the rebuild drill. Teams discover that a full replay takes days only during the incident that needs it. Rehearse it and record the time.
• Letting a projector silently skip failing events. Skipping keeps lag green while the read model quietly diverges. Park the event, alert, and make skipping an explicit, logged decision.
• Global-order assumptions. Code that assumes events across aggregates arrive in wall-clock order works in testing on one partition and breaks in production on many.

Direct answer

A three-tier architecture splits a web application into three independently deployed parts: the presentation tier (what runs in or is sent to the user's browser or app), the application tier (the servers that apply business rules) and the data tier (the databases and stores that keep state). Each tier talks only to its neighbour, so the browser never touches the database directly. You pay for an extra network hop and more moving parts, and in exchange you get a stateless middle tier you can scale by adding servers, one place to enforce security and rules, and parts you can test and deploy separately.

A useful distinction up front: a tier is a physical deployment unit (separate machines or containers), while a layer is a logical grouping of code inside one program. Three-tier is about tiers.

The three tiers and what each owns

Tracing one request: GET /orders/123

1. Client. The browser already loaded the presentation code from the CDN. The user opens "My orders", and the JavaScript sends GET /api/orders/123 over HTTPS with a session token.
2. Ingress and load balancing. DNS resolves the API hostname to a load balancer (or, in Kubernetes (a container-orchestration system: software that runs and manages many server instances across a fleet of machines for you, restarting failed ones and scheduling new ones), an ingress controller: the component that admits outside traffic into the cluster). It terminates TLS (decrypts HTTPS), checks which application instances are passing health checks (a periodic ping each instance must answer, so the load balancer stops sending traffic to one that has stopped responding), and forwards the request to one of them.
3. Application tier. The chosen instance validates the token, checks the rule "a customer may only read their own orders", and asks its data-access code for order 123. It may check a cache first.
4. Data tier. A connection is borrowed from the instance's connection pool (a set of already-open database connections the instance keeps ready, so a request reuses one instead of paying the cost, tens of milliseconds, of opening a fresh connection to the database each time), an indexed SELECT runs against the orders table, and rows come back.
5. Back up the chain. The application maps rows to a JSON response (dropping internal columns), returns 200, the load balancer relays it, and the browser renders the page.

sequenceDiagram
    participant B as Browser
    participant LB as Load balancer
    participant App as App server
    participant DB as Database
    B->>LB: GET /api/orders/123 over HTTPS
    LB->>App: forward to a healthy instance
    App->>App: check token and ownership rule
    App->>DB: SELECT order 123
    DB-->>App: rows
    App-->>LB: 200 JSON
    LB-->>B: 200 JSON

Because step 3 keeps no per-user state in memory, any instance can serve the next request from the same user. That property is what lets the middle tier scale out.

Worked example: where the bottleneck moves

Suppose peak load is 1,200 requests per second (RPS) and a load test shows one application instance sustains about 150 RPS.

• Instances needed: 1,200 / 150 = 8. Run 10 so that losing one or two still leaves capacity (25% headroom).
• Each instance keeps a database connection pool of 20 connections: 10 x 20 = 200 connections.
• PostgreSQL's max_connections default is typically 100 (the documentation notes it can be lower if kernel settings do not support 100).

So the design that scales the application tier by adding boxes will exhaust the data tier at peak, exactly when the autoscaler (the system that automatically adds or removes application instances based on load) adds instances. Fixes: shrink each pool (10 x 8 = 80 connections), or put a connection pooler such as PgBouncer between the tiers: a lightweight service that holds a smaller, fixed number of real connections to Postgres and multiplexes many application-side requests through them. PgBouncer's own default is session pooling, which hands one real connection to an application connection for its whole session and would not shrink the connection count here; the setting this design actually needs is transaction pooling mode, which hands a connection out per transaction and takes it back the instant that transaction commits or rolls back, then reuses it for the next request. PgBouncer also offers a statement pooling mode that returns the connection after every individual query, but that mode explicitly disallows transactions spanning multiple statements, which would break a multi-step operation like a funds transfer, so transaction mode, not the default and not statement mode, is the setting to reach for. That lets the same 200 application-side connections be served through, say, 50 real database connections instead of colliding with the 100-connection ceiling. The general lesson is that the application tier scales horizontally cheaply and the data tier does not, so in a three-tier system the bottleneck usually migrates to the database.

Trade-offs versus two-tier and event-driven designs

A two-tier design has the client talk straight to the database (a desktop app with a database driver, or a small server-rendered app where UI code and SQL live in one process). An event-driven design has components publish events ("OrderPlaced") to a broker such as Kafka (a system dedicated to durably queueing and delivering these events to every interested component, so the publisher does not need to know who is listening or whether they are online right now), and other components react asynchronously instead of being called directly.

Recommendation. Use three-tier as the default for a customer-facing web product. Use two-tier only for small internal tools with a handful of trusted users. Add event-driven pieces for work the user does not need to wait for (emails, search indexing, analytics), which gives a hybrid: synchronous three-tier for the request path, events for side effects. What would flip this: if most work is naturally asynchronous (ingesting sensor data, processing uploads), an event-driven core is the better starting point.

Pitfalls that separate strong answers

• Stateful application servers. Sessions kept in instance memory force "sticky" routing (the load balancer must keep sending one user to the same instance every time, because that instance is the only one holding their session in memory), and users get logged out when an instance dies. Keep sessions in a token or a shared store.
• A pass-through middle tier. If the application tier only forwards requests to SQL, you paid for a hop and got nothing; the rules have leaked into the client or into stored procedures (business logic written and run inside the database itself, rather than in the application tier).
• Confusing tiers with layers. Splitting every code layer onto its own network hop adds latency without adding isolation you need.
• Forgetting that tiers fail together. In a synchronous chain, a slow data tier makes the whole request slow; the architecture does not isolate that by itself.