Architectural Patterns and Anti-Patterns Questions

Direct answer

A realistic case is a small product team that adopted backend-for-frontend (BFF), one dedicated API service per client type, because it was "the recommended pattern when you have multiple frontends". BFF solves a real problem: very different clients, owned by separate teams, that need very different data shapes. This team had none of those forces. Its web and mobile apps showed nearly the same screens and were built by the same engineers, so the pattern mostly bought triplicated code, drift between clients and extra operational load. The mismatch was visible early in measurable signals: nearly identical endpoints across the BFFs, business rules leaking into them, and every feature needing several pull requests.

This is the golden-hammer anti-pattern: choosing a familiar or fashionable tool before naming the problem it is meant to solve.

The case (story skeleton)

Context. A seven-engineer team builds a B2B (business-to-business) ordering product: a web app for office staff and iOS and Android apps for field sales. One core API service owns orders, pricing and customers.

Decision. Early on, the team adds three BFF services (web, iOS, Android), each aggregating calls to the core API and shaping responses for its client. The design doc cites the pattern and large companies using it; it does not state which client need the core API fails to meet.

What happened over the following months.

• Change amplification: a new field on the order screen needed a change in the core API plus the same change in three BFFs: four pull requests, four deploys, four sets of tests for one user story.
• Logic drift: "just for now", a discount-eligibility check was added in the web BFF to unblock a release. The mobile BFFs never got it, so mobile users briefly saw prices that the web app correctly refused. The bug came from business logic living in an aggregation layer, which a BFF is not supposed to hold.
• Operational load: three extra services to patch, scale, monitor and put on call, for a team with no dedicated platform engineer.
• Latency: every call now crossed an extra network hop for responses the core API could have served directly.

Remediation. The team collapsed the three BFFs into the core API, which gained optional field selection (clients request the fields they need) and a couple of screen-oriented composite endpoints. The discount rule moved into the pricing domain code where every client gets it. A single BFF was kept later, but for a genuinely different consumer: a partner integration with its own authentication, rate limits and response format.

Why it was the wrong fit

Every pattern resolves specific forces. Checking them against this context:

With none of the forces present, the pattern contributed only its costs: more deployables, more duplicated code and a new place for business logic to hide. The value of a pattern is always relative to the problem; out of context, what remains is its overhead.

Signals that would have exposed the mismatch earlier

1. The design doc named the pattern before naming the problem. A good architecture decision record (ADR, a short document capturing context, decision and consequences) states the forces first. "Because Netflix does it" is not a force.
2. Endpoint similarity. Diffing the three BFFs' routes and response shapes would have shown them to be nearly identical. If most endpoints are copies, the per-client split has no purpose.
3. Change amplification per story. Tracking how many repositories or services a typical feature touches: one story needing four pull requests is a structural smell, not a process one.
4. Business logic outside the domain. Any if about pricing, eligibility or permissions in an aggregation layer is a warning; a lint rule or review checklist can flag it.
5. Ownership mismatch. In Sam Newman's original framing, a BFF is ideally owned by the team that owns the client. One team owning all of them is a hint the split is not buying autonomy.
6. Operational cost against team size. Services per engineer rising without a matching rise in independent release cadence.

A lightweight guard against golden hammers

Before adopting any architecture-level pattern, answer four questions in writing:

1. What specific force or pain does it resolve, and is that pain happening now?
2. What does it cost to build, run and staff?
3. What is the simplest alternative, and why is it insufficient?
4. How hard is it to reverse, and what signal would tell us to reverse it?

If question 1 has no concrete answer, defer the pattern. Adding a BFF later, when a genuinely divergent client appears, is cheap; removing three that have accumulated business logic is not.

Trade-offs and pitfalls

• The overcorrection: refusing all patterns leads to the opposite smell, one general API contorted to serve incompatible clients. The point is fit, not avoidance.
• Fixing the symptom: moving the discount rule into all three BFFs "for consistency" would have hidden the drift while cementing the duplication.
• Sunk cost: teams keep an ill-fitting pattern because removing it feels like admitting a mistake. Treat reversals as a normal outcome of an ADR with an explicit reversal signal.

Direct answer

Put each cross-cutting concern at the outermost place that has enough information to do it correctly, and make that place shared infrastructure rather than per-service code. Concretely: verify end-user identity and enforce per-client rate limits once at the edge (an API gateway), secure service-to-service traffic in the platform (a service mesh: a dedicated network layer, usually a fleet of per-service proxies plus a shared control plane, that handles this uniformly, or equivalent), and keep only the parts that need business context inside each service, delivered through one shared, versioned middleware library. Consistency then comes from a paved-road service template plus automated conformance checks, not from asking every team to remember.

Cross-cutting concern means behaviour that applies to nearly every request regardless of what the request does (identity, throttling, tracing, logging). Copying it into every service is how you get the drift the question describes.

The placement options

Where each of the three concerns should live

Authentication (who is calling).

• Edge: the gateway validates the end-user token (a JWT, JSON Web Token: a signed token carrying the user's identity and expiry) once, rejects bad ones, and forwards the verified identity.
• Between services: mTLS (mutual TLS, where both sides present certificates so each proves its identity) issued by the mesh, so a service knows which service is calling it.
• Inside the service: a thin shared middleware still checks that the forwarded identity is present and meant for this service. This follows zero trust: never trust a request merely because it arrived from inside the network.
• Authorization ("may this user read this invoice?") stays in the service, because only the service knows who owns the invoice. Pushing that into the gateway is how gateways grow into god services (services that have absorbed so many unrelated responsibilities that no one can change them safely).

Rate limiting.

• Per-client quotas ("each API key gets 600 requests per minute") belong at the gateway, backed by a shared counter.
• Self-protection limits ("this service accepts at most 200 concurrent requests") belong in the service or its sidecar, because they depend on that service's own capacity.

Tracing (recording each request's path across services as a tree of timed spans, where each span is one unit of work, such as one service handling one call, with its own start time and duration).

• It must be partly in-process. A sidecar can time each network hop, but it cannot see work inside the process, and it cannot connect an inbound request to the outbound calls it causes unless the application forwards the trace context header (the W3C traceparent header, which carries the trace ID and current span ID so the next hop can attach itself to the same trace). So the shared library must include OpenTelemetry (the vendor-neutral standard and SDK for traces, metrics and logs) instrumentation that propagates context automatically.

flowchart LR
    C[Client] --> G[API gateway: verify JWT, per-client quota]
    G --> P1[Sidecar: mTLS]
    P1 --> S1[Service A: shared middleware, authz, tracing SDK]
    S1 --> P2[Sidecar: mTLS]
    P2 --> S2[Service B: shared middleware, authz, tracing SDK]

Worked example: two numbers that decide placement

Distributed rate limits. A 600 requests-per-minute limit per API key is configured on 3 gateway replicas, each counting locally. A client whose traffic is spread round-robin across replicas can send up to 3 x 600 = 1,800 requests per minute before any replica says no. Fixes: keep the counter in a shared store such as Redis (a fast, shared in-memory data store), or give each replica 600 / 3 = 200 and accept that uneven routing will throttle some clients early. Placing the limit "in the gateway" is not enough; where the counter lives is part of the design.

Patch latency. Say 40 services depend on an auth library and a token-validation bug is found. If teams upgrade at about 5 services a week (an assumption; measure your own), full coverage takes 40 / 5 = 8 weeks, during which some services are still vulnerable. The same fix in the gateway is one deploy. That asymmetry is why token validation belongs at the edge and the library only double-checks.

Keeping it consistent as services are added

1. Paved road (a default, pre-wired path that is easier to follow than to deviate from). New services start from a template that already wires in the middleware, the tracing SDK and the mesh annotations (configuration labels attached to a service that tell the mesh's control plane how to handle its traffic). The easy path is the correct path.
2. Minimum-version policy. CI (continuous integration) fails a build whose shared-library version is older than the supported floor, which turns skew into a visible, bounded window.
3. Policy as code. Gateway routes, rate-limit rules and mesh authorization policies live in one reviewed repository, not in consoles.
4. Conformance tests. An automated check hits every newly registered service without a token and expects 401, and checks that its spans appear in the trace backend. This is an architecture fitness function: an automated test of an architectural property.
5. Clear ownership. A platform team owns the gateway, mesh and library; product teams own authorization and business rules.

Trade-offs and pitfalls

• Gateway creep. Once authentication lives in the gateway, teams add request transformation, then business rules, and the gateway becomes a god service every team must change. Keep it domain-agnostic.
• Edge-only security. If services trust any request that reached them, one compromised internal service can impersonate everyone. Keep mTLS and in-service identity checks.
• Mesh adoption for one concern. Adopting a service mesh only to get tracing is golden-hammer thinking (reaching for one favourite tool for every problem, regardless of fit); the SDK does most of that. Adopt a mesh when you need uniform mTLS and traffic policy (rules for how requests are routed, retried and load-balanced) across many languages.
• What would flip the recommendation: a small system with 3 services in one language does not need a gateway or mesh; a single shared middleware library plus a service template is enough.

Direct answer

Two services are tightly coupled when one cannot change, deploy or even stay up without the other: it depends on the other's internal details (field names, database tables, timing, availability). They are loosely coupled when each depends only on a small, stable, published contract and can tolerate the other being slow, down, or on a different version. It matters for shipping because coupling decides the unit of release: if Team A's change forces a change or a coordinated deploy in Team B's service, the two teams move at the speed of their slowest shared calendar. To loosen coupling I use stable versioned contracts, events instead of synchronous calls where the caller does not need an immediate answer, and each service owning its own data. To measure it I combine three views: the static dependency graph, the runtime call graph from traces, and change history.

The kinds of coupling (it is not one thing)

A useful mental model: coupling is the number of things the other side could change that would break you. Loosening coupling shrinks that list.

Techniques to loosen an existing tightly coupled system

1. Put an explicit, versioned contract in front of every dependency. Replace direct table reads with an API or event, version it, and make changes additive only (add fields, never rename or remove in place).
2. Consumer-driven contract tests. Each consumer declares the fields and behaviours it relies on; the provider runs those checks in its own build, so it learns before deploying whether it breaks anyone. This is what lets deploys become independent.
3. Move from synchronous calls to events where no immediate answer is needed. The publisher announces a fact ("OrderPlaced") and does not know or care who listens.
4. Give each service its own data, with local read copies. If Shipping needs a customer's address, it keeps its own copy updated from CustomerAddressChanged events instead of calling Customer on every shipment.
5. Anti-corruption layer (a translation layer at the boundary): when you must integrate with a messy legacy model, translate it into your own model at the edge so its quirks do not spread into your code.
6. Tolerant reader. Consumers ignore unknown fields and do not depend on field order, so providers can add fields freely.

Choosing between REST, event-driven and async messaging: a worked example

Scenario: an e-commerce platform where Checkout currently calls Loyalty, Email, Analytics and Inventory synchronously during order placement.

Before: order placement had 4 synchronous downstream dependencies, so an Email outage failed checkouts. After: 1 synchronous dependency (Inventory), and Email being down only delays receipts. The distinction that drives the choice is "does the caller need the result to continue?" plus "is this a fact for anyone, or a command for one worker?".

Measuring coupling across a codebase or service map

View 1: static dependency graph (what the code declares)

Build the graph of which service or module depends on which, from build files, imports or client libraries. For each node compute:

• Ca (afferent coupling): how many others depend on it.
• Ce (efferent coupling): how many it depends on.
• Instability I = Ce / (Ca + Ce), from 0 (everyone depends on it, it depends on nothing) to 1 (it depends on everything, nothing depends on it).

Example: Checkout depends on 6 services and 2 depend on it, so I = 6 / (2 + 6) = 0.75, close to 1 (fully unstable). That is fine for a top-level orchestrator (a service like Checkout that coordinates a business process by calling out to several others, rather than being called deep inside someone else's process): nothing else in the system depends on it in any deep way (Ca = 2 is low), so a change to Checkout has nowhere further outward to ripple; being "unstable" just means it is free to change because it sits at the edge of the graph, not the centre.

The warning sign is a shared service everyone depends on (high Ca) that also has high Ce, and it is worth working a number for: suppose a shared Pricing service has Ca = 8 (eight other services call it) and Ce = 5 (it itself calls five other services or data sources, for tax rates, currency conversion and so on). Its instability is I = 5 / (8 + 5) ≈ 0.38, a middling number that looks unremarkable if you only glance at I. The two raw counts are the real story: high Ca means eight other services need to change, or at least be retested, whenever Pricing's contract changes (hard to change safely), and high Ce means Pricing itself is exposed to five other services' changes rippling into it (likely to change often). A healthy shared service instead aims for high Ca with low Ce, an I closer to 0.1-0.2: many dependents, but few dependencies of its own that could knock it over. Also flag cycles in this graph; every cycle is a set of services that can no longer be released in any safe order.

View 2: runtime call graph (what actually happens)

Distributed tracing, instrumentation that tags every call belonging to one request with a shared id so the calls across every service can be reassembled afterward into one end-to-end record (for example OpenTelemetry, an open standard for doing this), records every call a request makes. From traces, build the live service map and track per endpoint:

• Fan-out: downstream calls per inbound request.
• Synchronous depth: longest chain of blocking calls.

This catches coupling that static analysis misses, such as calls built from configuration, or a service calling another 40 times in a loop.

View 3: change history (how the code actually evolves)

From version control, compute temporal coupling: how often two services change in the same ticket or release. High temporal coupling with no declared dependency is the most interesting finding, because it means a hidden dependency (a shared schema, a duplicated rule).

Put all three on one dashboard. Each alone misleads: the static graph shows declared dependencies, traces show runtime behaviour, history shows what actually slows teams down.

Trade-offs and pitfalls

• Loose coupling has a price. Events give you eventual consistency (subscribers see the change a little later), harder debugging across asynchronous hops, and message schemas that still need versioning. Use them where decoupling is worth that, not everywhere.
• Async is not automatically loose. An event that carries the publisher's full internal entity couples every subscriber to that entity's shape.
• Over-abstracting. A generic "universal integration layer" in front of everything becomes its own tightly coupled god component: one component that has absorbed so much shared responsibility that everything else ends up coupled through it.
• Cohesion is the other half. Low coupling between services only helps if each service is cohesive (its parts belong together). Splitting a cohesive unit apart raises coupling, however many events you add.

Direct answer

Keep one codebase and one release train for every customer (a release train is a fixed release schedule that ships whatever is ready on time, instead of a separate release per customer) and never fork for a customer. Offer customization as a ladder, cheapest rung first: schema-validated configuration, then feature flags and entitlements, then extension points through a small published plugin or webhook API (a webhook is a URL the customer gives you that your system calls, with an event's data, when something happens). Only when a request cannot be met on those rungs does it become either a product feature (if other customers want it too) or a paid, out-of-process integration that the customer's code runs against your stable API. Release management then treats each customer's configuration and extensions as test fixtures that every release candidate (a build that is a candidate for shipping, pending final checks) must pass.

The customization ladder

Worked example: a custom invoice-approval flow

A customer wants invoices above 50,000 USD to need two approvers, and wants their own budget system consulted before approval.

1. Configuration covers the first half. The product already supports approval rules, so add approval.threshold_amount and approval.required_approvers to the schema (integer, minimum 1, maximum 5). The customer's config sets 50,000 and 2. No code is specific to this customer.
2. An extension point covers the second half. Add a beforeInvoiceApproval hook that calls a customer-registered webhook with the invoice summary and expects allow or deny. Define its contract: a 3-second timeout, a signed request, and a documented default when the webhook fails (hold the invoice for manual review, never auto-approve). The customer's integration code runs on their side, not in your core.
3. Productize the hook. Other customers can now register the same hook, so the work becomes a product feature rather than a one-off.

Why not forks, and why flags need discipline: the arithmetic

• Forks. With 5 customer forks and 24 releases a year, each release must be merged and regression-tested 5 extra times: 5 x 24 = 120 extra merge-and-test cycles a year, on top of the 24 for the main line. The work grows with customers times releases, which is why forked products stall.
• Flags. 12 independent on/off flags allow 2 to the power 12 = 4,096 combinations; you cannot test them all. So test the combinations that actually exist: each real customer's configuration becomes a named profile, and a mandatory expiry date on release flags keeps the count from growing without bound.

Release management

1. Single trunk (everyone commits to one shared mainline, with no long-lived customer branches), one version for everyone, with customizations carried as data (config, flags) and as extensions against a versioned API.
2. Versioned extension API. Semantic versioning (version numbers written MAJOR.MINOR.PATCH, where a MAJOR bump means a breaking change) for the hook contracts, a published deprecation window (for example two major releases), and a compatibility test suite extension authors can run.
3. Customer profiles in CI. Each release candidate runs the regression suite (the automated tests that check old behaviour still works) against every significant customer's configuration and their registered extension contracts (with the webhooks stubbed, meaning replaced with a fake that returns their documented response instead of calling the customer's real endpoint).
4. Progressive rollout in rings (releasing to successively larger groups instead of everyone at once): internal tenants (your own company's teams, using the product first as tenants of the same shared system customers will later use), then early-adopter customers, then everyone, with flags as the kill switch (a flag you can flip to turn the feature back off instantly, without a deploy).
5. Customization governance. Every custom request is reviewed against the ladder: can it be config, a flag or a hook? Anything needing core changes for one customer needs explicit product sign-off. Retire flags on a schedule.

Trade-offs and pitfalls

• Configuration as a programming language. When config grows conditionals and loops, you have built an untested scripting language; move that logic to a proper extension point.
• In-process plugins with full access. Customer code in your process can crash it, leak other customers' data or block upgrades. Prefer out-of-process extensions unless latency truly requires in-process, and then use a narrow, sandboxed interface.
• Leaky extension contracts. A hook that passes internal database entities makes your schema a public API; pass a documented, versioned model.
• Golden hammer (reaching for one familiar pattern regardless of whether it fits the problem) in either direction. Building a plugin framework before a second customer needs it is premature; saying yes with an if customer == "acme" branch is the start of a fork. Build the extension point when the second similar request arrives.
• What would flip the approach: a single, very large customer paying for a dedicated deployment might justify a separate instance (still the same code, with its own configuration and release timing), but not a code fork.

Direct answer

A distributed monolith is a system that is split into separately deployed services but still behaves like one program: the services cannot be changed, tested or released independently. You pay every cost of a distributed system (network latency, partial failures, harder debugging) and get none of the main benefit, which is independent teams shipping independently. Two strong signs are lockstep deployments (a feature routinely requires several services to release together, in a specific order) and long synchronous call chains (one user request cannot complete unless many services are all up at the same moment). A concrete first mitigation is to make deploys independent with versioned, backward-compatible APIs checked by consumer-driven contract tests, so each service can ship without waiting for the others.

What it looks like

Think of an online store split into Cart, Pricing, Inventory, Orders and Payments. On paper these are five microservices. In practice:

• Adding a "gift wrap" option means changing all five, and the release plan is a spreadsheet saying "deploy Pricing first, then Orders, then Cart, within 30 minutes, or checkout breaks".
• Placing an order calls Cart → Pricing → Inventory → Orders → Payments, one after another, synchronously.
• Orders and Inventory read and write the same products table.

Sign 1: lockstep deployment

What you observe: releases are coordinated across services; there is a shared release train (a fixed, recurring release schedule that several services are bundled onto and ship together, rather than each shipping on its own whenever its change is ready) or a "deploy order" document; rolling back one service requires rolling back others.

Why it happens: services share a data model or API that changes without versioning. If Pricing renames a field in its response, Cart breaks until Cart is updated, so they must deploy together.

How to detect it: from the deploy log, compute the fraction of releases in which a service was deployed within the same short window as another service for the same ticket. If Cart ships with Pricing in most of its releases, those two are one deployable in disguise.

Sign 2: synchronous call chains that multiply failure

What you observe: a distributed trace (a record, stitched together from the timestamped spans each service emits for the same request, showing every service that request touched and how long each one took) of one request shows a deep chain of blocking calls, and an outage in any one service takes down the whole user flow.

Why it matters, with numbers: if a request needs 5 services in series and each is independently available 99.9% of the time, the chain is available only when all 5 are up:

• 0.999 × 0.999 × 0.999 × 0.999 × 0.999 = 0.999^5 ≈ 0.99501, about 99.50%.
• Over a 30-day month (30 × 24 × 60 = 43,200 minutes), one service at 99.9% is down about 0.001 × 43,200 = 43.2 minutes.
• The 5-service chain is down about (1 − 0.99501) × 43,200 ≈ 215.6 minutes, roughly five times as much.

A monolith would have had one component's worth of downtime; the distributed monolith has five, with the same features. (This simple model assumes failures are independent; correlated failures change the number but not the lesson.)

Other signs worth naming

• Services sharing a database schema, so a column change needs coordination.
• A shared library of domain objects that every service must upgrade together.
• End-to-end tests that need every service running before anyone can merge.

The mitigation: break deploy coupling first

Why start here: lockstep deploys are the most expensive symptom (they slow every team, every release) and fixing them is incremental. Steps:

1. Version the contract. API changes become additive: add a new field, never rename or remove one in place. Consumers use a tolerant reader (ignore fields you do not recognise, do not fail on missing optional ones).
2. Add consumer-driven contract tests. Each consumer (Cart) publishes the exact requests and response fields it depends on; the provider (Pricing) runs those contracts in its own CI. Tools such as Pact implement this. Now Pricing learns in its own build, before deploying, whether it would break Cart.
3. Deploy in expand-then-contract order. To rename price to unit_price: Pricing ships a version returning both; Cart switches to unit_price whenever it likes; once contracts show no consumer reads price, Pricing removes it. Three independent deploys, no coordination window.
4. Measure the result. Track the lockstep share from the deploy log. If Cart and Pricing co-deployed in 8 of 10 releases before, the target is near zero within a couple of quarters.

If the contracts reveal that two services cannot evolve separately no matter what (they change together on every feature), the honest fix is to merge them back into one service. A boundary that always moves together was never a real boundary.

Pitfalls

• Fixing the symptom with tooling. A better release-orchestration tool makes lockstep deploys easier to run, which hides the coupling instead of removing it.
• Adding more services. Splitting further to "fix" coupling usually adds more edges.
• Replacing sync calls with async messages carrying the same shared data model. The deploy coupling survives; it just moved into the message schema.