Architecture Documentation and Communication Questions

Direct answer

I would produce a small hierarchy of diagrams, each at one level of zoom and each with one named audience, rather than one giant picture. For an integration with APIs, message brokers, ETL (extract, transform, load: copying and reshaping data between systems) and a central data lake (a large store holding raw data from many sources), I would produce five: a context diagram for executives and business owners, a logical data-flow diagram for architects and data teams, a container view for engineers, a deployment diagram for operations and security, and sequence diagrams for the developers building each interface.

The set and its audience

(C4 is a convention with four zoom levels: context, container, component, code. Diagrams 1, 3 and part of 4 map onto it; the logical flow and sequence diagrams complement it. "Context" means our system as one box with its neighbours, and "container" means the separately deployable parts inside it.)

Worked example: an on-premises ERP plus cloud platform proposal

Take a proposal to connect an on-premises ERP (enterprise resource planning system) to a cloud lake. The hierarchy is: context (ERP, cloud platform, partners) then logical (order events flow ERP to broker to lake) then deployment (the ERP zone on the corporate network, a private link into the cloud account, the broker in a private subnet, the lake in a separate data account) then sequence (order event: publish, broker acknowledges, ETL consumes, writes to raw zone, acknowledges; failure branch: retry three times, then dead-letter queue, meaning a holding queue for messages that could not be processed). Terms used above: a private link is a private connection into the cloud that avoids the public internet; a private subnet is a network segment with no direct route from the internet; a trust boundary is a line where data passes from one security zone to another, so it needs checks; an API gateway is the single front door that receives API calls and applies authentication and limits; the raw zone is the lake area where data lands unchanged before cleaning; a broker acknowledges (sends an "ack") when it has safely stored a message.

Sketch of diagram 1 as text (Mermaid, a text format that renders to a picture):

flowchart LR
  ERP["On-prem ERP"] -->|order events| P["Integration platform (our system)"]
  Partner["Partners"] -->|API calls| P
  P -->|curated data| Users["Analytics users"]

Sketch of diagram 5, the order flow:

sequenceDiagram
  participant ERP
  participant Broker
  participant ETL
  participant Lake
  ERP->>Broker: publish OrderCreated
  Broker-->>ERP: ack
  ETL->>Broker: consume
  ETL->>Lake: write to raw zone
  ETL-->>Broker: ack
  Note over ETL,Broker: on failure retry 3 times, then dead-letter queue

The deployment diagram is where a security boundary earns its keep (mutual TLS means both sides present certificates, so the server also verifies the client): label each crossing with protocol and authentication, for instance "mutual TLS, certificate rotated quarterly" as an illustrative label.

Traceability

Give each diagram element a short ID (for example C3 for the third box in the container view, SVC-2 for the second service, B1 and B2 for the first and second trust boundaries on the deployment diagram) and reference the requirement IDs it satisfies in a small table beside the diagram ("REQ-14 data encrypted in transit: satisfied at boundaries B1 and B2"). A reviewer can then check coverage in both directions, from requirement to picture and picture to requirement.

Trade-offs and pitfalls

• One diagram for all audiences fails everyone: executives see noise, engineers see too little.
• Every diagram needs a title, a legend, a date or version, and the same box shapes for the same things across the set.
• Keep diagrams as text (for example Mermaid or PlantUML) in the repository so they are reviewed and updated with the code.
• Do not draw more than about a dozen boxes per diagram; split instead.

Direct answer

The document must state, for a public API, who can call it, how callers prove who they are, what they are allowed to do, how data is protected in transit and at rest, and how abuse is limited and detected. A security reviewer wants threats, controls and evidence; a developer wants how to authenticate correctly and what errors to expect. I would write one source of truth and give each reader a different entry point into it.

Required content

1. Authentication: the mechanism (for example OAuth 2.0, the standard for delegated access, with short-lived access tokens (an access token is a temporary string the caller sends with each request to prove it was authorised), or API keys for server-to-server use), where tokens are issued, lifetime, refresh and revocation.
2. Authorization: the model (scopes are named permissions carried on the token, such as orders:read; roles group permissions by job) and where it is enforced (gateway versus service), including object-level checks (a caller must own the record it requests: if user A asks for /orders/1002 and that order belongs to user B, the API must refuse, even though A's token is valid; broken object-level authorization is the top risk in the OWASP API Security list, OWASP being an open security-community project).
3. Trust boundaries and data flow: what is exposed publicly, what sits behind the gateway, and where TLS ends (TLS is the encryption that protects traffic on the network; "where it ends" is the point where traffic is decrypted and must be trusted from then on).
4. Data protection: encryption in transit and at rest, classification of fields, what is never logged.
5. Abuse controls: rate limits, quotas, input validation, size limits, and how bad clients are blocked.
6. Threat model: a threat model is a structured list of what could go wrong and who might attack, each threat paired with the control that answers it. Record the residual risk (what remains after the controls) and the named person who accepts it.
7. Secrets and key management, audit logging, incident contacts.

Presenting to each reader

Worked example (illustrative fragment)

Security reviewer fragment, threat table rows:

Developer fragment, the "how to get a token" page (client-credentials grant means a server application exchanges its own id and secret for a token, with no human user involved; a bearer token is one that works for whoever holds it, so it must be sent only over TLS):

POST /oauth/token
grant_type=client_credentials&client_id=<your-id>&client_secret=<your-secret>&scope=orders:read
-> {"access_token": "<token>", "expires_in": 900}

GET /orders/1002
Authorization: Bearer <token>

Errors: 401 means the API does not know who you are (no token, malformed or expired), so fetch a new token. 403 means it knows who you are but you may not do this (the token lacks the scope, or the order is not yours), so requesting a new token will not help. That difference is authentication (proving who you are) versus authorization (what you are allowed to do).

Pitfalls: describing the design without evidence a reviewer can check; documenting the mechanism but not the failure responses; putting secrets or real tokens in examples; letting the doc and the gateway configuration diverge. Generate the developer reference from the API's OpenAPI description (a machine-readable file listing every endpoint, parameter and response) where you can, so it cannot drift.

Direct answer

I would ship a small, layered documentation set that is owned, versioned and tested by someone other than the original authors: a one-page orientation, an architecture pack, runbooks (step-by-step operating instructions for a specific situation), and a knowledge-transfer plan that ends with the client's engineers performing the work while we watch. The test of success is that a new engineer, with no access to the original team, can run, change and fix the system using only the pack. I would prove that before handover, not assume it.

The documentation set (what and for whom)

Keep these in the same repository as the code (docs-as-code: plain text reviewed in pull requests), so a change to the system and a change to its docs go through one review. Each page gets a named owner role (not a person) and a "last verified" date.

Knowledge-transfer plan

1. Weeks before handover: the client's engineers shadow us on a real deployment and a real incident drill.
2. Reverse shadowing: they lead, we watch and only correct.
3. Handover gate: a client engineer who was not involved performs a clean deploy, a rollback and one runbook drill using only the docs. Every place they get stuck is a docs bug we fix before sign-off.
4. After handover: a defined warranty period with a named contact, and a docs-review date in the client's calendar.

A new engineer's first week (worked example)

• Day 1: access checklist (accounts, VPN, secrets vault), read the one-page orientation, run the local environment from the README.
• Day 2: read the context and container diagrams and the five most recent ADRs; explain the data flow back to a buddy.
• Day 3: deploy a trivial change to a non-production environment.
• Day 4: run a runbook in a rehearsal (for example restore a database backup into a scratch environment) with the buddy observing.
• Day 5: fix one small, pre-selected "starter" issue end to end, and note every doc gap found. Those notes become their first documentation pull request.

Trade-offs and pitfalls

• Docs rot fastest where nobody is measured on them. Tie them to the definition of done (a change that alters behaviour updates the docs) and review them on a schedule.
• Exhaustive documentation nobody reads is worse than a short pack that is exercised. Prefer runbooks that are executed in drills over long prose.
• Several teams over years means turnover: write for someone who has never met us, and record the why (ADRs), because the what is discoverable from the code.
• If the client will not staff an owner, I would flag that in writing as a handover risk; no documentation compensates for an unowned system.

Direct answer

The ADR (Architecture Decision Record, a short document capturing one design decision, what else was considered, and what the team accepts as the price) states plainly that we accept eventual consistency, meaning replicas may briefly disagree but converge to the same value if writes stop, for the read-heavy user-profile and cart reads, and keep strong consistency (every read sees the latest acknowledged write) only where being wrong costs money. The consequences section is written as things a user can see, with a mitigation for each, not as a list of architectural virtues.

The ADR excerpt (cart service, illustrative numbers are targets, not measurements)

Terms used in the excerpt: the primary region is the one data-center region that accepts all writes; read replicas are copies of the data in other regions that only serve reads; replication lag (also called convergence lag) is the delay before a change on the primary appears on a replica; a network partition is a break in connectivity between regions; a runbook is a step-by-step guide for handling a specific problem.

ADR-0031: Serve cart and profile reads with eventual consistency
Status: Proposed

## Decision
We will write cart and profile changes to the primary region and serve
reads from regional read replicas and a 30-second cache. Reads are
eventually consistent by default. Checkout, payment and refunds always
read from the primary (strongly consistent). Concurrent edits to the same
cart line are resolved by version number: highest version wins, deletes
are kept as tombstones (markers that say "this item was removed"). Concurrent adds of different items are merged as a union. Losing one of two concurrent quantity changes is an accepted cost.

## Alternatives considered
A. Strong consistency everywhere (single primary for all reads).
   Rejected: reads outnumber writes many times over, so every read would
   pay cross-region latency and the primary becomes the scaling limit.
B. Eventual consistency everywhere, including checkout.
   Rejected: a stale price or stock count at payment is a money error.
C. Read-your-writes only for the writing session (the session sends the
   version it last wrote and reads wait for it). Adopted as a refinement
   of the decision, not a separate option.

## Consequences
What users can see:
- Add an item on the phone, open the laptop within a few seconds: the
  cart may look empty. Mitigation: session token carries the last version
  written, so the same device always sees its own change.
  Limit: a different device has no such token, so the phone-to-laptop
  case is accepted, not solved. It clears once the replica catches up
  (2 s target).
- Remove an item on one device while adding on another: the removed item
  can reappear. Mitigation: tombstones, and the UI shows "cart updated".
- Price on the cart page may be up to 32 seconds old (2 s replication
  target + 30 s cache). Mitigation: price is re-read from the primary at
  checkout; a change shows a "price changed" prompt before payment.
- If the primary region is unreachable, users can still browse and edit
  carts from a replica, but changes queue on the client and reach the primary later (replicas themselves are read-only).
What operations inherit: convergence lag must be monitored; an alert fires
when replication lag exceeds 5 s for 5 minutes; a runbook covers a stuck
replica; tests below prove the behaviour.

Why this wording is honest

• Every negative is a user-visible scene ("the cart looks empty"), so a product manager can judge it, not just an engineer.
• Each has a named mitigation and its limit, and the 32 s figure is arithmetic the reader can check: 2 + 30 = 32.
• Alternatives include a rejected extreme on both sides.

Conflict resolution and compensation (the rules the Decision section relies on)

This explains the merge rules that the ADR's Decision paragraph names. Version-based "highest version wins" can silently drop one of two concurrent changes. The decision states that cost, and adds a compensation: if two devices change the quantity of the same line within the convergence window, we keep the higher version and show the user the final cart, since losing a quantity tweak is cheap. For adds we prefer a union of items, since losing an added item is worse than a duplicate the user can delete.

Traced with real cart values. The cart starts as {shoes x1 at version 4, hat x1 at version 2}.

• Adds (union): the phone adds socks and the laptop, still on the old copy, adds a scarf. Merge: {shoes, hat, socks, scarf}. Nothing is lost.
• Quantity (higher version wins): the phone sets shoes to 2 and the laptop sets shoes to 3, both starting from version 4. The primary assigns version numbers in arrival order, so the phone's write becomes version 5 and the laptop's becomes version 6. Highest version wins: shoes x3. The phone's change to 2 is lost, and the user sees the final cart with "cart updated".

Payments-ledger variant

For a payments ledger the same ADR would choose the opposite: strongly consistent writes and reads from the primary. The SLO (service-level objective, the reliability target we promise) changes: "no double spend and balance reads are never stale" is the objective, at the cost of higher latency and lower availability during a network partition (the primary must be reachable). The consequences would say that plainly: during a regional outage, payments pause rather than risk an incorrect balance.

Validation entries recorded in the ADR

• Test: two simulated devices edit the same cart within the lag window; assert the final cart equals the defined merge result.
• Monitoring: replication lag, cache hit rate, and a counter of "tombstone resurrections" reported by clients.
• Playbook (a written checklist of steps to follow): how to fail reads over to the primary if lag stays above threshold.

Pitfalls

• Writing "may experience slight delays" hides the anomaly. Name it.
• Do not leave out what would make us reverse the decision. Add a revisit trigger, such as "if support tickets about disappearing items pass an agreed rate, revisit read-your-writes scope".

Direct answer

Make the rejected option a first-class part of the same decision record (an ADR, architecture decision record: a short dated note capturing one significant design decision and why), with an honest "Alternatives considered" section: what the option is, the real merits it had, why it lost under the constraints of the moment, and the specific condition that would reopen the question. Keep it in the same record by default because the reader looking at the chosen design is exactly the one who will ask "why not the other one?". Use a separate record only in the narrow cases below.

How to write the rejected alternative so it prevents a blind reopening

1. Steelman it (present the opposing option in its strongest, fairest form): state its strengths in terms its advocates would accept. A strawman (a weak, distorted version that is easy to knock down) invites someone to "rediscover" the option.
2. Name the criteria and how each option scored (a small table, qualitative labels are fine).
3. State why it lost: the constraint or weight that decided it, and dated to the context ("with a two-person team and no broker experience").
4. Give a revisit trigger: the measurable change that would reopen the debate.

Worked example: service-to-service call path (Orders needs a price from Pricing)

(gRPC is a common way for one service to call another directly and wait for the answer. A message broker is a middle service that holds messages so a sender can publish an event without waiting, and receivers pick it up later. Asynchronous means the sender does not wait for the reply.)

Alternatives section text: "Option B (events) has real merits: it isolates checkout from Pricing outages and scales to many consumers. We rejected it because checkout must show the exact price synchronously, and the team has no operating experience with a broker. Revisit if Pricing availability becomes the leading cause of checkout failures, or three or more services need price changes." If someone reopens it later, they start from those conditions instead of from zero.

Same record or separate?

• Default: one ADR per decision, alternatives inside it. Discoverability is best because the reader finds the alternatives exactly where they look for the decision.
• Separate ADRs when the decisions are genuinely independent (choosing the wire protocol, the format and rules for messages sent between services, versus choosing service discovery, the way services find each other's network addresses); bundling them hides one decision behind another.
• A standalone "Rejected" record when an alternative is a big proposal with an independent life, such as one that went through its own RFC (request for comments, a circulated proposal) and may come back. Link it from the chosen ADR and give it status Rejected.
• Never spread one decision's alternatives across several files with no links: nobody will piece them together.

Evaluation criteria list (regional-cache ADR)

For an ADR choosing a regional cache (a copy of frequently read data kept close to users in each region): read-latency improvement, tolerance for stale data, invalidation approach (how cached copies are cleared or refreshed when the source changes), operational complexity, cost, behaviour when the cache is down, and data-residency limits by region (rules that certain data must stay in certain countries). Listing criteria before options keeps the comparison fair.

Illustrative scoring (1 to 5, higher is better; the team's judgement) for two options, a per-region cache and a single central cache:

Data residency is really a pass/fail gate: if some data may not leave a region, the central option is out for that data whatever it scores elsewhere. The scores then show the trade: better latency and compliance against more to operate.

Trade-offs and pitfalls

• Alternatives written after the fact to justify the choice: readers detect this.
• Editing the ADR when you change your mind: write a new one that supersedes it and links back.
• No revisit trigger: the debate returns at the next architecture meeting.