Architecture Documentation and Communication Questions

Direct answer

Interpreting "service orchestration platform" as the runtime that schedules and runs your services (an orchestrator is software that decides which machine each copy of a service runs on, restarts crashed copies, scales copies up or down and rolls out new versions; Kubernetes is the most common one), versus each team choosing its own, a one-page ADR (architecture decision record: a short document capturing one decision) needs five things: the decision in one sentence, two or three alternatives, the rationale tied to a real pain, migration risks, and a rollback plan. It stays to one page by linking out, using tight bullets, and giving each section a line budget.

The one-page ADR

# ADR-12: Standardise service runtime on one container orchestrator
Status: Proposed | Date: 2026-09-28 | Owner: platform lead

**Decision.** All new and migrated services run on the managed Kubernetes
cluster (a cloud provider operates the cluster's control machinery for us). Other runtimes need a written exception.

**Context.** Six services run on four different platforms (VMs, two PaaS
(platform-as-a-service) products, hand-rolled scripts). Each has its own deploy, logging and
on-call setup, so incidents take longer to diagnose and new joiners learn
four systems.

**Alternatives.** (1) Keep team choice. (2) Serverless functions (code the cloud vendor runs on demand, with no servers to manage) for
everything. (3) One orchestrator (chosen).
Rejected 1: keeps the cost we are trying to remove. Rejected 2: poor fit
for long-running and stateful services.

**Rationale.** One deploy path, one observability setup, one skills base.

**Migration risks.** Team learning curve; two systems in parallel during
the move; noisy-neighbour effects on shared nodes (one service hogging a machine's CPU or memory and slowing the others); cost increase before
old platforms are retired.

**Rollback.** Migrate one service at a time. Keep the old deployment
warm until the new one has run a full release cycle; rollback is
re-pointing traffic. Stop the programme if two consecutive migrations
miss their success checks (the agreed targets: deploy time no worse than before, no rise in incident count).

**Review.** 6 months, against example targets: services migrated 0 of 6 today, target 6 of 6; deploy time 40 min today, target under 15 min; time to diagnose an incident 2 h today, target under 1 h.

How to keep it to one page

• Decision first, then a sentence per section. Bullets, not paragraphs.
• Link to the cluster design, cost model and migration runbook instead of pasting them.
• One decision only. Choice of service mesh (a layer that manages service-to-service traffic) or ingress (the entry point that routes outside traffic in) is a separate ADR.
• Cut anything a reader would not miss: history of how you got here belongs in the linked proposal.

ML variant: model-serving platform choice (optional extension, only if the interviewer steers to ML)

Same shape, different options. For serving machine-learning models, compare three hosting styles:

A reasonable commit: containers on the standard platform for steady, latency-sensitive models, and a managed service for experiments. State what would flip it: if traffic is very spiky and models are small, serverless wins.

Pitfalls

• Standardising without an exception path creates shadow platforms.
• A rollback plan that says "revert" without saying what stays running during the move.
• Listing benefits with no measurable check; the review line above is what makes the decision testable.

Direct answer

The observability section of an architecture document should say what signals the service emits, what "healthy" means in numbers, where an on-call engineer looks first, and how to go from a symptom to a cause. It is a map to the monitoring system, not a copy of it: it holds the design intent and the stable names, while dashboards and alert rules live in the tooling, linked from here.

Contents (in this order)

1. Service level indicators (SLIs) and objectives (SLOs): an SLI is a measured quality of the service (for example the fraction of requests that succeed); an SLO is the target for it. List the two to four that describe user-visible health.
2. Golden signals per component: latency, traffic, errors, saturation (how full the resource is). These four come from Google's Site Reliability Engineering book and are widely used as a minimal set that covers most user-visible trouble.
3. Signals mapped to the three telemetry types: metrics (numbers over time, cheap, good for alerting), traces (one request followed across services) and logs (event detail). For each SLI say which metric measures it, which trace shows why it slipped, and which log confirms the cause.
4. Instrumentation points: where in the architecture telemetry is inserted (ingress, meaning where requests enter the service; outbound calls, queue consumers, database client), and the correlation ID (a unique ID passed along a request so its logs and traces can be joined).
5. Alerts: each alert names the symptom, the SLO it protects, severity, and the runbook link (a runbook is a step-by-step guide for handling that alert). An SLA (service level agreement) is the contractual promise to customers; the SLO is set stricter than the SLA, and the error budget is the failure allowance the SLO leaves (a 99.9 percent SLO leaves 0.1 percent). Alert on symptoms users feel, not every internal cause.
6. Dashboards and dependencies: links, and the upstream and downstream services with their own health pages.
7. Known blind spots: what we cannot currently see.

Keeping it useful without becoming a monitoring manual

• Include only what a stranger woken at night needs; put threshold tuning and query syntax in the monitoring tool.
• One table, one page: signal, what it means, first action.
• Link, do not paste screenshots that rot.

Worked variants (each a small row set)

What "healthy in numbers" looks like (illustrative targets to agree with the owning team):

And the one-table deliverable for the search row:

Pitfalls: alerts with no owner or runbook; listing hundreds of metrics; describing tools instead of decisions; no statement of what "healthy" is.

Direct answer

I would keep one small, structured record per service, stored as a file next to the service's code (so it is reviewed with changes), with fixed fields for where it can fail, what depends on it, and what breaks when it fails. Because the fields are machine-readable, a script can walk the dependency links to compute blast radius, and the same records feed a failure-mode analysis.

Terms

• Failure domain: a set of things that fail together (one process, one host, one availability zone (AZ), one region, one account, one third-party provider).
• Blast radius: how much of the system and how many users are affected when one thing fails.
• FMEA (failure mode and effects analysis): list each way a component can fail, the effect, and score severity, occurrence and detectability, each from 1 to 10. Their product is the risk priority number (RPN); higher means fix first. Scale direction matters: severity 10 is the worst impact, occurrence 10 is the most frequent, and detectability 10 means the failure is almost never caught (1 means it is always caught), so a hard-to-detect failure raises the RPN.

Required fields

Plain meanings: a hard dependency is one the service cannot work without; a soft dependency can be skipped in a degraded mode. Sync means the service waits for the reply; async means it hands work off and continues. The criticality tier ranks how much the business suffers if the service is down (for example tier 1 = revenue-critical). A single point of failure is one component with no backup whose failure stops the service. A game-day is a rehearsal where the team deliberately breaks something to test detection and recovery. A failure mode is one specific way a component can fail.

One record filled in (illustrative, for the checkout service)

service: checkout
owner: payments-team
criticality_tier: 1
failure_domains: [region-a, az-1, az-2]
depends_on:
  hard: [payments, inventory]
  soft: [recommend]
failure_modes:
  - mode: payments timeout
    effect: users cannot pay
    fallback: retry once, then show "try again" page
    detection: payment error rate alert
rto: 15m
rpo: 1m
single_point_of_failure: false
last_game_day: 2026-06-10

Worked example (illustrative data, script run as shown)

What the script does: depends_on maps each service to the services it needs. To find who breaks when failed goes down, blast_radius repeatedly finds every service that depends on anything in the current set (the frontier), adds them to the affected list, and repeats until no new services appear. So stock-db breaks inventory and search, which in turn break checkout, which breaks web.

# who breaks when a service fails? reverse traversal of "depends_on"
depends_on = {
    "checkout":  ["payments", "inventory"],
    "payments":  ["card-gateway"],
    "inventory": ["stock-db"],
    "search":    ["stock-db"],
    "recommend": ["model-api"],
    "web":       ["checkout", "search", "recommend"],
}

def blast_radius(failed):
    hit, frontier = set(), {failed}
    while frontier:
        nxt = {s for s, deps in depends_on.items() if frontier & set(deps)} - hit
        hit |= nxt
        frontier = nxt
    return sorted(hit)

for svc in ["stock-db", "model-api"]:
    print(svc, "->", blast_radius(svc))

# FMEA risk priority number = severity x occurrence x detectability (each 1-10, illustrative)
rows = [("model-api timeout", 6, 5, 3), ("model-api quota exhausted", 6, 4, 7), ("stale features", 4, 6, 8)]
for name, s, o, d in sorted(rows, key=lambda r: -(r[1]*r[2]*r[3])):
    print(f"{name}: RPN = {s}*{o}*{d} = {s*o*d}")

Output:

stock-db -> ['checkout', 'inventory', 'search', 'web']
model-api -> ['recommend', 'web']
stale features: RPN = 4*6*8 = 192
model-api quota exhausted: RPN = 6*4*7 = 168
model-api timeout: RPN = 6*5*3 = 90

Worked FMEA rows for a third-party model API (the dependency behind the recommend service)

Rows for the model API: timeout (visible, so easy to catch: detectability score 3 on the scale where 1 is always caught and 10 is almost never caught), quota exhausted (the provider's usage limit is used up; silent until requests fail, hard to catch: score 7), stale features (silent quality loss, hardest to catch: score 8, so highest RPN). The record names the fallback for each (cached result, smaller local model, or explicit "recommendations unavailable"), and the third party is its own failure domain.

Trade-offs and pitfalls

• RPN scores are judgement, not measurement. Use them to rank, and have the owning team score them, not the author alone.
• Records rot. Tie a review to each game-day and to any dependency change.
• Blast radius here is service-level; user-percentage estimates need traffic data.

Direct answer

No single diagram describes a distributed service, so I draw a small set where each answers one question: context (who and what surrounds the system), container (what deployable pieces it is made of), sequence (what happens over time in a key flow, including failure), deployment (where it runs), and, when needed, component (the inside of one complex container), data flow and state diagrams: seven in all, matching the table below. These follow the C4 model (Context, Container, Component, Code: four zoom levels) plus UML (Unified Modeling Language, a standard diagram notation) style sequence diagrams (participants across the top, time flowing down) and state diagrams (the states an entity moves through and what triggers each move). Start with context, container and sequence: those three are asked about most; data flow and state are occasional. For SREs (site reliability engineers, who run the service in production) I lead with deployment, component and failure sequence; for a mixed audience I show context, container and one happy-path sequence (the flow where nothing goes wrong).

Which diagram answers which question

SRE review versus mixed audience

• SREs: a component view (the internal structure of one service) plus a deployment view, then a sequence of the failure path and the dependency list with timeouts. They want blast radius (how much breaks when one part fails), capacity and failover.
• Mixed audience (product, QA, engineering): context, container in plain names, and one narrated sequence. No cloud-icon wall, no acronyms without a legend.

Worked example: mobile client, API, auth service, database

1. Context: User, Mobile app system, external identity provider.
2. Container: Mobile app, API, Auth service, Database, each with a one-line responsibility.
3. Sequence: app sends login, API asks Auth, Auth checks the database, token returns. Timeout on the Auth call is stated (for example 2 seconds).
   That is three views for three questions, not one overloaded picture. The sequence for the login flow, drawn as Mermaid:

sequenceDiagram
  participant M as Mobile app
  participant A as API
  participant U as Auth service
  participant D as Database
  M->>A: login(user, password)
  A->>U: verify credentials (timeout 2 s)
  U->>D: look up user
  D-->>U: user record
  U-->>A: token
  A-->>M: token

And the container view as text, one line per box:

Mobile app -> API      : HTTPS, JSON
API        -> Auth     : verifies credentials
Auth       -> Database : reads user records

Conventions I apply to all of them

• Naming: use the same names as the code repository and dashboards, one name per thing.
• Tools: diagrams as code (Mermaid, PlantUML and Structurizr are text-based diagram tools; Mermaid is the easiest start because many Git hosts render it directly) kept in the repository so changes are reviewed in pull requests; a drawing tool for one-off workshop sketches.
• Annotate: state assumptions ("peak 500 requests per second, assumed"), non-functional requirements (NFRs, targets such as availability or latency), and a version and date on each diagram.
• QA audience: sequence and state diagrams are what testers turn into test cases, and the context view shows integration points to stub (replace with a fake in tests).

Trade-offs and pitfalls

• Drawing all diagrams for every service is wasted effort; draw the ones whose question is open.
• Mixing levels (a database next to a class name) is the most common defect and confuses every reader.

Direct answer

An Architecture Decision Record (ADR) is a short document that captures ONE significant architecture decision: the situation that forced it, what was chosen, what else was considered, and what the team accepted as a consequence. A good ADR is written at the time of the decision, is honest about the downsides, and is never silently rewritten later, so a future engineer can learn why the system looks the way it does, not just how.

What it contains

• Title and number: a short statement of the decision, such as "ADR-0007: Use gRPC for internal service calls". (gRPC is Google's remote procedure call framework; REST is the common style of calling services over plain HTTP with JSON.)
• Status: Proposed, Accepted, Rejected, Deprecated, or "Superseded by ADR-00NN".
• Context: the forces at play (requirements, constraints, team skills, deadlines) in a few sentences a newcomer can follow.
• Decision: one clear sentence in active voice ("We will...").
• Alternatives considered: the real options and why each lost.
• Consequences: what gets easier, what gets harder, and what new risks we now carry. Both good and bad.
• Date and deciders, so people know who to ask.

Worked example (REST vs gRPC)

ADR-0007: Use gRPC for calls between Orders and Pricing
Status: Accepted (2025-03-12)
Context: Orders calls Pricing on every checkout. Both are internal, both
  are owned by us, and we want a typed contract (request and reply shapes declared in a schema, so a mismatch is a build error rather than a production surprise).
Decision: We will define the contract in protobuf (a compact, typed message format declared in a schema file) and call Pricing over gRPC.
Alternatives: REST+JSON (familiar, easy to debug with curl, a command-line tool for sending web requests, but the
  contract is only documentation); message queue (decoupled, but checkout
  needs the price synchronously).
Consequences: (+) generated clients, compile-time contract checks.
  (-) browsers cannot call it directly, so the public edge stays REST;
  (-) the team must learn protobuf tooling and gRPC debugging.

When your team writes one

Write an ADR when a decision is hard to reverse, affects more than one team, had real alternatives, or would make a new hire ask "why on earth did we do it this way?". Cross-team adoption works best when the ADR arrives as a pull request (a proposed change others review) in a shared or per-repo folder, so the discussion happens on the record and other teams can link to it or reuse it rather than re-deciding.

Storing, versioning and surfacing them so rationale survives years

• Keep them in version control next to the code (for example docs/adr/0007-use-grpc-for-internal-calls.md), one file per decision, numbered and never deleted. Git history (the stored record of every change and who made it) then gives you versioning and authorship for free.
• Treat accepted ADRs as append-only (you add new records but never rewrite old ones): to change your mind, write a new ADR and mark the old one "Superseded by ADR-0015", with a link in both directions.
• Maintain an index page (or publish the folder to the docs site) listing number, title, status and date, so people can search rather than browse.
• Link ADRs from the places engineers actually look: the service README, the pull request that implemented it, and the service catalog entry (the internal directory listing each service with its owner and docs). Auditors get a dated, reviewed, immutable (cannot be altered afterwards) trail.

Trade-offs and pitfalls

• Too long: nobody reads a five-page ADR. Aim for one page.
• Too many: writing ADRs for trivial choices creates fatigue and buries the important ones.
• Decision without alternatives or consequences reads as a press release, not a record.
• Editing history: rewriting an old ADR destroys the very evidence it exists to preserve.