Architecture Documentation and Communication Questions

Direct answer

Build one landscape page with four panels and a one-line decision at the top. Each panel answers one question a leader will ask: how busy are we now, how fast is demand growing, how much room is left before users notice, and what happens automatically when load rises. An inference service is the API that serves predictions from a trained machine learning (ML) model. Headroom is the spare capacity left before the service misses its latency target. Autoscaling means the platform adds or removes replicas (identical copies of the model server, each usually holding one GPU) on its own.

The four panels

Worked example (illustrative service)

Load tests show one replica holds 50 RPS while keeping 95th-percentile (P95) latency under the 300 ms objective (95 of every 100 requests finish within 300 ms; this kind of target is often called a service-level objective). Quota is the maximum number of replicas the cloud account is allowed to run; here it is 30.

Where the months come from. Growing 8% a month means multiplying by 1.08 each month, so after n months the peak is 620 x 1.08^n. We want the n where that reaches 1,050. Taking logarithms (ln, the function that answers "what power gives this number?") turns the power into a multiplication: n = ln(1,050 / 620) / ln(1.08) = 0.527 / 0.077 = about 6.8. Check by hand: 620 x 1.08^6 = 984 and 620 x 1.08^7 = 1,063, so the line crosses 1,050 during month 7. For another service, swap in its own current peak, ceiling and growth rate.

Sizing the ask. The decision line needs a number. At 8% a month the peak reaches 620 x 1.08^12 = about 1,561 RPS at month 12, which is already above today's 1,500 RPS hard ceiling. To stay under 70% at that point the fleet needs 1,561 / 0.7 = about 2,230 RPS of safe-plus-buffer capacity, that is 2,230 / 50 = about 45 replicas, so the request is for roughly 15 more replicas than the current quota of 30 (about 45 in total). Print that figure beside the decision line, and price it in one line.

Projected peak: 781 RPS at month 3, 984 at month 6, 1,063 at month 7. If more GPU capacity takes about 3 months to obtain (an assumption to confirm), the request for it must land by roughly month 4. That gives the top-of-page decision line: "Approve additional GPU quota by month 4 to stay under 70% of capacity through month 12."

Autoscaling policy panel (settings, not measurements)

• Signal: RPS per replica. Target 35 (70% of 50). Scale out (add replicas) when the average stays above 35 for 3 minutes.
• Scale in (remove replicas) only after the average stays below 25 for 15 minutes, so the fleet does not flap (repeatedly add and remove replicas as load wobbles around the threshold).
• Minimum 8 replicas so one failure or a slow model load does not hit users. Maximum 30 (the quota).
• State the limit plainly: autoscaling cannot exceed the quota, so beyond 1,500 RPS the service degrades.

Trade-offs and pitfalls

• Show peaks, not averages. An average hides the hour that breaks the objective.
• GPU utilisation alone is misleading. 90% can be healthy or on fire; tie every utilisation line to the latency objective.
• Compounding growth looks flat early and steep late. A straight-line projection understates the date.
• Show a range for growth, not one line, and print the assumption next to it.
• Put the cost of the ask in one line. Do not add more charts to prove a point already made.

Direct answer

Because the decision tells people what was chosen, but only the reasoning tells them whether it is still the right choice. Without it, later engineers either avoid changing something they do not understand or change it without knowing what it was protecting. An ADR (Architecture Decision Record, a short document capturing one design decision) should therefore record the constraints, the alternatives, the assumptions and a revisit trigger, not just a title.

What goes wrong without it

• Re-litigation: the same argument recurs every year because nobody remembers it was settled or why.
• Cargo-culting: people copy a pattern into places where its reason does not apply (copying a practice because it worked elsewhere, without understanding why it worked).
• Frozen or reckless changes: "nobody dares touch it" or "let's rip it out" (a fence taken down without asking why it was built, from the idea known as Chesterton's fence: before removing something, find out why it was put there).
• Onboarding and handoff: newcomers and successors see structure but not intent.
• Audit: an auditor or reviewer asking "why did you accept this risk, on what evidence, and who agreed?" gets no answer.

Which elements matter most

1. Constraints (time, budget, team skills, regulations, existing systems): the most valuable, because they explain the choice and expire on their own.
2. Assumptions (what we believed about traffic, growth, users): these are what silently become false.
3. Alternatives, and why each lost: stops people proposing the discarded option again.
4. Revisit triggers (a measurable condition that reopens the decision): converts an assumption into a tripwire.

Worked example

Decision: run the app in a single region.
Constraints: launch in 3 months, 4 engineers, users almost all in one country.
Alternatives: multi-region (rejected: doubles operational work for the team we have).
Assumptions: traffic stays in one country; a few hours of regional outage is tolerable.
Revisit trigger: more than 20% of users outside the region, or the availability target is missed twice in a quarter.

Two years later a new team sees one region. Without the record they either spend months "fixing" it or never notice the company expanded. With it, the trigger tells them exactly when to reopen.

Other uses

• Handoff: the new owner gets the reasoning along with the code.
• Audit: a dated trail of who decided what, with which evidence.
• Future review: reviewers can compare current reality with the recorded assumptions.

Reusable structure

Title, status, date; context and constraints; decision; alternatives with reasons rejected; assumptions; consequences; revisit trigger.

ML angle: batch versus online inference

Inference is running a trained model to get predictions. Batch inference computes predictions in bulk on a schedule; online inference computes them per request. A short rationale for it:

Revisit trigger for the ML choice: if the freshness need drops below hourly, or volume grows enough to change the cost picture. That table lets a later reviewer grasp the trade-off in a minute. With numbers (illustrative): a store has 2 million users and a slowly changing catalog. Batch: one nightly job of about 1 hour, so recommendations are at most about 25 hours old: 24 hours between runs plus the hour the job itself takes. Online: at a peak of 500 requests per second, each needing about 20 ms of model compute, the always-on servers need 500 x 0.02 = 10 cores busy at peak, plus headroom. Rationale recorded: 24-hour freshness is acceptable, so batch is cheaper. Revisit trigger: if product needs a click reflected within 5 minutes, batch cannot meet it and the decision reopens.

Pitfalls

Writing only the conclusion, or recording rationale that is a sales pitch with no alternatives. And never editing an old ADR to make past reasoning look better.

Direct answer

Run the service on Kubernetes (the common system that runs containerised programs across a cluster of machines). Each running copy of the model server is a pod (the smallest unit Kubernetes runs), and pods land on a dedicated node pool (a group of identical machines, here ones with GPUs). Mark that pool with a taint (a label that keeps ordinary workloads off those expensive nodes), scale on request pressure, and document it in two views: an operator view (what is running, is it healthy, what do I do when it is not) and a platform view (what capacity, scheduling rules and versions this service requires of the shared cluster).

Topology

flowchart LR
  C[Callers] --> GW[Gateway with rate limits]
  GW --> S[Inference pods]
  S --> N[GPU node pool, tainted]
  S -.-> W[(Model weights in object storage)]
  S --> M[Metrics and alerts]
  Q[Request queue depth] --> AS[Autoscaler]
  AS --> S

Key decisions to document:

• One model replica per GPU by default: simple, predictable memory. Sharing a GPU is an optimisation to record as a decision, not a default.
• Scheduling: pods request nvidia.com/gpu (one GPU device) and carry a toleration (permission to be placed on nodes that have the matching taint), as in this fragment (illustrative names):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-inference
spec:
  replicas: 3
  selector:
    matchLabels: {app: model-inference}
  template:
    metadata:
      labels: {app: model-inference}
    spec:
      tolerations:
        - key: gpu
          operator: Equal
          value: "true"
          effect: NoSchedule
      containers:
        - name: server
          image: registry.example.com/model-server:1.4.2
          resources:
            limits:
              nvidia.com/gpu: 1

How to read the fragment: replicas: 3 asks for three copies of the server. The tolerations block says "I am allowed onto nodes tainted gpu=true; effect: NoSchedule means pods without this permission are never placed there". The limits: nvidia.com/gpu: 1 line asks the scheduler for exactly one GPU per pod, so the pod only starts on a node that has a free one. The taint keeps others out; the limit reserves the GPU. You need both.

• Autoscaling on queue depth or in-flight requests, not CPU, because GPU work is not visible in CPU use.
• Cold start (the slow first period while a new pod loads the model before it can serve): loading model weights from object storage is slow, so keep a warm minimum of replicas and pull weights to local cache at start-up.
• Batching: grouping requests raises throughput but adds waiting time; the chosen batch window is documented against the latency target. For example, with a P99 target of 200 ms (the slowest 1 request in 100 must still finish within 200 ms) and a model call that takes 150 ms, a 10 ms batch window leaves room, while a 100 ms window would put the slowest requests over the target. Write down the window, the target and the throughput gain you measured.

What each reader must be able to read

Trade-offs and pitfalls

• Scaling GPU capacity is slow and costly; document the warm minimum and its price, or the docs promise elasticity the system lacks.
• Model version and infrastructure version must both appear in the docs; failures often come from a mismatch.