API Documentation and Developer Experience Questions

Direct answer

Write it in terms of business outcome, rules and consequences, not HTTP mechanics. A product manager needs to know what the endpoint lets the business do, what must be true for it to work, what happens after it succeeds, and what can go wrong for customers.

The paragraph

"Create Order is the call our storefront and partner systems make to place a customer's order. It needs a customer, at least one item that is in stock, and a delivery address; if any of these is missing, no order is created and the caller is told which one. When it succeeds, the order is saved in a 'pending' state, stock for those items is reserved, and the order appears in the customer's history and in the daily sales report. Payment is collected separately, so a pending order can still be abandoned if payment fails. It is safe for a caller to retry after a timeout without creating a duplicate order. Orders above our fraud-review threshold are held for manual approval, which can delay confirmation."

What I included and why

• Purpose in business words: who calls it and what it enables.
• Preconditions that change what customers experience (needs stock, needs address).
• What happens next: reserved stock, pending status, reporting. A PM plans features on these downstream effects.
• Consequences of failure: abandoned pending orders, held orders.
• Anything that changes an outcome: retry safety means no duplicate orders, so a customer is never asked to pay twice for one purchase by a retry (payment itself is collected by a separate call, which needs its own retry protection), which is a customer-facing promise.

What I left out on purpose

• Header names, authentication token format, field types and JSON structure.
• Exact status codes, the database tables touched, internal service names.
• Pagination, versioning details and other things the PM will never act on.

If the same summary were for an engineer, those omitted items become the core; the facts stay the same, the audience changes which ones matter.

Pitfalls

Copying the reference description ("POST /orders creates an order resource") gives the PM nothing to decide with. Also avoid promising business behaviour the code does not implement: check each claim (stock reservation, fraud threshold) with the owning team before it goes in the paragraph.

Direct answer

Developer experience (DX) for an API product is the quality of the whole journey a developer has with it: discovering it, getting a first call working, building an integration, operating it in production, troubleshooting, and upgrading. It is not just docs or pretty SDKs. Judge it by outcomes across that journey, using about five metrics, one per stage, each with a clear numerator (the top of the fraction) and denominator (the bottom). An integration below means one customer's application connected to your API, and a developer usually has one or more. Two status-code families appear: 2xx means the request succeeded and 4xx means the caller made a mistake (bad input, missing auth).

The metrics I would track

Add a small survey (one question, "how easy was it to integrate?") for the human view the numbers miss, but keep it a companion, not the target.

My north star (the one number the team steers by): the activation rate, with the time to first successful call as its speed companion. Activation says whether developers succeed at all; time says how painful it was.

Worked example of the arithmetic

A month with 200 signups, 120 of whom make a successful call within 7 days: activation is 120 / 200 = 60%. The docs change; next month 220 signups and 154 activate: 154 / 220 = 70%. Support: 40 how-to tickets across 500 active integrations is 40 / 500 x 100 = 8 per 100. Numbers are only comparable when denominators are defined the same way each month.

The other three, with made-up numbers:

• Time to first successful call. Ten new developers take 4, 5, 6, 7, 8, 9, 11, 14, 22 and 45 minutes. The median is the middle of the sorted list, (8 + 9) / 2 = 8.5 minutes. The p90 (nearest-rank method) is the 9th of 10 values, 22 minutes, so the slowest tail is what a docs fix should target.
• Client-error share. New integrations sent 1,000 responses in their first 30 days and 150 were 4xx: 150 / 1,000 = 15%. If clearer error messages and examples cut that to 90 of 1,000, it is 9%. A high 4xx share means developers keep sending requests the API rejects, which points at unclear docs or errors (it does not count server failures, which are 5xx and a reliability issue).
• Deprecation migration rate. A deprecated version is an old API version you have announced will be retired, and its sunset date is the day it stops working. If 40 integrations were on v1 and 30 had moved to v2 by the sunset date, the rate is 30 / 40 = 75%. The 10 left behind are the ones to contact before you switch it off.

How each is gamed, and the guard

• Time to first call can be improved by counting a trivial ping: require a real endpoint.
• Support contacts fall if you make contacting support hard: pair with the survey and with client-error share.
• Activation rises if signups are filtered harder: watch signup volume alongside it.

Pitfalls

• Vanity metrics (numbers that look impressive but do not show developers succeeding, such as page views, downloads, GitHub stars) that say nothing about success.
• Twenty metrics: nobody acts on them. Five, one per stage, each with an owner.

Direct answer

Split the key into two tiers. Give every developer an instant, self-serve sandbox key that works in seconds against fake data, and keep the approval step for the production key only. Product gets its under-ten-minute first successful call (measured on the sandbox), and the business keeps its review before real data or real money is touched.

Why this resolves the conflict

The two goals are attacking different things. Approval exists to control risk in production: abuse, fraud, regulated data, cost. The ten-minute target exists to prove the API works and is understandable. A sandbox satisfies the second without weakening the first, because a sandbox key can do no real harm.

What the onboarding experience looks like

The steps add up to 5.5 minutes of budget, leaving slack under the ten-minute target for a slow install or a typo.

Making the approval step itself painless

• Ask for approval data up front and once: use case, expected volume, company, in one short form shown after the first sandbox success, when the developer is motivated.
• Start the review clock at request time, show status ("in review, usually 1 business day") and email on each change. A silent wait is what makes developers leave.
• Automate the easy cases: verified company domain and low volume are auto-approved; only unusual requests go to a human. This reduces the queue you would otherwise have to staff.
• Publish the approval service level (the promised turnaround) and measure against it.
• Keep the sandbox alive while waiting so the developer can build the whole integration during review.

What I would measure

Two numbers, not one: median time from signup to first successful sandbox call (the ten-minute target), and median time from production-access request to key issued (the approval turnaround). The first tests the docs and onboarding, the second tests the review process. Conflating them hides which one is the problem.

What would flip my call

• If the API cannot be sandboxed (every call has real-world effects, for example moving money or reading regulated records), a sandbox key is impossible. Then give a no-key mock explorer with recorded responses so the developer can still see the shape and make a "first call" to the mock inside ten minutes, and treat the true first call as an approval-gated milestone with its own service level.
• If abuse of free keys is severe even in sandbox (it costs real compute), add friction there: email verification is already in the flow, add tight rate limits before adding approval.

Pitfalls

• Making everyone wait for approval "to be safe" and then reporting a ten-minute metric that quietly starts after approval.
• A sandbox that behaves unlike production (different error formats, missing endpoints): developers pass the sandbox and fail in production. Keep parity and document the differences.
• Approving slowly but with no visibility: the wait is tolerable, the silence is not.

Direct answer

(Deflection means a developer solves their problem without opening a ticket.) Start by reading the actual tickets and sorting them by cause, because "cut tickets in half" is only reachable if a large share are the kind that docs, better errors or self-service tools can prevent. Then plan work in order of tickets removed per unit of effort, and measure deflection with a rate (tickets per 1,000 active developers) validated by a holdout, not with raw ticket counts or self-reported "article helped" clicks.

Step 1: find the removable tickets

Pull the last 3 to 6 months of tickets and tag a random sample of a few hundred by root cause and by what would have prevented it.

Worked example (illustrative numbers)

200 sampled tickets: 60 authentication and key setup, 44 unexplained error responses, 30 how-to (pagination, webhooks, which are calls our server makes to the developer's URL when something happens), 26 real bugs, 20 billing and account, 20 other.

docs / self-service addressable: 60 + 44 + 30 = 134
134 / 200 = 67%

Not all 134 will vanish, because some readers still will not read a page. If a good fix removes, say, 60% of a category, the docs-addressable 67% yields about 40% of total volume. Reaching 50% therefore needs more than docs: also fix the API itself (clearer error messages, bugs) and add self-service. State that honestly to support, do not promise 50% off docs alone.

Step 2: plan work as a prioritised roadmap

Order by (tickets in category times expected removal rate) divided by effort. Applying it to the sample, with removal rates and effort points (1 = a few days, 2 = a few weeks) as judgement calls you would revisit:

auth and key setup: 60 x 0.70 / 1 = 42.0
how-to guides:      30 x 0.50 / 1 = 15.0
unexplained errors: 44 x 0.60 / 2 = 13.2

Auth goes first, then the how-to guides, then error messages, which cost more engineering but stay in the first half of the year. Twelve-month shape: quarter one, docs and error messages (cheap, biggest categories); quarter two, self-service tools; quarters three and four, SDK improvements and whatever the re-tagged tickets show still remains. Re-sample tickets each quarter because fixing the top category reveals the next one.

Step 3: measure deflection honestly

• Normalise: raw ticket counts rise when adoption rises. Baseline 1,200 tickets per month with 4,000 active developers is 1,200 / 4,000 x 1,000 = 300 tickets per 1,000 developers. Halving means 150 per 1,000, even if total tickets stay flat because developers doubled.
• Do not count "article views without a ticket" as deflection. Most of those readers would never have filed a ticket anyway. Instead, run a holdout (a small group deliberately left on the old experience, as a control). Example: 2,000 new developers sign up in a month; randomly keep 200 on the old experience and give 1,800 the new error-page link. The held-out group files 30 tickets, 30 / 200 = 15 per 100 developers. The treated group files 180, 180 / 1,800 = 10 per 100. Honest deflection is (15 - 10) / 15 = 33% fewer tickets, measured against what would have happened anyway.
• Track guardrails (numbers that must not get worse while you chase the main goal): developer satisfaction, activation (share of signups who make a call) and time to first call (minutes from signup to the first successful API request) must not fall. A hidden contact form deflects tickets by making developers give up.
• Track category-level rates, so the auth-ticket rate should fall after the auth guide ships, and not just the total.

Pitfalls

• Counting tickets closed by an auto-reply bot as resolved.
• Cutting volume by making support harder to reach.
• Ignoring reopen rate (the share of closed tickets the developer reopens or refiles): a deflected ticket that returns as two tickets is not a win.

Direct answer

I would publish one short documentation standard (a minimum contract for every endpoint plus a style guide), turn it into automated checks that run in CI (continuous integration, the automated checks on every pull request) from one shared ruleset, roll it out in stages so it never blocks people on day one, and win acceptance by making the gate cheap, fair and tied to real pain. A gate that engineers experience as bureaucracy gets bypassed; a gate that saves them support tickets gets defended.

The standard (kept to one page)

• Every operation has a summary, description, documented parameters with types, at least one request example, and all error responses.
• Shared conventions: one error format, one pagination style, naming rules, auth described once.
• Docs live with the code (docs-as-code: plain text files in the repository, reviewed like code) and each service names an owner.
• Changes to public behaviour require a changelog entry.

Automated checks (same job in every repo)

Terms: OpenAPI is the standard machine-readable file that describes an API (its paths, parameters and responses), so a tool can check it; a spec is that file; an operationId is the unique name each operation carries, which SDK generators and links rely on. Start with the first two rows (spec lint, then breaking-change diff): they are cheap, need no running service, and catch most gaps. Contract tests, snippet tests, Vale and preview builds are worth adding later, once the first two are trusted.

extends: ["spectral:oas"]
rules:
  operation-description: error
  operation-operationId: error
  operation-summary-required:
    description: Every operation needs a summary
    severity: error
    given: "$.paths[*][get,post,put,patch,delete]"
    then:
      field: summary
      function: truthy

What the ruleset says, line by line: extends: ["spectral:oas"] starts from Spectral's built-in OpenAPI rules. operation-description: error and operation-operationId: error turn two of those built-in rules into failures. The custom rule operation-summary-required says: given selects every GET, POST, PUT, PATCH or DELETE operation under paths; then says its summary field must exist and be non-empty (function: truthy); severity: error makes a miss fail the build (versus warn, which only prints). To write another rule, change given (where to look) and then (what must hold).

Publish the ruleset in one central repository and version it, so teams upgrade deliberately.

Getting engineers to accept the gate

1. Start warn-only (checks report problems but do not fail the build) for several weeks and publish a scoreboard (a simple table of findings per team) so nobody is surprised.
2. Ratchet, do not cliff. (A ratchet only lets the bar go up, never back down; a cliff is switching every rule on at once.) Enforce as errors only on new and changed endpoints; existing debt sits on a baseline list (a file of known existing violations that are tolerated for now) with expiry dates, and shrinks over time.
3. Make it easy: a template spec, editor and pre-commit checks (checks that run on the author's machine before a commit is recorded) so failures appear before the pull request, and messages that say how to fix.
4. Keep it fast and quiet. A slow or noisy check gets ignored. Track the false-positive rate (how often a warning is about something that is actually fine) and fix or remove rules that cry wolf.
5. Give a documented escape hatch (a sanctioned way to override a rule) (an override needing a reason and a second reviewer) and count overrides; the count is a health signal.
6. Tie it to the pain. Show support tickets and integration delays traced to undocumented behaviour. Recruit one champion (a volunteer who advocates the standard) per team who helps others fix findings.
7. Own the standard jointly with engineers: rule changes proposed through pull requests to the central repo.

Worked example. A team adds DELETE /invoices/{id} with no summary, description or operationId. Run against the ruleset above (executed with Spectral), the change produces three findings: operation-description, operation-operationId and operation-summary-required, plus a built-in warning that the operation has no tags. In week one (warn-only) the pull request shows them as warnings with links to fix guides. After enforcement the same change fails on those three, the author adds a summary, a description and an operationId, and it merges the same day. Note that this ruleset does not check for error responses such as a 404: the standard requires them, so add a custom rule for that or leave it to review. The old undocumented endpoints stay on the baseline until their owning team's quarterly cleanup.

Trade-offs

• Strict rules improve consistency but increase friction: start with a few high-value rules rather than fifty.
• Checking presence is easy; checking accuracy is not, so keep contract tests and human review for correctness.
• Central ownership scales the standard but can feel imposed unless teams help shape it.