API Documentation and Developer Experience Questions

Direct answer

OpenAPI is a machine-readable description format for HTTP APIs (a YAML or JSON file). Treat the spec as a product with two audiences: humans reading rendered reference docs, and code generators producing SDKs (software development kits, client libraries in a given language). Schemas alone give you neither: readable docs need descriptions, examples and errors, and usable clients need stable operation names, auth and servers. Drift means the file slowly stops matching what the running API really does. Keep it from drifting by making CI (continuous integration, the automated checks on every pull request) fail when the spec and the running API disagree.

What goes in the spec beyond schemas

• operationId on every operation. Unique and stable. Generators typically turn it into the SDK method name, so renaming it is a breaking change for SDK users.
• summary, description and tags. These become the docs page text and navigation groups.
• examples on requests and responses. They render as sample payloads and drive mock servers (fake API servers that answer from the examples, so front-end developers can build before the real service exists).
• Every error response, not just 200, referencing one shared error schema.
• servers and securitySchemes. Base URLs per environment, and how auth works (for example a bearer token sent in the Authorization header), so clients configure themselves.
• Reusable components (parameters, responses, schemas) referenced with $ref (a pointer that means "use the definition stored over there"), so pagination or error shapes are defined once.
• deprecated: true on retiring operations, plus vendor extensions (fields starting x-, such as x-codeSamples, an optional extension some doc renderers use to show code samples) for extras.
• Split a big spec across files and bundle it (merge the pieces back into one file) for publishing.

paths:
  /invoices:
    post:
      operationId: createInvoice
      summary: Create an invoice
      description: Creates a draft invoice for an existing customer.
      tags: [Invoices]
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [customer_id, amount_cents]
              properties:
                customer_id: {type: string, example: cus_123}
                amount_cents: {type: integer, minimum: 1, example: 2500}
      responses:
        "201": {description: Created}
        "422":
          description: Validation failed
          content:
            application/json:
              schema: {$ref: "#/components/schemas/Error"}
components:
  schemas:
    Error:
      type: object
      required: [code, message]
      properties:
        code: {type: string}
        message: {type: string}

Reading it line by line: operationId: createInvoice becomes the SDK method name (typically client.invoices.create(...)); summary is the one-line title on the docs page and description is its body text; tags: [Invoices] decides which navigation group and which SDK class the operation lands in; the example values are the sample payload shown in the docs; and the 422 response, pointing at the shared Error schema through $ref, becomes both the documented error body and the error type the SDK raises.

Keeping it from drifting

The standard core set is a linter, a breaking-change diff and a contract test; the rest is optional polish.

1. Lint the spec (a linter is a tool that checks a file against style rules) with Spectral, the common OpenAPI linter: the built-in spectral:oas ruleset only WARNS on a missing operation description or operationId (executed: Spectral run on the shipped spec with both removed printed operation-description and operation-operationId as warnings, 0 errors), and it does not flag a missing example at all, because its example rule only validates examples that exist. So write a custom ruleset that raises those rules to error and adds a rule requiring examples, then make CI fail on errors.
2. Detect breaking changes (edits that make existing clients stop working, such as removing a field or renaming an operation): diff the spec against the main branch (for example with oasdiff, a tool that compares two versions of a spec and reports what changed) and block removed fields or renamed operations without an explicit override.
3. Contract-test the real service: a tool such as Schemathesis (the usual choice today; Dredd is an older alternative you may meet) sends requests generated from the spec to a test deployment and fails when responses do not match. This is the only check that catches code diverging from the spec.
4. Generate an SDK in CI and compile it, so a spec that produces broken clients fails before release.
5. One owner and one review path: spec changes go through the same pull request as the code that implements them.

Trade-offs and pitfalls

• Descriptions rot faster than types, so lint for presence and review for accuracy.
• A hand-edited published copy of the spec is the classic drift source: publish only from the repository build.
• Over-strict schemas make clients fail when the server adds a field: additionalProperties: false means "any field not listed here is invalid", so a validating client rejects a response the moment the server adds a harmless new field. Decide deliberately.

Direct answer

"Above the fold" means the part of the page visible before scrolling. Experienced integration engineers want to see the API work, not read marketing. Above the fold I would put a one-sentence value statement, a primary button to get sandbox keys instantly, a secondary link to the API reference, and a copy-paste first request with the response beside it. The measure of success is how quickly and how often a new visitor reaches a successful sandbox call.

Above the fold

• One line saying what the API does ("Accept and reconcile B2B payments with one API").
• Primary call to action (CTA, the main button): "Get sandbox keys". No sales call and no approval queue for the sandbox.
• Secondary CTA: "API reference".
• A live first request: a curl command for creating a test payment with the visitor's key pre-filled once signed in, language tabs, and the JSON response shown next to it.
• Trust essentials in one row: the status page (live and historical uptime), the changelog (a dated list of what changed in the API, so they can see it is maintained), the SDK list (the client libraries, software development kits, available per language) and the security and compliance page. Integration engineers check these before committing.

Rough sketch of the layout:

[ Logo ]                          [ API reference ] [ Sign in ]

Accept and reconcile B2B payments with one API
[ Get sandbox keys ]  (API reference)

 curl https://api.example.com/v1/payments      { "id": "pay_9137",
   -H "Authorization: Bearer sk_test_..."        "status": "pending" }
   -d amount=2500 -d currency=USD              [ cURL | Python | Node ]

Status | Changelog | SDKs | Security

Leave out hero videos (large autoplay marketing clips), customer logos as the main element, and long feature tours.

Path from landing to a working call

1. Land, read the one line and the sample request.
2. Click "Get sandbox keys", sign up with work email, keys appear immediately.
3. Copy the request with the key filled in, run it, see a 201 response.
4. Next-step cards: "Accept your first real payment flow", webhooks, error handling, go-live checklist.
5. Production access is a separate, clearly described step (business verification) so nobody is surprised.

How to judge success

Define time to first call (TTFC): minutes from account creation to the first successful (2xx) sandbox request. Track:

• Activation rate: share of new sign-ups making a first successful call within 15 minutes.
• Median TTFC for those who do.
• Sandbox-to-production conversion weeks later, which tells you whether easy sandbox success turned into real integrations.
• Docs search queries that return nothing (missing content).

Worked example: 200 sign-ups in a week, 130 with a first 2xx inside 15 minutes: 130 / 200 = 65% activation. If the next design lifts it to 75%, that is a real gain: with 200 sign-ups in each week, the gap of 10 points is about 2.2 times the normal week-to-week wobble (standard error, roughly 4.6 points), so it is likely real, though you would confirm with another week. A lift from 65% to 67% would be only about 0.4 of the wobble, which is noise. A page-view increase would tell you nothing.

Pitfalls

Judging by page views or sign-up counts (vanity metrics, numbers that look good but do not show integration) rewards curiosity, not integration. A pre-filled sandbox key is only safe because it is a test key; never pre-fill live credentials.

Direct answer

An arm is one version of the experience being compared. Randomly assign new developer sign-ups to the widget or the guided quickstart, with one primary success measure that reflects a real integration, a few guardrails, and a pre-agreed decision rule. The biggest trap is measurement: a browser widget can make "successful calls" that are not the developer's own integration, so the primary metric must count something the widget cannot inflate.

Design

• Unit and assignment: randomise per account (organisation), not per user, so colleagues at one company do not land in different arms and share tips. Split 50/50 at first visit to the onboarding page and keep it sticky (once an account is assigned to an arm it stays there on every later visit).
• Hypothesis: the widget increases the share of new developers who make a first successful call from their own code within 7 days.
• Primary metric: first successful API call made with the developer's own key from outside the widget (server-side or CLI, identified by origin or a header), within 7 days of signup. Not "widget runs".
• Secondary: median time to first call; completion of the tutorial; 30-day active use; trial-to-paid conversion (the share of free-trial accounts that become paying) at 30 or 60 days (slow, so read it later).
• Guardrails (measures that must not get worse, even if the main metric improves): support tickets per 100 sign-ups, error rate in the first week, sign-up rate itself (the widget must not scare people off), page load speed.
• Decision rule written beforehand: ship the arm that wins the primary metric with a confidence interval above zero (a range of plausible true differences between the arms; if the whole range sits above zero, the lift is unlikely to be luck) and no guardrail worse than an agreed limit; if tied, choose the cheaper one to maintain, or ship a hybrid (widget followed by a "copy this into your code" step).

Sample size (power calculation)

Say baseline activation is 20% and a 4-point lift (to 24%) is worth acting on. With 5% significance (false-positive rate) and 80% power (chance to detect a true lift):

In words: the more noise in the metric (the variance term) the more accounts you need, and the smaller the lift you care about (p2 - p1, squared) the far more you need. z_alpha (about 1.96 for 5% significance, two-sided) and z_power (about 0.84 for 80% power) are the standard-normal cut-offs for those two error rates, and the formula multiplies their sum, squared, by the variance and divides by the squared lift. Hand check for 20% to 24%: variance = 0.20 x 0.80 + 0.24 x 0.76 = 0.3424; (1.96 + 0.84)^2 = about 7.85; 7.85 x 0.3424 / 0.04^2 = about 1,680.

from math import ceil
from statistics import NormalDist


def n_per_arm(p1, p2, alpha=0.05, power=0.80):
    z = NormalDist().inv_cdf
    z_alpha, z_power = z(1 - alpha / 2), z(power)
    variance = p1 * (1 - p1) + p2 * (1 - p2)
    return ceil((z_alpha + z_power) ** 2 * variance / (p2 - p1) ** 2)


print("activation 20% -> 24%:", n_per_arm(0.20, 0.24), "per arm")
print("trial-to-paid 5% -> 6%:", n_per_arm(0.05, 0.06), "per arm")

Output:

activation 20% -> 24%: 1680 per arm
trial-to-paid 5% -> 6%: 8155 per arm

So about 3,360 accounts for activation. Trial-to-paid needs far more because its base rate is small, which is why it is a follow-up read, not the deciding metric.

A variation: testing auto-generated per-language sample code

The same method also answers whether sample code generated in the visitor's own programming language lifts trial-to-paid conversion. The same structure fits "does generated sample code in the visitor's language lift trial-to-paid conversion?": randomise at signup, primary metric trial-to-paid, secondary first-call rate and time to first call, guardrails on tickets and code-sample error rates. Given the 8,155-per-arm requirement, use first-call rate as the decision metric and check conversion as a longer-running confirmation.

What could make the result misleading

• Metric contamination: widget calls counted as activation (fixed above).
• Sample ratio mismatch (the arm sizes differ from the split you intended): with 10,000 accounts a fair 50/50 split should land within about 100 of 5,000 each (two standard deviations); 5,200 vs 4,800 is four standard deviations out, essentially impossible by chance, so assignment or logging is broken. Check this before reading results.
• Novelty effect (people try something new because it is new): a shiny widget wins week 1 then fades; run at least two full weeks.
• Segment mix: the widget may help beginners and hurt experienced developers; look at pre-declared segments, not fishing.
• Peeking (checking results repeatedly and stopping the moment they look good) inflates false positives. If you must rank the traps: metric contamination and sample ratio mismatch matter most, because they invalidate the result entirely.
• Confounders: a launch or marketing burst mid-test changes who signs up.

Pitfalls

Testing with too few sign-ups and declaring a winner from noise. If traffic is too low for a 4-point lift, test a bigger change or run a small qualitative study first.

Direct answer

Announce early, repeat on a fixed schedule, make the migration guide good enough to follow without asking us, and track migration per partner from day one. I would run a 12-month runway with machine-readable warnings in the API itself, a partner-by-partner usage report, scheduled test outages ("brownouts") near the end, and a hard, documented shutdown that returns HTTP 410 Gone with a pointer to the guide.

Notice schedule

Example: v1 shuts down on 30 June 2027. "T-12 months" means twelve months before that date. A changelog is a dated public list of API changes, and a status page shows live service health and scheduled outages.

The Deprecation header carries the date the endpoint became deprecated and Sunset the removal date. The sunset date must not be earlier than the deprecation date. A partner would see this in their logs on every v1 response:

Deprecation: @1782777600
Sunset: Wed, 30 Jun 2027 23:59:59 GMT
Link: <https://docs.example.com/migrate-v2>; rel="deprecation"

@1782777600 is the deprecation moment as a Unix timestamp (30 June 2026, 00:00 UTC), the Sunset value is an ordinary HTTP date, and the Link header points at the migration guide. A sample announcement email:

Subject: Action needed: Orders API v1 retires on 30 June 2027

Hi Priya,
Orders API v1 will stop working on 30 June 2027. Your key ak_live_41 made
12,300 v1 calls last week. Migrate to v2 using this guide: docs.example.com/migrate-v2.
v2 is available in sandbox from today. Reply to this email or book a call
with your account manager (the person who owns your commercial relationship) if you need help.

The shutdown returns 410 Gone rather than 404 Not Found because 410 says the resource existed and was removed on purpose, so a developer knows it is not a typo or a temporary fault.

What the migration guide must contain

• What changes and why, as a before/after table of fields, paths and behaviours.
• A visual diff of the breaking changes (changes that stop old client code working), so people see them at a glance:

- POST /v1/orders   {"customer_name": "Ada Lovelace", "total": 25.00}
+ POST /v2/orders   {"customer": {"first_name": "Ada", "last_name": "Lovelace"}, "total_cents": 2500}

• Compatibility rules: which v1 behaviours still hold in v2, what is additive, and that unknown fields must be ignored (a client should not crash when a response gains a field it does not recognise).
• Step-by-step migration: get v2 sandbox access, change base URL, update mappings, run your test suite, switch traffic gradually, confirm zero v1 calls.
• Before and after code in the languages partners actually use.
• How to test and roll back, the exact dates, and who to contact.
• Consumer notification: where partners are told about changes to this guide (changelog subscription).

Tracking who has not moved

Log v1 calls per API key and map each key to a partner and an owner. Report weekly: partner, share of calls still on v1, last v1 call. Example:

Stubborn clients and forced changes

• For ordinary deprecations, offer help first (a call, a code review of their integration), then a time-boxed adapter, a translation layer that accepts v1 shapes and calls v2, with its own end date. Extensions are exceptions approved per partner, not defaults.
• For a legally forced field removal with a short timeline (say 30 days), the schedule compresses but the mechanics stay: notify everyone at once, give the exact date, and offer an adapter that keeps the response shape intact but returns the field as null or empty so partner parsers do not crash. Before: {"id": "ord_77", "total": 25.00, "buyer_national_id": "AB123456C"}. After: {"id": "ord_77", "total": 25.00, "buyer_national_id": null}. The key is still present, so a parser that requires it keeps working. You cannot extend the removal for anyone, because the law is the deadline, so contact the biggest callers personally in week one.

Pitfalls

Announcing once and going silent, hiding the date in a long changelog, and cutting off partners who never received a notice because it went to a stale contact are the common failures. Verify contacts before day one.

Direct answer

Spend the first 30 days learning where the pain really is, using existing data and direct conversations, while shipping only fixes that are cheap and obviously right. Spend days 31 to 90 on the structural changes the evidence supports (four are listed below; if the ticket classification shows fewer are justified, do only those), and explicitly defer anything large that the evidence does not yet justify (for example a full docs-platform migration or a new SDK for a long-tail language).

Days 1 to 30: measure and listen

• Baseline the three outcomes: activation (signups reaching a first successful call within 7 days, divided by signups), support tickets per 1,000 active developers, and adoption trend (active developers per month). Without a baseline you cannot claim anything later.
• Classify the support volume: read a sample of a couple hundred recent tickets and tag each with a cause (auth setup, unclear error, missing docs, real bug, billing). This tells you how much volume docs could remove.
• Talk to people: five developers who stalled, five who are thriving, and the support engineers who answer the tickets daily. Support staff know the top ten confusions from memory.
• Audit the docs by walking them: do the quickstart yourself from a clean machine and write down every place you got stuck.

Quick wins to ship inside 30 days

1. Fix the quickstart so it runs end to end (tested by someone else).
2. Add an error-code page for the five most common error messages found in the tickets.
3. Turn the most-asked support answer into a docs page and link to it from the error message.
4. Add a "was this helpful" control and a feedback link so the pipeline of signal exists.

Days 31 to 90: structural changes

• Generate the reference from an OpenAPI description (the standard machine-readable API spec) and add CI checks so docs cannot drift from code.
• Build task guides for the top three jobs developers attempt.
• Automate the samples so they are executed in CI (continuous integration, the build run on every change).
• Set up a monthly docs-and-support review so tickets flow back into docs.

What I would defer

New SDKs (software development kits, language-specific client libraries) in more languages, a portal redesign, and translation. They are expensive, and none should start before the ticket classification shows they would remove real volume.

Worked example of the ticket read

Of 200 sampled tickets, if 60 are "how do I authenticate", 40 are "what does this error mean", 30 are "how do I paginate", 30 are genuine bugs and 40 are billing or account, then 130 of 200 (65%) are plausibly answerable by docs. That justifies putting docs first. If instead 130 were genuine bugs, the right first investment would be in the API itself.

Pitfalls

• Announcing a docs rewrite before understanding the problem. Poor docs can be a symptom of a confusing API design, which docs cannot fully repair.
• Shipping many changes at once, so you cannot tell which helped.
• Treating stalled adoption as purely a docs issue without checking pricing, reliability and product fit.