API Gateways and API Management Questions
The API gateway as a component and the API program it enforces: routing and aggregation, a policy chain for authentication hand-off, throttling hooks, request validation and transformation, protocol translation, telemetry, TLS termination, failure isolation, scaling, multi-region placement, response caching with stampede protection, tenant-aware routing and isolation, traffic steering by version or experiment cohort, and safe policy rollout and testing, plus gateway versus service mesh versus BFF placement of cross-cutting concerns. Also the management surface around it: key and credential provisioning with approval, plan and tier enforcement, separating public from partner audiences, usage analytics, and central policy governance. Excludes throttling algorithms and distributed counters, OAuth/JWT mechanics, load-balancer algorithms, documentation and developer-portal content, and whole-API-surface design.
Sketch an illustrative configuration for an API gateway fronting a few services: routes, how callers are authenticated, a throttling rule, and a cache policy. Then say which concerns you would deliberately keep out of the gateway to avoid centralization risk.
Sample Answer
The idea in plain words
A gateway configuration is a declarative list: which public paths exist, which service each path goes to, what proof of identity each needs, how fast callers may go, and whether answers may be reused from a cache. The skill in the interview is not the syntax (every product differs) but choosing sensible policy per route and knowing which jobs not to put there. The configuration below is illustrative and uses a made-up format, not the syntax of any product.
Illustrative configuration
# Illustrative gateway config. Not the syntax of any real product.
listeners:
- name: public
host: api.example.test
tls:
min_version: "1.2" # refuse older TLS versions
certificate: edge-cert # certificate terminated at the gateway
upstreams:
orders: { address: orders.internal:8080, timeout_ms: 2000, max_in_flight: 200 }
users: { address: users.internal:8080, timeout_ms: 1000, max_in_flight: 100 }
catalog: { address: catalog.internal:8080, timeout_ms: 1500, max_in_flight: 150 }
auth_schemes:
user_token:
type: jwt # signed token sent as: Authorization: Bearer <token>
issuer: https://login.example.test
audience: api.example.test
jwks_uri: https://login.example.test/keys # public keys used to check signatures
partner_key:
type: api_key
header: X-Api-Key # key looked up against the key store
routes:
- path: /v1/orders/*
methods: [GET]
upstream: orders
auth: user_token
required_scope: orders:read # coarse check only
limit: { key: consumer, requests: 100, per_seconds: 60, burst: 20 }
cache: off
- path: /v1/orders/*
methods: [POST]
upstream: orders
auth: user_token
required_scope: orders:write # coarse check only
limit: { key: consumer, requests: 100, per_seconds: 60, burst: 20 }
cache: off
- path: /v1/users/me
methods: [GET]
upstream: users
auth: user_token
limit: { key: consumer, requests: 100, per_seconds: 60, burst: 20 }
cache: off
- path: /v1/catalog/*
methods: [GET]
upstream: catalog
auth: partner_key
limit: { key: api_key, requests: 100, per_seconds: 60, burst: 20 }
cache: { ttl_seconds: 60, vary: [Accept] } # shared cache; public data only
Every field, in plain words
- listeners.tls: the gateway decrypts HTTPS here (TLS termination), refuses protocol versions below 1.2 (TLS 1.0 and 1.1 are deprecated by RFC 8996 because of known weaknesses), and holds the certificate.
- upstreams: the backends. timeout_ms is how long the gateway waits for that service. max_in_flight caps simultaneous requests to it so one slow service cannot use every connection (a bulkhead).
- auth_schemes: how callers prove identity. A JWT (JSON Web Token, a signed token) is verified for signature, issuer, audience and expiry using public keys fetched from jwks_uri. JWKS stands for JSON Web Key Set: a JSON document, published by the login service, that lists the public keys whose matching private keys sign its tokens; the gateway downloads it, caches it and picks the key named in each token. Besides the signature and the expiry time, the claims checked are
issuer(who created the token, which must be the trusted login service),audience(who the token is meant for, which must be this API) and, per route, the scope (what the holder was granted: reading orders and writing orders are separate scopes). An API key is a shared secret passed in a header and looked up. - routes: path pattern, allowed methods (any other method gets 405 Method Not Allowed), the upstream, which auth scheme applies, and the policies. required_scope is a coarse check that the token was granted the permission (here one route per method so that reading orders does not demand the write scope); whether this user may touch this order is decided inside the orders service.
- limit: a token bucket, a counter that refills at a steady rate and spends one token per request. Here requests: 100 per per_seconds: 60 is a refill of 100 / 60 = 1.67 tokens per second, and burst: 20 means a caller can send 20 requests back to back before being slowed. key says what is counted: per consumer (the authenticated caller) or per API key, never per IP address alone, since many users can share one address.
- cache: for the catalog, a response may be reused for 60 seconds (
ttl_seconds, time to live), varying by the Accept header (vary: the named request headers become part of the cache key). It is a shared cache: one stored copy serves many callers, which is why it is only for public data. It is off for orders and user data, which are specific to a caller and change.
A timeline for GET /v1/catalog/items (illustrative): at second 0 a caller sends Accept: application/json; nothing is stored, so the gateway asks the catalog service (a miss) and stores the answer under the key (path, application/json). At second 20 another caller sends the same request: a hit, answered from the cache with no call to the catalog. At second 25 a third caller sends Accept: text/csv: different key, so a miss and a second stored copy. At second 61 the first copy has expired, so the next application/json request is a miss again.
What a throttled caller sees
A caller that has spent its 20 burst tokens gets a rejection. The time until one token is back is 1 / 1.67 = 0.6 seconds; Retry-After is whole seconds (or a date), so the gateway rounds up to 1. This exchange is illustrative:
HTTP/1.1 429 Too Many Requests
Retry-After: 1
Content-Type: application/json
Content-Length: 31
{"error":"rate_limit_exceeded"}
429 is defined in RFC 6585 and means this caller sent too many requests. The Content-Length of 31 is the byte length of the body shown.
Concerns to keep out of the gateway, and why
- Business rules and data-dependent validation (refund limits, who owns an order). They change with the product and need data the gateway does not have.
- Orchestration that calls several services and merges results. It makes the gateway an application with its own bugs, and every team's release depends on it.
- Fine-grained authorization. The gateway sees a token and a path, not the resource.
- Per-service transformations beyond simple renaming and header mapping, which hide behavior from the service's own tests.
- Anything that needs its own database or long-lived state per user.
Centralization risk is that the gateway is on every request path and shared by every team: a mistake in one team's rule can affect all traffic, and one team becomes the queue for everyone's changes. Keep the shared chain thin, keep the config in version control, review it, and roll changes out gradually.
Common mistakes
- One global rate limit for everyone, so the heaviest customer decides everyone's experience.
- Caching a response that depends on who is calling, which can serve one user's data to another. Cache only public, caller-independent data, or include the caller in the cache key.
- Treating the gateway's token check as the only security layer.
Design a multi-region API gateway layer for a microservices platform handling about 200k requests per second across several service domains. How do routing, caching, observability and failure modes work across regions, and how do gateway configuration and upstream schema changes move through CI/CD without outages? Add what the on-call team must watch during a rollout.
Sample Answer
The idea in plain words
An API gateway is the single front door that receives every client request, applies shared rules (authentication, limits, validation, logging) and forwards the request to the right internal service. A multi-region layer runs that front door in several geographic regions at once so that losing one region, or one bad change, does not take the platform down. The central insight: at 200k requests per second the hard parts are not the proxying itself but (1) sizing so that the survivors can carry a lost region, (2) keeping the data plane (the nodes that carry traffic) running when the control plane (the system that pushes configuration) is down, and (3) shipping every configuration change as a small, observable, reversible experiment.
If you only get a minute: size every region for the loss of one, keep nodes serving from their last good configuration, and ship config changes as small reversible canaries. The seven steps below elaborate those three ideas.
All figures below rest on stated assumptions: 3 regions, active-active (every region serves live traffic), traffic split evenly, one gateway node can handle 10,000 requests per second at its limit, and nodes are planned to run at no more than 60 percent of that limit (6,000 requests per second).
Step 1: Clarify requirements
- Availability target and what "outage" means (for example 99.95 percent of requests succeed within 500 ms).
- Which traffic is cacheable, which is personalised.
- Whether services are replicated in every region or only some (this decides whether a region can serve a request alone).
- Data residency rules (legal or contractual requirements that some data stay in a given country or region) that forbid sending some requests across regions.
Step 2: Routing across regions
- Clients resolve one hostname. GeoDNS (DNS that answers with the nearest region) or anycast (the same IP announced from several regions, so the network delivers to the nearest) sends each client to its closest healthy region.
- A health check per region (a synthetic request through the full path, not only a TCP ping) removes an unhealthy region from DNS answers or from the anycast announcement. DNS answers are cached by resolvers, so use a short TTL (time to live, for example 30 seconds) and accept that some clients keep going to the dead region until it expires.
- Inside a region, the gateway routes by host and path to the local service instances first. Cross-region calls are a fallback with an explicit cap, because cross-region latency (tens of milliseconds) and egress cost (the fee clouds charge for data leaving a region) make them a last resort.
- Each gateway keeps its routing table in memory, loaded from a signed configuration snapshot, so no request ever waits on a database.
Step 3: Capacity arithmetic
Steady state: 200,000 / 3 = 66,667 requests per second per region.
If one region fails, the other two carry 200,000 / 2 = 100,000 each, which is 1.5 times their normal load.
Nodes per region must satisfy the failover case: 100,000 / 6,000 = 16.67, so 17 nodes. Check: steady load per node is 66,667 / 17 = 3,922 requests per second (39.2 percent of the 10,000 limit); after losing a region it is 100,000 / 17 = 5,882 (58.8 percent), under the 60 percent plan.
The price of surviving a region loss is 3 x 17 = 51 nodes versus 3 x 12 = 36 nodes if each region were sized only for its own steady load (66,667 / 6,000 = 11.1, so 12): 15 extra nodes, 41.7 percent more. With 4 regions the same rule gives 50,000 steady, 66,667 after a loss, 66,667 / 6,000 = 11.1 so 12 nodes per region and 48 nodes in total, which is why more, smaller regions make failover capacity cheaper. This sizing is only useful if the services behind the gateway and their data stores can also carry the 1.5 times load.
Step 4: Caching
The gateway caches only responses that are safe to share: GET requests with Cache-Control: public (the response may be stored by shared caches) and a max-age (how many seconds it stays fresh), keyed by method, host, path, query and the headers named in Vary (the response header listing which request headers change the answer, so a copy stored for one Accept-Language is not served to another). Personalised responses are private or not cached at the gateway.
Illustrative effect (assumption: 30 percent of requests are cacheable GETs with an 80 percent hit ratio): upstream load falls by 0.30 x 0.80 = 24 percent, from 200,000 to 152,000 requests per second. The danger is the inverse, a thundering herd (many requests hitting the upstream at the same moment): after a region failover the surviving region's cache is partly cold, so upstream load jumps above that figure. Defences: request coalescing (when many requests ask for the same missing key, one fetch goes upstream and the rest wait for its result), serving stale entries for a short window while refreshing in the background, and ramping traffic back to a recovered region in steps rather than all at once.
Step 5: Observability
- Metrics from every request, kept as aggregates: request count, error rate and a latency histogram (counts of requests per latency bucket, from which p50 or p99 can be read; p99 is the time that 99 percent of requests beat) per region, route and status class. These are cheap and exact.
- Traces: sample. At 1 percent, 200,000 x 0.01 = 2,000 traces per second.
- Logs: a full access log of about 400 bytes per request would be 200,000 x 400 = 80,000,000 bytes per second (80 MB/s, about 6.9 TB per day, using 80 MB/s x 86,400 s = 6,912,000 MB). So log all errors and slow requests plus a small sample of successes.
- Every request carries an ID (a correlation ID) generated at the gateway if missing and returned in a response header, plus the W3C
traceparentheader (a standard header carrying the trace id and the calling span's id, so every hop adds its timing to one trace) so a trace spans regions. - A per-region synthetic probe measures what a client sees, independently of the gateway's own metrics.
Step 6: Five failure modes and the response
- Region outage: health checks remove it; the remaining regions carry 1.5 times load (sized above); alert on the load jump.
- Upstream service failing in one region: the gateway uses timeouts, a circuit breaker (stops calling a failing service for a short time) and a capped cross-region fallback; it does not retry blindly, which would multiply load.
- Bad configuration push: staged rollout and automatic rollback (Step 7).
- Control plane outage: nodes keep the last known good configuration and keep serving; only changes are blocked.
- Cold cache or thundering herd after failover: coalescing, stale serving and traffic ramping.
Step 7: Configuration and schema changes through CI/CD
Treat gateway configuration as code in version control, with every route, policy and limit reviewed. Nodes load it as a signed configuration snapshot: one versioned file with a digital signature that each node verifies, so only reviewed configuration is ever applied.
- Static checks: schema validation of the config, lint for routes that shadow each other, policy ordering.
- Contract tests against both the current and the next version of each upstream's API schema (a machine-readable description of the API's requests and responses: an OpenAPI file for HTTP and JSON, or a protobuf file for gRPC). Compatibility rule: the gateway config must work with the old and the new upstream, so the two can deploy independently.
- Staging replay of recorded production traffic, comparing responses with the old config.
- Canary: apply to one node in one region. One node of 17 carries 3,922 requests per second, which is 1.96 percent of global traffic. Five minutes at that rate is about 1.18 million requests (3,922 x 300). How many are enough? Suppose healthy nodes fail 0.2 percent of requests and a bad change would raise that to 0.3 percent, a 0.1 point rise. A standard two-proportion sample size (95 percent confidence, 80 percent power, requests treated as independent) gives the figure the program below prints, computed with these assumptions:
from math import sqrt
node_rps = 3922 # requests per second on one canary node
baseline_error = 0.002 # assumed healthy error rate: 0.2 percent
bad_error = 0.003 # a regression worth catching: 0.3 percent (0.1 point higher)
z_alpha, z_power = 1.96, 0.8416 # 95 percent confidence, 80 percent power
variance = baseline_error * (1 - baseline_error) + bad_error * (1 - bad_error)
needed = (z_alpha + z_power) ** 2 * variance / (bad_error - baseline_error) ** 2
five_minutes = node_rps * 300
noise = sqrt(baseline_error * (1 - baseline_error) / five_minutes) * 100
print(f"requests needed per group: {needed:,.0f}")
print(f"seconds for the canary node to see them: {needed / node_rps:.1f}")
print(f"requests in five minutes: {five_minutes:,}")
print(f"noise in the canary error rate after five minutes: {noise:.4f} percentage points")
Output:
requests needed per group: 39,143
seconds for the canary node to see them: 10.0
requests in five minutes: 1,176,600
noise in the canary error rate after five minutes: 0.0041 percentage points
So about 39,000 requests per group would reveal a 0.1 point rise, and the canary node supplies that in about 10 seconds. The five-minute window is therefore not about error-rate statistics; it is there so latency percentiles settle and slower problems such as connection or cache effects have time to appear. The count shrinks if the baseline error rate is lower or the rise you must catch is larger, and grows if either is the other way.
5. Then one whole region, bake (wait while watching the signals) for a set time, then the next regions, one at a time. A single region is 33 percent of traffic, so the blast radius of each wave is bounded.
6. Rollback is one action: re-point nodes to the previous signed snapshot. It must be tested before it is needed.
Upstream schema changes use expand then contract: first the upstream accepts both old and new shapes, then the gateway transformation switches, then the old shape is removed after the last consumer moves. Example: renaming the response field phone to phone_number. Expand: the upstream returns both fields (and accepts either on input) and the schema documents both. Switch: the gateway transformation and the clients move to phone_number, while request logs are watched for any remaining use of phone. Contract: once those logs show zero use for an agreed period, a later release removes phone. At every stage the previous config still works, so each step can be rolled back alone. Never ship a gateway change and a breaking upstream change in the same release.
What the on-call team watches during a rollout
This table is an illustration of thresholds; real values come from the service's SLO (service level objective, the target for success rate and latency). The pattern behind the numbers: a trigger sits well above the ordinary difference between two healthy groups of nodes (about 0.004 percentage points for the error rate in the calculation above) and well below the level at which customers notice, which is why the 5xx trigger is 0.5 points rather than 0.01. Blast radius, used above, means the share of users or traffic harmed if a change is bad.
| Signal | Compare | Illustrative trigger |
|---|---|---|
| 5xx rate | canary nodes versus untouched nodes in the same region | canary more than 0.5 percentage points higher for 5 minutes |
| p99 latency | canary versus baseline | more than 20 percent higher |
| Upstream connect errors and timeouts | per upstream | any new upstream appearing in the top five |
| Node CPU and connection counts | per node | above 70 percent of capacity |
| Config version per node | across the fleet | nodes more than 10 minutes behind the target |
| Cache hit ratio and upstream request rate | per region | hit ratio drop with upstream rate spike |
| Health-check flapping (a region's health flipping between healthy and unhealthy) | per region | more than two state changes in 10 minutes |
Rollback first, diagnose second.
The same design at 2,000 requests per second
Assume a node limit of 1,000 requests per second, planned at 60 percent (600). 2,000 / 600 = 3.33, so 4 nodes. Spread over 3 availability zones (separate data centres in one region) with 2 per zone gives 6 nodes. Losing a zone leaves 4 nodes x 600 = 2,400 requests per second of planned capacity, above the 2,000 needed. Steady load per node is 2,000 / 6 = 333 (33 percent); after the loss it is 2,000 / 4 = 500 (50 percent). The same rollout stages apply: canary one node (about 17 percent of the fleet at this size, so a smaller bake risk is traded for a bigger blast share), then the rest.
Common mistakes
- Sizing each region for its own load only, so a region loss overloads the remaining regions.
- Treating a TCP check as a health check.
- Letting the control plane sit on the request path.
- Pushing configuration to all regions at once.
- Caching personalised responses.
Judgment
I would run 3 regions active-active with 17 nodes each, GeoDNS plus anycast-style health withdrawal, node-local configuration snapshots, and one-node canaries with automatic rollback on canary-versus-baseline comparison. Moving to 4 regions would lower failover cost (48 versus 51 nodes) at the price of more operational surface; the decision flips if data residency forbids cross-region failover for some traffic, in which case those routes need in-region spare capacity instead.
Design a multi-tenant API gateway layer that centralizes authentication, throttling hooks, routing and observability. Requirements: per-tenant quotas, per-tenant allow and deny lists of endpoints, minimal per-tenant overhead on the request path, and policy changes that take effect without redeploying gateway code. Outline the components, where tenant policy is stored and how it reaches the gateway nodes, and how you would verify tenant isolation.
Sample Answer
The idea in plain words
A multi-tenant gateway serves many customer organizations (tenants) through one fleet of proxy servers. Each tenant has its own policy: how many requests it may send (a quota), which endpoints it may call (an allow list) or may not call (a deny list). The design splits the system in two. A control plane stores and validates tenant policy and decides what every gateway should know. A data plane is the fleet of gateway nodes that handle live requests using a compiled in-memory copy of that policy, so no database is read per request and a policy change is a data push, not a code deploy. Isolation is then something you test, not assume.
Three ideas to state first in an interview: policy is copied to the nodes as a versioned snapshot and never read from the database per request; anything not explicitly allowed is denied; and quotas use leased batches of tokens so the shared counter is touched rarely. A token in this design is permission to serve one request, and the sections below build on that meaning.
1. Components and where policy lives
- Policy store (source of truth). A relational or key-value database holding one record per tenant: plan, rate and quota values, allow and deny rules, version number, who changed it and when. Every change creates a new immutable version (a saved version is never edited, only superseded), so rollback is publishing the content of version N-1 again as a new, higher version.
- Admin API and validation. The only writer. It checks a policy before accepting it (known endpoint ids, no rule that denies everything by mistake, quota within plan limits) and writes an audit record.
- Distribution. The control plane compiles all tenant policies into a snapshot with a version number and nodes fetch it. Two common shapes: nodes poll for "anything newer than version V" every few seconds, or hold a long-lived stream to the control plane and are pushed changes. Polling is simpler and fails gently; I would start with polling plus a jittered interval: each node adds a small random offset to its wait so that hundreds of nodes do not all ask in the same instant.
- Gateway nodes (data plane). On start-up a node loads the latest snapshot and keeps it in memory; it swaps in a new snapshot atomically. The snapshot is compiled: turned once, on arrival, into a ready-to-use lookup table keyed by tenant id (for tenant
acme: its allow rules, deny rules and rate), so a request needs one hash-map lookup and no parsing. To swap, build the new structure beside the old one, then switch one pointer, so a request sees either the old or the new policy and never half of each. If the control plane is unreachable the node keeps the last known good snapshot and keeps serving. - Counter service for quotas. A shared store with atomic increment (one operation that reads the count and adds to it, so two nodes can never both read 599 and both admit a 600th request), used by all nodes (see section 3).
- Telemetry. Every access log line, metric and trace carries the tenant id, so usage and rejections are visible per tenant.
2. The request path and policy evaluation
- Authenticate the request (the credential resolves to a tenant id; the token check itself is the identity provider's job).
- Look up the tenant's compiled policy: one hash-map lookup, no network call.
- Match the request's route id against the policy. Rule: default deny. A route is permitted only if some allow rule matches and no deny rule matches; deny wins over allow.
- Check the tenant's quota and rate limit (section 3).
- Forward upstream, with the tenant id attached as a header the gateway sets itself (overwriting any client-supplied copy).
Illustrative policy for tenant acme (field names are this design's own), snapshot version 42. rate_per_min is the steady allowance: 600 per minute is 10 per second. burst is the size of the token bucket: a tenant that has been quiet may send up to 50 requests back to back, after which it is held to the steady 10 per second. Section 3 works with the steady rate only.
{"tenant":"acme","version":42,
"plan":{"rate_per_min":600,"burst":50},
"allow":["GET /orders/*","POST /orders"],
"deny":["DELETE /orders/*"]}
Decisions: GET /orders/9 matches an allow rule and no deny rule, so it is permitted. DELETE /orders/9 matches no allow rule, and also matches a deny rule, so it is refused with 403 (authenticated but not allowed). GET /invoices/1 matches nothing, so default deny refuses it with 403.
3. Per-tenant quotas with little overhead
Recall that a token is permission for one request. Assume (illustrative) 4 gateway nodes and a tenant with 600 requests per minute, which is 600 / 60 = 10 requests per second.
- Central counter on every request. Exact, but each request pays one network round trip to the counter store: 10 calls per second for this tenant, and the store becomes a dependency on the hot path.
- Token leasing. A node takes a batch of tokens (a lease) from the central counter and spends them locally. With a batch of 20, the central store is called at most about 10 / 20 = 0.5 times per second for this tenant. Because leases are drawn from the same central budget, total admitted traffic never exceeds the quota, provided unspent leases expire when the quota period ends. The cost is stranded tokens: a node that stops receiving traffic holds unspent tokens. With 4 nodes the most that can sit stranded is 4 x 20 = 80, which is 80 / 600 = 13.3 percent of a minute's budget, so a tenant can see a 429 slightly earlier than strict math says. A smaller batch trims that and raises counter traffic. Choose the batch from how much early-429 error you accept.
- Counter-store failure: decide per tenant tier whether to fail open (when the counter cannot be reached, keep admitting requests and accept possible over-use) or fail closed (when it cannot be reached, reject requests, which is safe for the quota but turns a counter outage into an API outage). I would fail open with a conservative local fallback limit, because a counter outage should not become a full outage.
The rejection itself is a 429 Too Many Requests (RFC 6585) with a Retry-After header telling the client how long to wait, as a number of seconds or an HTTP date (RFC 9110). Algorithms and distributed counters are their own subject; here the point is that the policy decides which limit, the counter enforces it.
4. Policy changes without redeploying
Edit policy, validate, store as version 43, publish. A node that polls every 5 seconds and adds up to 1 second of random jitter to each wait picks it up within 6 seconds at worst, plus the time to load and swap it, so the propagation bound is the poll interval plus the jitter (the longest a change can wait before a node asks for it), and the control plane can report each node's applied version so you can see when all four have reached 43. Rollback is publishing the content of version 42 again as version 44. It needs a new, higher number because nodes ask for anything newer than the version they hold, and a node at 43 would ignore a republished 42. Because code never changes, the risk is moved to validation and canary: a canary is a trial on a small slice before everyone gets the change, so roll a new snapshot to one node first, compare its rejection rate with the others, then the rest.
5. Verifying tenant isolation
- Cross-tenant matrix. Generate a test for every pair of tenants and every route: tenant A's credential on a route that only B may call must return 403, and the upstream must see zero requests. Expected results are computed from the policies themselves, not hand-written.
- Header spoofing. A request carrying another tenant's id in a header or path must not change the tenant the gateway attaches.
- Noisy neighbour. Send tenant A at 10 times its quota while tenant B sends normal traffic. B's success rate and latency must stay inside their agreed range, which also needs per-tenant concurrency caps so A cannot fill every worker.
- Shared state. Cache keys include the tenant id; logs, metrics and traces are tagged; one tenant's data never appears in another's log view.
- Config failure modes. A tenant missing from the snapshot is denied (not given a default policy). A corrupt snapshot is rejected and last known good keeps serving. Swaps under load drop no in-flight requests.
- Audit. Every policy change has an author and a version; unexpected cross-tenant 2xx responses raise an alert.
This list is a baseline, not a complete security review; real isolation work also covers credential theft, TLS identity, resource exhaustion by one tenant, and data stores behind the gateway.
Judgment
I would use poll-based snapshots with default-deny and last known good, and token leasing for quotas, because each choice fails toward continuing to serve with slightly stale or slightly conservative policy rather than toward an outage. What flips it: tenants that need a revocation to take effect in under a second need a push channel plus a small deny list consulted per request; very low quota tenants need exact central counting.
Common mistakes
- Reading the policy database on every request.
- Per-tenant gateway deployments "for isolation" that multiply cost and make every change N rollouts.
- Letting a missing policy default to allow.
- Testing isolation only with one tenant at a time.
Design the edge security posture of an API gateway for partner and mobile traffic. Requirements: tenant-scoped limits, protection from volumetric abuse, mutual TLS for partners, tamper-evident audit logs, and low latency worldwide. Where do TLS termination, caching and authentication enforcement sit relative to the CDN or WAF, and how do you manage certificates and test the whole flow?
Sample Answer
The idea in plain words
The edge is the outermost layer that internet traffic meets before it reaches your own servers. The origin is your own backend, here the API gateway and the services behind it, as opposed to the edge servers in front of it. For partner and mobile traffic the edge is built in layers, each doing the job it is positioned to do: a CDN (content delivery network, servers spread around the world close to users) and a WAF (web application firewall, which filters malicious requests) soak up volumetric abuse (floods of junk traffic meant to use up your bandwidth, as opposed to clever single requests) and terminate TLS for public clients (TLS is the encryption layer under HTTPS, and terminating it means being the device that decrypts); the API gateway behind them knows who the caller is and applies tenant limits and authentication; services check object-level permissions. The insight is placement: a control only works where the traffic it must see actually passes, and where the information it needs (such as the tenant) is available. Partner mutual TLS (mTLS, where the client also proves its identity with a certificate) is the case that breaks the simple "terminate everything at the CDN" picture. The core of the answer is placement of TLS termination, authentication and limits (Steps 1 to 4); certificates and the audit log (Step 5) and the latency arithmetic (Step 6) support it, and the remaining steps are checks.
Step 1: The layers and what each enforces
Request path for mobile and other public clients:
client, then DNS or anycast (one address announced from many locations, so the network delivers each client to the nearest one) to the nearest edge, then CDN and WAF (terminate TLS, soak up DDoS, filter, cache public responses), then a second TLS connection to the origin gateway, then the gateway (authentication, tenant limits, validation, audit), then services.
| Layer | Enforces | Why here |
|---|---|---|
| Edge network (DDoS protection) | Volumetric floods at the network and transport layers | Only large spare capacity close to the sources can survive floods larger than your origin links |
| CDN and WAF | TLS termination for public traffic, bot and injection filtering, coarse per-IP limits, caching of public GETs | Sees the request in plain text early, and is close to users |
| Gateway | Authentication, tenant-scoped limits, schema validation, audit log, partner mTLS | The tenant and caller identity are known only after credentials are checked |
| Services | Object-level authorisation and business rules | Only they know what an object means |
Origin lock (making the origin unreachable except through the CDN): the origin accepts connections only from the CDN (allow-listed address ranges, or mutual TLS from the CDN, or a shared secret header checked at the gateway). Otherwise an attacker finds the origin address and sends traffic around the WAF. Test this explicitly (Step 8).
Step 2: TLS termination and mutual TLS for partners
TLS termination means the device that decrypts the traffic holds the server certificate. A CDN that terminates TLS creates two separate TLS connections: partner to CDN, and CDN to origin. The partner's client certificate is presented only on the first one, so it is validated, or not, at the CDN. The second connection is a new handshake in which the CDN is the client, so the gateway sees the CDN's certificate, never the partner's, unless the CDN forwards the partner's identity in a header. Forwarded identity headers are only as trustworthy as the channel carrying them.
Recommendation: give partners their own hostname (for example partners.example.com) whose traffic reaches the gateway listener directly, where the gateway requires and validates a client certificate. Direct does not mean unprotected: a network-layer DDoS service can sit in front and forward the raw TCP bytes without decrypting them. It is not a CDN because it never holds the server certificate or reads HTTP, so the TLS handshake still runs end to end between the partner and the gateway and the client certificate arrives intact. Public and mobile traffic uses the CDN hostname. Reasons: the certificate chain is checked by the component that enforces authorisation; partner traffic is predictable and low volume, so losing CDN caching costs little; and the audit log can record the verified certificate identity. If the CDN offers client-certificate validation and your risk assessment accepts it, the alternative is to validate there and forward a signed identity assertion, with the origin accepting it only over the locked CDN-to-origin channel.
Step 3: Tenant-scoped limits and abuse protection
- Edge: only things visible without credentials, such as IP address, network, request rate and request shape. Per-IP limits are coarse (many users share a mobile carrier IP address), so keep them loose.
- Gateway: per-tenant limits keyed by the verified client or tenant ID, because that identity exists only after authentication.
- Together: the edge protects the gateway's capacity; the gateway protects tenants from each other.
Step 4: Caching at the edge
Cache only responses that are the same for everyone: public data with Cache-Control: public, max-age=.... Authenticated responses use private or no-store unless the cache key includes the tenant. Never rely on Vary: Authorization (a header telling caches to keep a separate copy per value of the named request header) to separate users; it creates one cache entry per token. A mistake here leaks one tenant's data to another, so a test that requests the same URL as two tenants and compares bodies belongs in the pipeline.
Step 5: Certificates and audit logs
Certificate management:
- Server certificates: automated issue and renewal (the ACME protocol, a standard way for software to request and renew certificates from a certificate authority without a human), short lifetimes, alerts at 30, 14 and 7 days before expiry.
- Partner client certificates: issued by a private certificate authority (CA, the signer) you control, valid for a limited period (for example 90 days), with an overlap window so a partner can install the new certificate before the old one expires, and a revocation path (a revocation list the gateway reloads, or simply very short lifetimes).
- Mobile apps: pinning means the app accepts only a specific certificate or key instead of any certificate a trusted authority signed. A certificate chain runs from the leaf (the server's own certificate, which rotates often) through an intermediate (a CA certificate that signed the leaf) to a root. Avoid pinning the leaf. If you pin, pin the public key of an intermediate and include a backup pin; a wrong pin locks every installed app out until users update.
Tamper-evident audit logs: each entry stores the hash of the previous entry, so changing or deleting one breaks every later hash. Illustrative: entry_n stores hash_n = SHA-256(hash_(n-1) followed by the entry text). A three-entry chain, with each hash shown as its first 8 hex characters for reading (the chain itself uses the full 64), shows what verification sees:
import hashlib
def link(prev_hash, text):
return hashlib.sha256((prev_hash + text).encode()).hexdigest() # full 64 hex characters
entries = [
"1 allow tenant=acme route=/orders",
"2 deny tenant=acme route=/admin",
"3 limited tenant=zed route=/orders",
]
def build(entries):
chain, prev = [], "start"
for text in entries:
prev = link(prev, text)
chain.append((text, prev))
return chain
def first_break(stored, entries):
prev = "start"
for i, (text, saved) in enumerate(zip(entries, [h for _, h in stored])):
prev = link(prev, text)
if prev != saved:
return i + 1
return None
stored = build(entries)
for text, h in stored:
print(h[:8], text) # shortened to 8 characters only for display
print("intact:", first_break(stored, entries))
print("entry 2 deleted:", first_break(stored, [entries[0], entries[2]]))
print("entry 2 edited:", first_break(stored, [entries[0], "2 allow tenant=acme route=/admin", entries[2]]))
Output:
df6198f4 1 allow tenant=acme route=/orders
3f08517f 2 deny tenant=acme route=/admin
4bd5a95b 3 limited tenant=zed route=/orders
intact: None
entry 2 deleted: 2
entry 2 edited: 2
Each stored hash covers everything before it. Verification recomputes the chain from the first entry and reports the first position where the recomputed hash differs from the stored one: None means the chain is intact, and both deleting entry 2 and editing it are caught at position 2. The chain alone has two limits. An attacker who rewrites every later hash produces a chain that verifies, and cutting entries off the end leaves a shorter chain that still verifies. So ship entries immediately to storage that the gateway can append to but not rewrite, and periodically write the most recent hash to a separate system (an anchor) that the attacker cannot reach. An attacker with access to the gateway host can still stop logging, so alert on a gap in sequence numbers or a missing heartbeat. The log records caller identity, tenant, route, decision (allow, deny, limited), request ID and time, never request bodies containing secrets.
Step 6: Global latency, with arithmetic
Setting up a new HTTPS connection costs round trips before the first request byte can be sent: 1 round trip for the TCP handshake plus 1 for TLS 1.3 (RFC 8446), so 2 round trips. Assume a user in a distant region has a round-trip time (RTT) of 150 ms to the origin and 20 ms to the nearest edge.
- Directly to origin: 2 x 150 = 300 ms before the request is sent.
- To the edge: 2 x 20 = 40 ms; the edge keeps warm connections to the origin, so the user does not pay the long path for setup.
HTTP/3 runs over QUIC, a transport built on UDP that carries the TLS 1.3 handshake inside its own connection setup, so there is no separate TCP handshake: the client's first packet starts both, and one round trip completes them (20 ms at the edge). The assumed RTTs are illustrations, not measurements of any real network. The edge also removes the setup cost for repeat visits through connection reuse.
Step 7: Mobile-client specifics
- Mobile networks change addresses and drop connections; HTTP/3 survives address changes better than TCP.
- Retries by the app must be bounded with jitter, and non-idempotent calls carry an idempotency key.
- Per-IP limits hit shared carrier addresses, so prefer per-device or per-account limits at the gateway.
- Include the app version in a header to allow forced upgrade and targeted blocking.
- Keep configuration endpoints cacheable at the edge.
Step 8: Testing the whole flow
- Functional TLS tests: valid partner certificate accepted; expired, wrong-CA, revoked and missing certificate each rejected, checked with
openssl s_client -cert ... -key ...orcurl --cert ... --key .... - Origin bypass: connect directly to the origin address from a non-CDN network; it must fail.
- WAF rules first in log-only mode against replayed production traffic, then enforce.
- Cache isolation: two tenants, same URL, different bodies.
- Limit tests: a noisy tenant cannot raise another tenant's 429 rate.
- Audit: delete a log entry in a copy; verification must fail at that point.
- Performance: synthetic probes from several continents report connection, TLS and first-byte times separately; load tests ramp to peak plus headroom.
- Certificate expiry drill in staging.
Common mistakes
- Putting authentication only at the CDN and leaving the origin reachable.
- Trusting identity headers from the CDN without locking the origin.
- Caching authenticated responses with a shared key.
- Pinning a certificate with no backup.
Judgment
I would terminate public and mobile TLS at the CDN with a locked origin, serve partners on a separate hostname with mTLS validated at the gateway, enforce tenant limits and authentication at the gateway, and chain audit entries with an external anchor. What would flip the partner decision: a CDN that validates client certificates and a risk review that accepts forwarded identity.
Compare an API gateway with a service mesh: what each one owns (north-south vs east-west traffic), their performance and operational costs, and when you would pick one or both. Give an example architecture that uses both and justify the split of responsibilities.
Sample Answer
The idea in plain words
Traffic into your system from outside (browsers, mobile apps, partners) is called north-south. Traffic between your own services is called east-west. An API gateway is the front door for north-south traffic: it deals with callers you do not control. A service mesh is a layer of proxies that handles east-west traffic: it deals with callers you do own. The mesh usually puts a small proxy process (a sidecar, a helper process running next to each service instance) beside every service, and a control plane (the management component that pushes policy to the proxies) configures them all. Most mature platforms use a gateway; only some need a mesh. Two terms used below: idempotent calls are ones that can be repeated without changing the outcome beyond the first time (reading a record, or setting a value to X), so a retry is safe; circuit breaking means that when a service keeps failing, callers stop calling it for a short while instead of piling on, then probe it again.
What each one owns
| Question | API gateway | Service mesh |
|---|---|---|
| Whose traffic | External clients, partners | Service-to-service calls inside |
| Identity of caller | API key, user token, tenant | Workload identity (a verifiable name for which service is calling, such as orders, rather than who the end user is), usually via mutual TLS (mTLS), where both sides present certificates: a certificate is a digital identity document signed by a trusted authority, so each side can check who the other is and the traffic is encrypted |
| Typical policies | Authentication, plans and quotas, request validation, response caching, version routing, TLS termination for public names | Encryption between services, retries and timeouts per call, circuit breaking, traffic splitting between versions (sending a chosen share of calls to a new version), service-level metrics (counts, error rates and latencies per service) and traces (the recorded path of one request across services) |
| Who configures it | A platform or API team, per route | A platform team, per service or namespace (a named group of services in the cluster, used to scope policy) |
| Unit of scale | A small fleet of gateway instances | One proxy per service instance, so the count grows with the platform |
Costs
Performance. Each proxy a request passes through adds processing time and a little latency. Assume 1 ms per proxy traversal (an assumption; it depends on the proxy, the policies and the load). Take one request: client to gateway, gateway to orders, orders calls inventory, orders calls pricing, in sequence.
- Gateway only: 1 proxy traversal (the gateway) = 1 ms.
- Gateway plus mesh: gateway = 1, gateway to orders arrives through orders' inbound sidecar = 1, orders to inventory goes out through orders' sidecar and in through inventory's sidecar = 2, orders to pricing the same = 2. Total 1 + 1 + 2 + 2 = 6 traversals = 6 ms.
That is 5 ms more on this path under the stated assumption, a small share of a typical user-facing latency but not nothing for deep call chains.
Resources and operations. If the platform has 200 service instances and each sidecar reserves 0.1 CPU core and 50 MB of memory (assumptions), the sidecars reserve 200 x 0.1 = 20 CPU cores and 200 x 50 = 10,000 MB (about 10 GB, using 1,000 MB per GB). The gateway for the same platform might be a dozen instances. The mesh also adds a control plane to operate, version upgrades that touch every workload, and a new place to debug when a call fails (is it the application, the sidecar, or policy?).
When to pick which
- Gateway only: you have external consumers needing keys, plans and quotas; services are few enough that library-level resilience (retries and timeouts written into each service's own code through a shared library) and TLS inside the private network are acceptable.
- Mesh only: uncommon. A system with no external API but many internal services needing uniform encryption and retries.
- Both: you have external consumers and a large service estate (the whole set of services you run) with a real east-west need, such as encrypted and identity-checked calls between services, uniform retry and timeout policy across many languages, or traffic shifting between versions of internal services. This is a common shape at scale.
- Neither yet: a handful of services behind a framework or reverse proxy.
The policy that most often justifies a mesh on its own is uniform encrypted, identity-checked calls between services; retries, timeouts, splitting and telemetry are useful additions that libraries can also supply. I would add a mesh when a concrete east-west requirement exists that libraries cannot meet well, and a team exists that will own it. The cost grows with the number of services, so it must be paid for by a need.
Example architecture using both, with the split
Assumed estate: public web and mobile clients, a partner API, 60 internal services in two languages.
- Gateway (north-south): terminates TLS on public hostnames, validates tokens and API keys, applies per-consumer rate limits and plans, routes by path and version, caches public catalog responses, writes the first request ID and access log. It forwards a verified identity to the services.
- Mesh (east-west): encrypts every service-to-service call with mutual TLS, applies a default timeout and a limited retry policy for idempotent calls, does canary traffic splitting of a new orders version (a canary sends a small share, for example 5 percent of calls, to the new version first and compares it with the old), and emits per-call metrics and traces.
- Boundary: the gateway is itself inside the mesh, so the hop from gateway to the first service is also encrypted and identified. User-level authorization stays in the services.
Justification of the split: the gateway needs knowledge of consumers and plans that do not exist inside the cluster; the mesh needs a proxy beside every instance, which would be pointless and costly at the edge. Each is the cheapest place to enforce its own kind of traffic.
Common mistakes
- Saying the mesh replaces the gateway (it does not manage external consumers, keys or plans) or that the gateway replaces the mesh (it does not see calls between services).
- Doing the same retry in the gateway, the sidecar and the client library, which multiplies attempts: three layers each making up to 3 attempts can send up to 3 x 3 x 3 = 27 attempts for one user request. Decide which layer owns retries. A one-request timeline shows why. Suppose inventory is failing. The gateway calls orders (gateway attempt 1). Inside orders, the client library calls inventory, and each library attempt goes out through orders' sidecar, which itself tries up to 3 times. So one library attempt reaches inventory up to 3 times, the library makes up to 3 attempts, that is 3 x 3 = 9 calls to inventory, and orders then returns an error to the gateway. The gateway retries orders twice more (attempts 2 and 3), each repeating the 9 calls. Total at inventory: 3 x 9 = 27 for one user request.
- Adopting a mesh for a dozen services with no team to operate it.
That is every published API Gateways and API Management question for Cloud Engineer so far. Browse the other topics in this category, or practice this one interactively.