Multi-Tenancy and Isolation Questions

Direct answer

A cross-tenant leak runbook has one organizing idea: every hour you work out two things faster, which tenants' data went where, and whether it is still flowing. The runbook moves through detection (automated tenant-mismatch signals plus customer reports), containment in minutes (kill switches, cache flush, token revocation), evidence preservation before anything is changed further, an exposure matrix that says which source tenant's data reached which recipient tenant, customer and regulator notification on legally driven clocks (GDPR's 72 hours for controllers is the one that sets the pace), and remediation that closes the whole class of bug, not just the instance.

Roles first

A leak is a security incident and a customer incident at the same time. Assign at declaration:

Legal is paged at declaration, not after the investigation, because notification clocks start at awareness, and because counsel may want the investigation run under legal privilege (communications made for the purpose of legal advice that cannot be compelled into evidence later, sometimes used to keep an incident investigation's internal findings confidential).

Phase 1: Detection signals

• Tenant-mismatch assertions in the data access layer. After every query, the repository checks that every returned row's tenant_id equals the request's tenant context. A mismatch is blocked and alerted. This is the single most valuable signal because it fires on the first leaked row.
• RLS denials or unexpected spikes in rows filtered by row-level security (RLS: the database silently filtering rows by a tenant policy) from a code path that normally sees none.
• Cache-key anomalies: a cache entry read under one tenant context that was written under another (store the writer's tenant ID inside the cached value and compare on read).
• Customer reports: "I can see an invoice that isn't ours." Support must have a one-click path to page security, with a script that says do not ask the customer to send screenshots containing the other tenant's data by email.
• Canary tenants: synthetic tenants whose unique marker strings should never appear in another tenant's responses, exports, or search results.
• Audit-log anomalies: an API key or user fetching object IDs belonging to many tenants (an IDOR pattern, insecure direct object reference: fetching an object by guessing its ID without an ownership check).

Phase 2: Immediate containment (target: first 30 minutes)

Containment beats diagnosis. In order:

1. Stop the flow. Flip the feature flag (a runtime switch that turns a code path off without a deploy) for the suspected endpoint, or roll back the last deploy if the timing correlates. If neither is clear, put the affected surface into read-only or maintenance mode for the implicated tenants.
2. Purge poisoned shared state. First export a snapshot of the affected cache keys and the tenant ID each one was storing, so Phase 3 can still size the exposure after the flush; then flush the relevant cache namespaces, invalidate CDN (content delivery network) objects for affected routes, and pause export or email jobs that might be sending leaked data out.
3. Revoke what may have been exposed. Rotate any credentials, API keys or session tokens that could have crossed the boundary.
4. Do not delete. Containment must not destroy evidence: disable, do not drop.

Phase 3: Forensic evidence collection

Before anyone "cleans up":

• Snapshot affected databases, and export the relevant logs, traces, CDN logs and audit logs to a write-once location (for example object storage with a retention lock, a setting that blocks deletion or modification for a fixed period, even by an administrator) with restricted access.
• Record hashes of every exported artifact and who collected it, when (chain of custody: an unbroken record of who handled the evidence and when, so it can later be shown that nothing was altered along the way).
• Freeze log retention for the incident window so rotation does not erase it.
• Capture the deployed build identifiers and configuration at the time of the incident.

Then build the exposure matrix, the artifact everything downstream depends on. Worked example of sizing it: the leak was a cache key missing the tenant prefix on /invoices/{id}, live for 108 minutes. Access logs show 1,940 requests to that route in the window; joining the request's tenant (from the access log) against the tenant that value was cached under (from the pre-flush cache export made during containment) finds 37 responses served cross-tenant, touching data of 6 source tenants seen by 9 recipient tenants. Those 6 source tenants are the ones owed a breach notice; the 9 recipients may be asked to confirm deletion of what they received.

Of those 37 leaked responses, 6 went from Tenant A to Tenant C; since each response is one invoice and an invoice can hold many line items, those 6 invoices held about 120 line items between them. The matrix records that pair like this:

("Records" here counts individual line items inside a leaked response, not the response count itself, since one leaked invoice can carry many line items.)

Phase 4: Notification timelines

The SaaS provider is usually a processor (acting on the customer's instructions) and the tenant is the controller (who decides why and how personal data is processed). That sets who tells whom.

Practical rule: aim to send an initial notice within 24 hours to affected source tenants even with incomplete facts, then follow up as the exposure matrix firms up. A late but complete notice is worse than an early partial one, because the tenant's own 72-hour clock may already be running. Recipient tenants get a separate, carefully worded request to delete and confirm, which must not reveal the source tenant's identity.

Phase 5: Long-term remediation

Fix the class, not the line:

• Make tenant scoping structural. A tenant-scoped repository or ORM (object-relational mapping) layer that refuses unscoped queries; RLS in the database as the backstop.
• Tenant in every key. Cache keys, object storage paths, search index filters and queue message envelopes all carry the tenant ID, enforced by shared libraries rather than convention.
• Automated cross-tenant tests in CI: a suite that authenticates as tenant A and attempts to read every resource type belonging to tenant B, expecting 404.
• Keep the tenant-mismatch assertion permanently in production.
• Blameless post-incident review within two weeks, with owners and dates for each action, and a customer-facing summary for affected tenants.

Trade-offs and pitfalls

• Rolling back vs disabling a flag. Rollback is fast but may also revert an unrelated security fix; a flag is surgical but only exists if someone built it. Build kill switches for every tenant-data-serving path in advance.
• Pitfall: cleaning up before snapshotting. Flushing the cache without first recording which keys held which tenant's data destroys your ability to scope the exposure matrix.
• Pitfall: notifying recipients before sources. Recipients learn a competitor's data existed before the owner does.
• Pitfall: over-notifying. Telling all 4,000 tenants "there may have been an issue" when the matrix shows 6 affected creates panic and regulator questions. Scope first, fast.

Direct answer

Three tiers with RPO (recovery point objective: maximum data loss) and RTO (recovery time objective: maximum downtime) ranging from 1 minute/15 minutes to 12 hours/24 hours cannot share one DR (disaster recovery) mechanism without paying Platinum prices for everyone. The key insight is that replication is configured per database cluster, not per tenant, so I place tenants into tier-specific cells (independent stacks of database plus services) and give each cell a different DR pattern: Platinum cells run a hot standby in a second region fed by continuous replication; Gold cells run pilot light (data continuously shipped to the second region, compute provisioned only on failover); Silver cells use backup and restore from cross-region backup copies. Each tier is tested at a cadence proportional to its promise.

Requirements

Assumed scale for the worked numbers: Platinum 40 tenants, 6 TB; Gold 400 tenants, 20 TB; Silver 6,000 tenants, 40 TB.

Architecture

flowchart TB
  GR[Global router: tenant to cell map]
  GR --> PC[Platinum cell, region 1]
  GR --> GC[Gold cell, region 1]
  GR --> SC[Silver cell, region 1]
  PC -->|async replication, seconds| PS[Platinum hot standby, region 2]
  GC -->|log shipping every 60s| GB[Gold backup bucket, region 2]
  SC -->|snapshot copy every 6h| SB[Silver backup bucket, region 2]
  GB -.->|on failover| GN[Gold compute launched from IaC]
  SB -.->|on failover| SN[Silver rebuilt from snapshot]

The global router maps each tenant to a cell and is the thing you flip on failover; it must itself run in both regions.

Replication strategy per tier

Platinum: hot standby (active-passive)

A hot standby is a full-size, continuously updated replica that can take over in minutes; a warm standby is a smaller or partly-stopped copy that needs manual work to become the primary; a cold standby is just backups with nothing running. Platinum needs the first of these, kept as an active-passive pair: one region actively serving traffic (active) while the other continuously replicates and stands ready to take over (passive) without serving requests itself.

• Asynchronous cross-region replication of the database (for example PostgreSQL streaming replication, where the standby continuously receives and applies the primary's write-ahead log, or a managed cross-region database feature). Synchronous cross-region replication would give RPO 0 but add a cross-region round trip (tens of milliseconds) to every commit; with a 1-minute RPO, asynchronous replication is enough and keeps writes fast.
• Alert when replication lag exceeds 30 seconds, half the RPO, so there is time to act before the promise is broken.
• Application tier in region 2 running at full size (not scaled down): scaling up a cold fleet during a regional event competes with every other customer doing the same.
• Queues, object storage and secrets replicated too; a database with no replicated file attachments is not a recovered tenant.

The 15-minute RTO has to be budgeted, not hoped for:

Gold: pilot light

• Continuous WAL (write-ahead log) archiving every 60 seconds to a region-2 bucket, plus a daily base backup copied cross-region. Worst-case data loss is about one archive interval plus copy delay, comfortably inside 15 minutes.
• On failover, infrastructure as code (IaC) creates the cell's database and services in region 2 and restores from base backup plus WAL. The RTO depends on restore throughput; at an assumed 500 MB/s, restoring the 20 TB Gold cell is 20,000,000 / 500 = 40,000 s, about 11.1 hours. That misses 2 hours. So Gold is split into cells of at most about 3 TB, each restorable in 3,000,000 / 500 = 6,000 s (100 minutes) in parallel, or Gold keeps a small always-on replica. I would split into seven cells of about 2.9 TB each (20 / 7): smaller cells also shrink the blast radius (blast radius: how much is affected when something goes wrong, here how many tenants and how much data one restore or failure touches). The throughput figure is replaced by the measured value after the first drill.

Silver: backup and restore

• Snapshots every 6 hours copied to region 2. The worst-case loss is one interval plus copy time (roughly 6 to 7 hours), inside 12 hours with margin for a failed copy.
• Rebuild in region 2 from IaC and the latest snapshot copy. With a 24-hour RTO, restoring 40 TB at the same assumed 500 MB/s (22.2 hours) is too tight to run serially, so Silver is split into four 10 TB cells restored in parallel (about 5.6 hours each).
• Silver restores after Platinum and Gold in a regional event, which is the whole point of tiering: restore capacity is also contended.

Backup cadence (independent of DR)

DR replication does not protect against bad writes; a replica faithfully copies a mistaken DELETE. All tiers therefore also keep point-in-time recovery backups (35 days) in a separate account with a retention lock (a setting that blocks deletion, even by an admin, until a set date passes). Platinum and Gold additionally get per-tenant logical exports nightly for single-tenant restores.

Failover testing

The deliverable of every test is a measured RPO and RTO compared against the contract. A test that is not measured is a demo.

Cost trade-offs

Let a cell's normal monthly running cost be C (compute plus primary storage), and assume compute is about 60% and storage about 40% of C:

So Platinum DR roughly doubles a cell's cost, which justifies charging Platinum tenants a meaningful premium and justifies keeping them on dedicated cells: putting 40 Platinum tenants in the same cell as 6,000 Silver tenants would force the hot standby cost onto 40 TB of Silver data.

Trade-offs and pitfalls

• Pitfall: tier promises on shared infrastructure. If a Gold tenant sits on the Platinum cell they get Platinum DR for free; if a Platinum tenant sits on a Silver cell the contract is quietly broken. The tenant-to-cell map is the source of truth, and a tier upgrade is a migration between cells.
• Pitfall: forgetting dependencies. Authentication, the global router, and payment integrations must meet the Platinum RTO too, or the Platinum cell recovers into a region where nobody can log in.
• Pitfall: failback. Failback (moving traffic back to the original region once it has recovered) needs reverse replication and a second planned cutover; plan and test it, because an unplanned failback is a second outage.
• What would change the design: an RPO of 0 for Platinum would force synchronous replication (or a multi-region consensus database, where nodes in multiple regions must agree on each write, for example via Raft or Paxos, before it commits) and accept the write-latency cost.

Direct answer

First I stop the bleeding without guessing: confirm which tenant is saturating which resource, then contain that tenant (cap or cancel its runaway queries, or move its work to separate compute) rather than scaling up the shared warehouse for everyone. Once the other tenants recover, the structural fix is to make the shared warehouse refuse to let any one tenant take more than its share: per-tenant concurrency and runtime limits enforced at admission, cost-based routing of heavy queries to an isolated pool, and elastic overflow capacity that turns on only while there is queueing. That buys isolation for the price of occasional burst capacity instead of permanent over-provisioning.

Immediate response (the first 30 minutes)

I will use Snowflake to make this concrete; the same moves exist on other warehouses (named at the end of this section). A virtual warehouse in Snowflake is a compute cluster; queries queue when it runs out of slots (the fixed number of queries a warehouse of a given size can run at once; extra queries wait until one frees up).

1. Confirm the diagnosis (5 minutes). "Timeouts for other tenants" could also be a platform issue. Look at what is running and queued, grouped by tenant. This works only if every query carries a tenant tag (for example the session's QUERY_TAG set to the tenant ID by the application):

SELECT query_tag,
       execution_status,
       COUNT(*)                        AS queries,
       SUM(total_elapsed_time) / 1000  AS elapsed_s,
       SUM(queued_overload_time) / 1000 AS queued_s,
       SUM(bytes_scanned) / POWER(1024, 3) AS gib_scanned
FROM TABLE(INFORMATION_SCHEMA.QUERY_HISTORY_BY_WAREHOUSE(
       WAREHOUSE_NAME => 'SHARED_WH', RESULT_LIMIT => 10000))
WHERE start_time > DATEADD('minute', -30, CURRENT_TIMESTAMP())
GROUP BY query_tag, execution_status
ORDER BY elapsed_s DESC;

queued_overload_time is the time queries waited because the warehouse was full; high queueing for many tags plus most of the elapsed time for one tag is the noisy-neighbour signature. If the queries have no tenant tag, that gap is itself the first structural finding.

2. Contain the offender, least destructive action first.

3. What I deliberately do not do first: resize the shared warehouse. It works, but it spends money on behalf of one tenant, teaches that tenant nothing, and the extra capacity is often consumed by the same tenant within minutes because its job simply fans out wider (splits its work into more parallel pieces to use whatever capacity is available, rather than finishing the same amount of work faster).

4. Communicate. Tell affected tenants the impact window; tell the offending tenant (through its account owner) what was capped and why. Keep the query IDs and the grouped output above as the incident record.

Equivalents elsewhere: Amazon Redshift's workload management (WLM: Redshift's feature for controlling how queries share cluster resources, using queues and rules) has query monitoring rules (conditions checked against a running query, each with an action to take if it trips) whose actions are log, hop (move the query to another queue), abort, or change priority, on metrics such as query_execution_time or query_temp_blocks_to_disk (how much of the query's intermediate working data spilled to disk instead of fitting in memory, a sign of a query starving others); BigQuery isolates compute through slot reservations (a unit of dedicated query-processing capacity, purchased or reserved per project instead of billed per query scanned) assigned per project.

Structural fix: isolation without over-provisioning

The design goal is that one tenant's burst consumes that tenant's share plus genuinely idle capacity, never other tenants' guaranteed share.

1. A query gateway with per-tenant admission control. Every query from the product goes through a thin service that knows the tenant. It enforces a per-tenant concurrency cap (e.g. at most 3 queries running at once for a standard tenant), a per-tenant queue with a maximum length, and fair ordering (the next slot goes to the tenant with the fewest running queries, not the oldest query). This is the piece that actually prevents recurrence; warehouse settings alone cannot see tenants.

2. Cost-based routing. Before running a query, estimate its cost (bytes to scan from table statistics, or the plan's estimate). Cheap interactive queries go to the shared interactive warehouse; heavy ones go to a separate batch warehouse where they can only slow each other.

3. Hard limits on the shared warehouse. Set STATEMENT_TIMEOUT_IN_SECONDS on the interactive warehouse (e.g. 120 seconds) and STATEMENT_QUEUED_TIMEOUT_IN_SECONDS so queries fail fast instead of piling up. A query that needs longer belongs on the batch path.

4. Elastic overflow instead of a bigger baseline. A multi-cluster warehouse, an Enterprise edition (one of Snowflake's paid tiers above the base Standard edition) feature where a warehouse can run as several identical clusters of the same size in parallel instead of just one, adds clusters only while queries are queueing and removes them afterwards. Paying for extra clusters for the three busy hours a day is far cheaper than sizing the baseline for the peak.

5. Dedicated compute only where the revenue justifies it. The largest tenants, or those with contractual performance terms, get their own warehouse. Everyone else shares.

6. Detection so the next one is caught in minutes. Alert when one tenant's share of warehouse elapsed time exceeds, say, 40% over 10 minutes while queue time for other tenants is non-zero. Feed per-tenant usage into billing so the heaviest users see (and pay for) their consumption.

Worked example: why not just give everyone their own warehouse?

Credit rates (a credit is Snowflake's billing unit: a fixed amount of compute-time, billed at your contracted price per credit) for standard Snowflake warehouses double with each size: X-Small 1 credit per hour, Small 2, Medium 4. Suppose 200 tenants.

The shared design costs 3,600 credits a month against 144,000 for always-on dedicated compute. Auto-suspend narrows the gap for dedicated warehouses (idle ones stop billing), but it adds a resume delay on every tenant's first query and still leaves you operating 200 warehouses. The shared-plus-overflow design, with admission control, is the one that protects tenants without over-provisioning.

Trade-offs and pitfalls

• Cancelling queries is visible to the customer. Prefer rerouting a legitimate workload over killing it; kill only what is pathological.
• Concurrency caps can starve a large tenant that legitimately needs parallelism. Make the cap a tier property that sales can raise, and let it borrow idle slots.
• Tags must be set by the platform, not the client. If the tenant ID in QUERY_TAG comes from user input, attribution and throttling can be dodged.
• Timeouts without a batch path just turn a slow query into a failed one; always give heavy work somewhere to go.
• Scaling up during the incident feels decisive but hides the root cause and resets the bill higher; if you do it to protect an SLA (service-level agreement, the contractual promise), schedule the scale-down as part of the incident.

Direct answer

With dozens of services calling each other, the design has three parts. Representation: the tenant and user travel as a signed, short-lived context token minted by the edge, verified locally by every service with a cached public key, never as a plain header. Enforcement: at three points, the edge (authentication, coarse routing), each service at its own boundary (is this caller allowed to invoke me on behalf of this tenant?), and the data layer (every query tenant-scoped, backed by row-level security, a database policy that automatically filters query results down to the caller's own tenant). Speed: decisions are made locally by a policy engine embedded as a library or sidecar, fed by policy bundles and cached relationship data pushed from the centre, so no request path round-trips to a central decision point.

Why the obvious designs fail

• Plain X-Tenant-Id header between services: any service (or attacker inside the network) can set it. That is tenant spoofing by design.
• Service-to-service calls with the service's own powerful credential: the "confused deputy" problem. The reporting service is allowed to read every tenant's data, so a user of tenant A who tricks it into fetching tenant B's report escalates privilege through it. Downstream calls must carry the end user's tenant context, not just the caller's identity.
• A central policy decision point (PDP, the service that answers "is this allowed?") called on every hop: correct but slow and fragile, as the numbers below show.

Representing and propagating identity

The edge authenticates the user, then mints an internal context token (a JWT, JSON Web Token, signed by an internal issuer):

• tenant_id, sub (user), roles, act (the calling service, following the "actor" claim idea from OAuth 2.0 Token Exchange, RFC 8693: a standard for trading one token for a new, narrower-scoped one, so each hop's token proves both the original user and which service is acting for them), aud (the next service), exp of a few minutes.
• Services verify the signature with a public key fetched once from the issuer's JWKS endpoint (JSON Web Key Set, the published signing keys) and cached; verification is a local cryptographic check with no network call.
• For each downstream call, the service either forwards the token (simple) or exchanges it for one with aud set to the next service (stronger: a token stolen from service X cannot be replayed against service Y). A trust zone is a group of services deployed, network-isolated and operated together, so a compromise of one is already assumed to put the others at some risk. Recommendation: forwarding inside a trust zone, exchange when crossing into sensitive services (billing, identity, exports).
• Transport: mTLS (mutual TLS, where both client and server present certificates) via a service mesh (infrastructure, often built from sidecars, that handles secure communication between services), so each service also knows which service is calling. The token says on whose behalf; mTLS says who is asking.

sequenceDiagram
  participant U as User
  participant E as Edge gateway
  participant O as Orders svc
  participant B as Billing svc
  participant D as Billing DB
  U->>E: request + user token
  E->>E: verify, derive tenant, strip client headers
  E->>O: context token (tenant=t_acme, aud=orders), mTLS
  O->>O: verify locally, local policy check
  O->>B: token exchanged (aud=billing), mTLS
  B->>B: verify, check caller=orders allowed for this action
  B->>D: query with session tenant=t_acme, RLS enforced

Where to enforce

Only the resource layer knows who owns row 42, so it cannot be delegated to a sidecar. Only the sidecar layer is uniform across languages, so generic checks belong there. Privilege escalation within a tenant (a viewer acting as admin) is caught at the service and resource layers by role and scope checks; escalation across tenants is caught by all three.

Keeping authorization fast

Distribute decisions, centralize policy:

• Policy as code, distributed as bundles. A policy engine such as OPA (Open Policy Agent, an open-source general-purpose policy engine) runs as a sidecar or library in each service; a control plane (the central system that manages and distributes configuration to every service, distinct from the data plane that carries actual traffic) pushes signed policy bundles (versioned packages containing the actual authorization rules). Evaluation is in memory.
• Relationship data cached locally. Rules like "user is a member of project P in tenant T" need data; publish membership changes as events and keep a local, tenant-partitioned cache in the services that need it, with a short TTL (time-to-live) as a safety net.
• Central service only for the rare hard case (for example deep sharing graphs), with the result cached per request.

Worked arithmetic. Suppose one user request touches 8 services and each consults a central PDP at an assumed 2 ms network cost and 99.9% availability:

So the central PDP alone adds 16 ms to every request and turns a 99.9% dependency into a 99.2% one, which at 1,000 requests per second is about 8 failing requests every second. Local evaluation removes both costs; the price is staleness (a revoked permission lingers until the bundle or cache refreshes).

Handling staleness honestly

• Token lifetime of about 5 minutes bounds how long a removed user keeps access via the token.
• Membership and policy changes pushed as events usually propagate in seconds; the TTL is only the fallback.
• For high-risk actions (exporting all data, changing billing, granting admin), accept the round trip and check the central source synchronously. Fast by default, strict where it matters.

Trade-offs and pitfalls

• Token bloat: putting every permission in the token makes it large and stale. Keep identity and tenant in the token; fetch fine-grained permissions from local policy data.
• Library drift: an in-process library in 40 services means 40 upgrade cycles. Mitigate with the sidecar for generic checks and a thin, versioned library for ownership checks, with CI tests that fail when a new endpoint lacks a tenant check.
• Async paths: queue consumers must receive the tenant context in the message and re-verify it; a job worker running with a global credential is the confused deputy again.
• What would change the design: if authorization rules are highly relational (document sharing across many users), a dedicated relationship-based authorization service (a system built specifically to answer "can user X do Y with object Z" by walking a graph of who has what relationship to what, the approach behind Google's Zanzibar) with local caches becomes worth its complexity.

Direct answer

A bulkhead splits shared capacity into compartments so a flood in one compartment cannot sink the ship. For a multi-tenant API, I would build bulkheads at three layers: at the request layer, per-tenant rate limits, concurrency caps, fair queues, and circuit breakers so a noisy tenant is throttled before it consumes shared capacity; at the resource layer, per-tenant slices of connection pools, worker threads and CPU/memory quotas; at the container/VM layer, cells and shuffle-sharded worker assignment so that even an unbounded failure (a crash loop: a process that crashes, gets restarted, and crashes again immediately, over and over; a memory leak; a poison request: a single malformed or pathological request that reliably crashes whatever worker handles it, so simply retrying it just crashes the next worker too) only reaches a small, known group of tenants. Each layer contains the tenant by limiting what it can take, not by blocking everyone, so healthy tenants keep their share.

Terms used below

• Bulkhead pattern: partitioning resources so that failure or overload in one partition is contained there. Named after the watertight compartments of a ship.
• Noisy neighbour: a tenant whose load degrades others sharing the same resources.
• Concurrency limit: a cap on how many requests from one tenant can be in progress at once.
• Circuit breaker: a switch that stops calling a failing dependency for a while, returning errors fast instead of piling up waiting requests.
• Load shedding: deliberately rejecting some requests (with HTTP 429 or 503) to protect the rest.
• cgroups: the Linux kernel feature that limits CPU and memory per process group; container limits in Kubernetes use it.
• Cell: an independent copy of the service stack serving a subset of tenants.
• Round-robin: serving one item from each of several queues in turn, cycling through them, instead of fully draining one queue before touching the next.
• Shuffle sharding: each tenant is assigned a small random combination of workers, so two tenants rarely share all of them.
• Little's law: in-flight requests = arrival rate x time each request spends in the system.

Layer 1: request layer (first line, cheapest to act)

• Per-tenant rate limit (token bucket) at the gateway, sized to the tenant's plan.
• Per-tenant concurrency cap. Rate limits alone miss slow requests: a tenant sending modest traffic to an endpoint that has become slow can still hold many connections. Capping in-flight requests per tenant bounds its share directly.
• Fair queueing: when the service is busy, requests wait in per-tenant queues served round-robin (see "Terms used below"), so a tenant with 10,000 queued requests does not delay a tenant with 3.
• Per-tenant circuit breakers and timeouts on downstream calls: if tenant X's requests to its custom webhook or its large report query keep failing, trip the breaker for X only.
• Load shedding with priority: under overload, shed the tenant furthest over its fair share first, not requests at random.

How this contains without cutting others off: the noisy tenant gets HTTP 429 once over its limits; everyone else's requests still pass.

Layer 2: resource layer

• Partitioned pools: a database connection pool of 100 split into a shared reserve plus a per-tenant cap (for example no tenant may hold more than 10). A thread pool for expensive operations, with the same per-tenant cap.
• Separate pools for separate work: interactive API calls and bulk exports never share a pool, so one tenant's export does not starve another's login.
• Container quotas: Kubernetes CPU and memory limits per pod (a pod is the smallest deployable unit in Kubernetes, usually one running copy of the service, containing one or more containers; the limits are enforced through cgroups), and namespace resource quotas (a namespace resource quota is a Kubernetes-enforced ceiling on total CPU, memory and object count for everything in one namespace, a way to group a tenant's own set of pods and cap them as a whole) where a tenant has dedicated workloads.
• Database-side limits: statement timeouts (the database automatically cancels a single query if it runs longer than a configured limit) and per-role connection limits, so a runaway query from one tenant is killed rather than holding locks.

Layer 3: container/VM and placement

• Cells: tenants are split across cells, each a full stack. A cell-level failure affects only that cell's tenants.
• Shuffle sharding within a cell: each tenant is served by 2 of 16 worker groups.
• Dedicated node pools or VMs for the few largest tenants, and microVM sandboxes (lightweight VMs such as Firecracker) for any tenant-supplied code, because a container shares the host kernel and a VM does not. A container is really just a restricted, walled-off process (or group of processes) running under the one operating-system kernel (the core of the OS that manages memory, CPU scheduling and hardware) the host is running; every other container on that host, from every tenant, shares that same kernel, so a kernel-level bug or exploit reachable from one tenant's code has a path to the host and to other tenants' containers. A VM boots its own separate kernel on top of virtualised hardware, so the same bug stays contained inside that one VM.

Worked example

Concurrency bulkhead, using Little's law. A noisy tenant sends 2,000 requests per second and each takes 50 ms:

That would take all 100 database connections. With a per-tenant cap of 10 in-flight requests, the most the tenant can complete is:

The other 1,800 requests per second get 429s, and 90 connections remain for everyone else.

Shuffle sharding. With 16 worker groups and 2 per tenant, the number of distinct pairs is:

If a poison request from one tenant crashes both of its worker groups, another tenant is fully cut off only if it has the identical pair: a 1 in 120 chance (about 0.8%). Every other tenant keeps at least one healthy group. Compare plain sharding into 8 fixed pairs, where every tenant on the noisy tenant's pair (1 in 8, 12.5%) is fully down.

flowchart TB
  R[Requests] --> L1[Request layer<br/>rate limit, concurrency cap, fair queue]
  L1 --> L2[Resource layer<br/>pool slices, cgroup limits]
  L2 --> C1[Cell A<br/>shuffle-sharded workers]
  L2 --> C2[Cell B<br/>shuffle-sharded workers]
  C1 --> D1[(Cell A data)]
  C2 --> D2[(Cell B data)]

Trade-offs and pitfalls

• Static partitions waste capacity. If each of 50 tenants gets a hard 2% of a pool, idle slices sit unused. Use a cap plus a shared reserve (work-conserving): tenants may borrow idle capacity but can never exceed their cap.
• Caps set too low throttle legitimate growth. Make them per plan, observable (tenants can see their usage), and adjustable without a deploy.
• Retries defeat bulkheads. A throttled tenant's clients retrying instantly multiply load. Return the Retry-After HTTP header (it tells the client exactly how long to wait before trying again), and have your SDKs use exponential backoff with jitter (each retry waits roughly twice as long as the previous one, with a bit of randomness mixed in so retries from many clients spread out instead of all landing at the same instant).
• Pitfall: bulkheads only at the edge. A gateway rate limit does not stop one expensive request from locking a shared table. The inner layers exist because some overload is shaped by cost, not count.
• What would change the design: very large or regulated tenants justify moving from shared cells to a dedicated cell, which is the strongest bulkhead but the most expensive.