Multi-Tenancy and Isolation Questions

Direct answer

I would put a small, versioned tenant placement map (tenant to home region, primary, and the replicas it is allowed to use) at the center, cache it in every query coordinator, and make every routing decision a pure function of three inputs: the tenant's placement entry, the statement type (read or write), and the client's session consistency token. Writes always go to the tenant's current primary. Reads go to an in-region replica only if that replica has caught up to the session token, otherwise to the primary. Residency is enforced by construction, because the map never lists a replica outside the tenant's permitted regions, and failover can only promote a replica from that same list.

Terms used below

• Coordinator (or gateway): the node that receives a SQL statement, decides where it runs, and returns the result. In a distributed SQL engine every node can usually act as one.
• Primary / leaseholder: the one replica allowed to accept writes for a piece of data. A lease is a time-limited right to act as primary; when it expires, another replica may take over.
• Epoch (or generation number): a counter bumped every time the primary for a tenant changes. Anything stamped with an older epoch is known to be stale.
• Read-your-writes (RYW): after you commit a write, your own later reads must see it, even if they land on a different node.
• Data residency: a contractual or legal rule that a tenant's data (including copies) stays in named jurisdictions, for example "EU only".

1. Routing metadata

One row per tenant, stored in a strongly consistent system table (replicated with a consensus protocol such as Raft, where a majority must agree before a change is visible):

For very large tenants that are sharded internally, the entry points to a list of key ranges, each with its own primary and epoch. The shape is the same, one level down. If you run CockroachDB, the closest built-in is a REGIONAL BY ROW table, where each row carries a region column and the engine homes the row there; the tenant map above is what you build when you want that decision per tenant rather than per row.

2. Coordinator cache design

• Full replica of the map in memory on every coordinator. 50,000 tenants at about 200 bytes per entry is 50,000 x 200 = 10,000,000 bytes, about 10 MB. That fits comfortably, so there is no reason for a remote lookup on the hot path.
• Kept fresh by a change feed (a stream the metadata store pushes to subscribers whenever a row changes, instead of subscribers polling it), not a TTL (time-to-live). The metadata store (the strongly consistent system table from section 1, holding the placement map) pushes every change with its epoch; coordinators apply it. A TTL alone either serves stale routes for the whole TTL or hammers the store.
• Self-correcting on staleness. Every request carries the epoch the coordinator used. The receiving node compares it to its own view. If the node is no longer primary for that tenant it rejects with "not primary, current epoch is 42, primary is X". The coordinator updates its cache from that hint and retries once. A stale cache therefore costs one extra round trip, never a wrong write.
• Fail closed on a cache miss for an unknown tenant: fetch from the metadata store, and if that is unreachable, return an error. Never guess a default region, because a guessed default is exactly how EU data ends up in a US replica.

sequenceDiagram
    participant C as Client
    participant G as Coordinator
    participant R as EU replica
    participant P as EU primary
    C->>G: SELECT ... (tenant t-4812, token ts=1005)
    G->>G: map lookup: epoch 41, allowed EU only
    G->>R: read if applied_ts >= 1005
    R-->>G: applied_ts=990, too old
    G->>P: read at primary (epoch 41)
    P-->>G: rows, commit_ts=1005
    G-->>C: rows + token ts=1005

3. Read-your-writes

Every commit returns a session token: the commit timestamp (or log position) of that write. The client driver or the application's session layer sends the highest token it holds with every later statement.

A replica may serve the read only if it has applied the log (replayed the primary's stream of committed changes, the same replication log every replica reads from) up to that token. If it has not, the coordinator can wait briefly for it to catch up (I would cap the wait at about 20 ms) and otherwise send the read to the primary. Statements with no token (a brand-new session, a batch job) can read any in-region replica with bounded staleness (a known limit on how far behind the data might be, rather than unlimited staleness).

The subtle case is the token surviving a failover. Tokens must be timestamps on a clock that keeps increasing across primary changes (hybrid logical clocks, which combine wall-clock time with a counter, do this), or log positions (an offset into that replication log: essentially "how many changes has this replica applied so far") that include the epoch. Otherwise a new primary could hand out a position that compares as "older" than a write the client already saw.

4. Routing under failover

1. The old primary's lease expires (say 9 s lease). Replicas in allowed_regions elect a new primary. Only replicas listed for the tenant are eligible, so residency cannot be broken by failover.
2. The new primary bumps the epoch to 42 and writes the new map entry to the metadata store. The change feed delivers it to coordinators.
3. A coordinator that has not yet seen epoch 42 sends a write to the old primary. The old primary (if alive) sees its lease is gone and rejects it with the hint. If the old primary is partitioned away instead (cut off from the rest of the cluster by a network failure, though still running), the lease rule protects you: the new lease cannot be granted until the old one has expired, and the old primary stops accepting writes when its own lease runs out. This relies on clock drift between nodes being bounded: each node judges its own lease expiry against its own clock, and if the old primary's clock could run arbitrarily slower than the rest of the cluster's, it could still believe its lease was valid long after everyone else had already handed the lease to a new primary, giving you two nodes simultaneously believing they are primary, a split brain. Bounding clock drift is what rules that out. The epoch also works as a fencing token: storage rejects any write stamped with an epoch lower than the highest it has seen.
4. Writes are unavailable for roughly lease expiry plus election plus propagation. Reads that do not need the latest data can keep going on in-region replicas throughout.

When the whole allowed region set is down, the correct behavior for a residency-bound tenant is to fail its requests, not to fail over to another jurisdiction. That is a product decision to make explicitly with the customer: offer two EU regions if they need regional failover.

Worked example

Tenant t-4812 is EU-only, primary in eu-west, replicas in eu-west and eu-central, epoch 41.

1. A user updates an invoice. It commits at timestamp 1005; the client now holds token 1005.
2. 30 ms later the same user loads the invoice list. The coordinator in eu-central picks the local replica, which has applied up to 990. 990 < 1005, so it waits up to 20 ms; the replica reaches only 998, so the read goes to the eu-west primary and returns the updated invoice.
3. eu-west loses power. After lease expiry the eu-central replica becomes primary at epoch 42. A coordinator still on epoch 41 sends a write to eu-west, gets a connection error, refreshes the map, sees epoch 42, and retries against eu-central.
4. A US coordinator receives a request for t-4812. The map says allowed_regions = EU, so it forwards the statement to an EU coordinator. It never reads a US copy, because there is no US copy in the map to read.

Trade-offs and pitfalls

• Caches, logs and backups are also copies. A coordinator that caches result sets, a query log with literal values, or a backup job writing to a default bucket in another region all break residency while the routing table looks perfect. Check each against allowed_regions.
• Waiting for a replica versus going to the primary trades latency for primary load. If most reads carry fresh tokens (chatty user sessions), routing them to the primary is honest; do not pretend replicas are taking that load.
• Cross-tenant queries (an internal report over all tenants) cannot follow per-tenant residency; run them over a separately governed, residency-aware aggregate store rather than letting the router fan out globally.
• Pitfall: trusting a routing cache TTL as the correctness mechanism. Correctness must come from the epoch check at the storage node; the cache is only a performance optimization.
• Pitfall: letting a tenant move between regions by updating primary alone. A move is copy, catch up, flip with a new epoch, then delete the old copies, and residency must hold at every step (both source and destination inside allowed_regions).

Direct answer

I would separate the system into three planes that scale and fail independently: file bytes in object storage under a per-tenant prefix encrypted with a per-tenant key, metadata (names, owners, tags, timestamps) in a tenant-partitioned database plus a search index, and archival handled by lifecycle rules that move cold bytes to an archive storage class while their metadata stays searchable. Isolation is enforced in layers: the tenant comes from the signed token, never from the request, every data path is scoped to that tenant, and each tenant's data is encrypted with its own key so that even a routing bug returns unreadable bytes. For the client I would hand over a one-page summary that states the decision, the cost logic, and the risks we accepted.

Terms used below

• Object storage: a store for whole files ("objects") addressed by a key, such as Amazon S3 (Simple Storage Service). Cheap, very durable, and not a file system.
• Prefix: the leading part of an object key, e.g. tenants/t-042/. Access policies and lifecycle rules can target a prefix.
• KMS (key management service): a managed service that holds encryption keys; with one key per tenant, revoking a tenant's key makes its data unreadable (crypto-shredding).
• Search index: a separate store (for example OpenSearch, an open-source search engine) optimised for "find files where tag = invoice and modified after March".
• Lifecycle rule: a storage policy such as "after 180 days without access, move to archive class".
• CDN (content delivery network): edge servers that cache downloads close to users.
• Signed URL: a time-limited link that grants access to one object without exposing credentials.

Architecture

flowchart LR
  C[Client app] --> API[File API<br/>tenant from token]
  API --> MD[(Metadata DB<br/>partitioned by tenant)]
  API --> OS[(Object storage<br/>prefix per tenant)]
  MD -->|change stream| IDX[(Search index)]
  API --> IDX
  OS -->|lifecycle| AR[(Archive class)]
  C -->|signed URL| CDN[CDN] --> OS

• Upload: client asks the API for an upload URL; the API writes a metadata row in pending state and returns a signed URL for tenants/{tenant_id}/{file_id}. On completion the row flips to active. The client never chooses the key, so it cannot write into another tenant's prefix.
• Download: API checks permission in metadata, returns a short-lived signed URL; bytes flow from storage or CDN, not through the API.
• Search: API queries the index with the tenant filter added server-side.

Data partitioning

• Object storage: one bucket per region with a prefix per tenant is my default; a bucket per tenant for tenants that need their own bucket policy (a resource-based permission document attached directly to a bucket, saying who may do what to it) or region. AWS documents at least 3,500 writes and 5,500 reads per second per partitioned prefix (S3 automatically splits its internal request handling by the leading part of the key, so distinct prefixes each get this throughput independently rather than sharing one pool), and there is no limit on the number of prefixes, so per-tenant prefixes also spread request load: a busy tenant can be further split with a hash segment (tenants/t-042/7f/...).
• Metadata database: primary key (tenant_id, file_id), sharded by tenant_id (split across multiple database instances, each called a shard, by tenant), so a tenant's queries touch one shard and a tenant can be moved as a unit.
• Search index: a shared index with routing by tenant_id for small tenants, meaning the index deliberately sends all of a tenant's documents to the same shard rather than scattering them (all of a tenant's documents land on one shard, and the query only touches that shard), and an index per tenant for large tenants. An index per tenant for everyone would create thousands of mostly empty shards, and each shard carries fixed memory overhead.

Caching strategy

Thumbnails and previews are generated once and stored as objects next to the original, which is cheaper than recomputing.

Security and isolation controls

1. Identity: tenant derived from the authenticated token; every API and data call carries it.
2. Authorization: permission check in the metadata layer before any URL is signed.
3. Storage policy: the service role for a request can only sign URLs under tenants/{its tenant}/.
4. Encryption: per-tenant KMS key for objects; the database and index encrypted at rest; TLS (Transport Layer Security, the standard protocol that encrypts data moving over the network) in transit.
5. Row-level security (RLS: the database itself adds the tenant filter to every query) in the metadata database as a second guard behind the application filter.
6. Audit: every read, write, share and delete logged with tenant, user and object; logs exportable to the client.
7. Tests: automated cross-tenant probes on every release (tenant A's token requesting tenant B's file IDs must get 404, not 403, so existence is not leaked).

Worked example: sizing and archival

Assume (illustrative) the client stores 200 TB with an average file size of 2 MB.

• Files: 200 TB / 2 MB = 200,000,000 MB / 2 MB = 100 million files.
• Metadata at about 1 KB per file: 100 million x 1 KB = 100 GB. That fits comfortably in a sharded database and a search cluster; the bytes do not need to be touched for search.
• Archival choice: S3 Glacier Deep Archive has a 180-day minimum storage duration (delete or move an object out before 180 days and you are still billed for the full 180 days, as if it had stayed) and restores typically within 12 hours (standard) or 48 hours (bulk); S3 Glacier Flexible Retrieval has a 90-day minimum (same rule, shorter window) and offers expedited restores in typically 1 to 5 minutes. If the client's contract says "archived files available within one business day", Deep Archive fits; if legal holds (a hold that blocks deletion of specific files for litigation or compliance, overriding the normal retention schedule) require same-hour access, Flexible Retrieval fits better.
• The small-file trap: each object in the two archival Glacier classes carries about 40 KB of extra metadata charged per object. For a 100 KB file that is a 40% overhead. So the lifecycle rule archives only objects above a size threshold (for example 1 MB), and small cold files are bundled into larger archive objects with an index kept in metadata.
• Archive storage costs over ten times less per GB-month than the standard class at current US East list prices, so moving, say, 60% of the 200 TB (120 TB) to archive removes most of the storage bill. Check current regional pricing before quoting a figure to the client.

One-page executive summary for the client

Decision: Your files are stored in a shared, highly durable object store, but logically and cryptographically separated: a dedicated storage area and a dedicated encryption key for your organisation. File details (names, tags, owners) live in a separate searchable catalogue.

Why: This gives you dedicated-level protection at shared-platform cost. Search stays fast as you grow because it never scans files, only the catalogue.

Isolation: Your identity decides which data can be reached; the client application cannot request another customer's data. Even in the event of a software fault, other customers' data is encrypted with keys your account cannot use. You receive a full access log.

Cost control: Files untouched for 180 days move automatically to archive storage, over ten times cheaper per GB. They remain searchable and, under our default archive tier, restorable within 12 hours (this assumes the standard tier fits your contract; if you need faster access, see Next steps, the faster tier costs more per GB).

Accepted trade-offs: Archived files are not instant; restores take hours. A dedicated bucket or region is available if your compliance team requires it, at additional cost.

Next steps: confirm archive retrieval time requirement, data residency region, and key ownership (ours or yours).

Trade-offs and pitfalls

• Prefix-per-tenant vs bucket-per-tenant: prefix is cheaper to operate and scales to many tenants; bucket-per-tenant gives a cleaner policy boundary and per-tenant settings. I default to prefix and upgrade specific tenants.
• Search index drift: the index is updated asynchronously from the database's change stream, so a just-uploaded file may take seconds to appear. Tell the client; do not promise instant search.
• Pitfall: archiving by age alone, meaning by time since last access as the lifecycle rule above is defined, not by an object's creation date. Either way, age alone says nothing about legal holds; the lifecycle rule must skip objects with a hold flag.
• Pitfall: caching permission decisions too long; a revoked user keeps access until the cache expires. Keep that TTL short and invalidate on revocation.

Direct answer

For a multi-tenant system across regions, combine three layers: continuous point-in-time recovery in the primary region (so you can rewind to any second in the last N days), cross-region copies of backups (so losing a whole region does not lose your data), and a per-tenant restore path (so one customer's mistake does not force a restore for everyone). Then map each tenant tier's RPO (recovery point objective: how much recent data you can afford to lose) and RTO (recovery time objective: how long you can be down) to the cheapest mechanism that meets it, instead of giving every tenant the most expensive protection.

The building blocks

A replica is not a backup. If someone runs DELETE FROM invoices WHERE tenant_id = 't_acme' by mistake, the replica deletes the rows a second later. Only a backup taken before the mistake helps.

Two very different recoveries

It helps to separate the two scenarios early, because they use different mechanisms and different people:

In a pooled database (all tenants share tables, separated by a tenant_id column), the database's own PITR restores everyone to the chosen time, so a single-tenant restore always goes via a side instance. In a silo model (a database per tenant), the native PITR restores just that tenant, which is one of the practical reasons premium tenants get silos.

Retention policies by tenant

Retention is a contract term and a cost driver, so make it a per-tier setting:

• Free / basic: 7 days of PITR, no long-term copies.
• Business: 35 days of PITR, cross-region copy of daily snapshots.
• Enterprise / regulated: 35 days of PITR plus monthly snapshots kept 12 months (or the contractual period), cross-region.

In a pooled store you cannot keep one tenant's data longer than another's inside a single snapshot, so longer retention for specific tenants is implemented with per-tenant logical exports on their own schedule, or by placing those tenants on dedicated storage.

Worked example: what retention costs

Assume a pooled cluster holding 20,000 GB with 2% of blocks changing per day (400 GB/day), and incremental snapshots taken daily. Stored bytes after keeping N daily snapshots are roughly the first full copy plus one day of changes per additional day:

• 7 days: $20{,}000 + 400 \times 6 = 22{,}400$ GB
• 35 days: $20{,}000 + 400 \times 34 = 33{,}600$ GB
• 365 days of dailies: $20{,}000 + 400 \times 364 = 165{,}600$ GB

Going from 35 days to a year of daily backups multiplies stored backup bytes by about 4.9 (165,600 / 33,600), and a cross-region copy doubles whichever figure you pick. That is why long retention is sold as a tier and implemented as sparse monthly snapshots (12 extra points in time) rather than 365 dailies. (This is a lower bound: blocks changed repeatedly across days and compression both move the real number, so measure your own change rate.)

Mapping RTO/RPO to tenant SLAs

Work from the SLA (service-level agreement) to the mechanism, cheapest first:

1. Write down each tier's contractual RPO and RTO.
2. For each, pick the cheapest mechanism that meets both. An RPO of 24 hours is met by daily snapshots; an RPO of 5 minutes needs continuous log shipping (continuously copying the write-ahead log to another location as it is written, instead of waiting for the next full snapshot); an RPO under a minute in a regional disaster needs a live cross-region replica.
3. Check the RTO honestly: the restore time of a 20 TB database from snapshot is hours, not minutes, so a short RTO means a standby that already exists, not a backup you restore.
4. Group tenants with the same requirement onto the same infrastructure, since replication and backup are configured per database cluster, not per row.
5. Test restores on a schedule; an untested backup has an unknown RTO.

Trade-offs and pitfalls

• Pitfall: equating replication with backup. It protects against hardware and region loss, not against bad writes.
• Pitfall: promising per-tenant restore from a pooled database without a merge plan. Restoring rows as of Tuesday into a table that has had Wednesday's changes needs clear rules (restore to a separate "recovered" area and let the customer choose, or overwrite only rows unchanged since).
• Pitfall: backups in the same account and region as production. Ransomware or a compromised admin credential deletes both. Keep copies in a separate account, ideally with a retention lock (a setting that blocks deletion, even by an account admin, until a set date passes).
• Cost lever: move backups older than a month to colder storage classes (cheaper storage tiers built for data you rarely read, with slower retrieval) and keep only sparse long-term points.

Direct answer

This is a noisy neighbour problem (one tenant's heavy workload degrading others on shared resources) on warehouse compute. Data can stay pooled; compute must not. I would put a query routing layer in front of the warehouse that knows each tenant's tier and each query's expected cost, send the handful of high-revenue tenants to dedicated compute (their own virtual warehouse in Snowflake, their own slot reservation in BigQuery), put the long tail on a shared, elastic pool with hard per-query limits, and send heavy batch jobs to a separate batch pool. Row access policies (or per-tenant secure views) keep data isolated on the shared tables, and query tags plus metering views attribute every credit or slot-hour to a tenant so cost stays visible as tenants grow.

Why it happens

In both warehouses, storage is separate from compute. Queries that run on the same compute cluster (a Snowflake virtual warehouse) or draw from the same slot pool (BigQuery's unit of compute) compete for it. When one tenant launches a large scan-and-join, it takes the capacity and other tenants' dashboards queue. Isolation therefore means separating compute pools and routing to them, not copying data.

Architecture

flowchart LR
  A[Tenant apps and BI tools] --> R[Query router]
  R -->|premium tenants| D1[Dedicated compute: tenant A]
  R -->|premium tenants| D2[Dedicated compute: tenant B]
  R -->|long tail, interactive| S[Shared elastic pool]
  R -->|heavy or batch jobs| H[Batch pool]
  D1 --> T[(Shared tables, row access policy on tenant_id)]
  D2 --> T
  S --> T
  H --> T

The query router

• Authenticates the caller, resolves the tenant, and sets a session tenant context used by row access policies.
• Classifies each query. In BigQuery, a dry run returns the bytes a query would scan without running it; in Snowflake, the router uses the query's source (scheduled export vs dashboard) and history of similar query shapes. Queries above a threshold go to the batch pool.
• Tags every query with the tenant (Snowflake's QUERY_TAG session parameter; BigQuery job labels) for cost attribution.

Workload isolation, per warehouse

The two rows that matter most are dedicated compute (hard isolation for premium tenants) and the elastic shared pool (soft isolation with limits, for everyone else); the rest of this table is the vendor-specific settings that implement those two ideas, useful to know but secondary to the pattern itself.

Recommendation for BigQuery: premium tenants get a baseline sized to their normal peak, so they never depend on idle slots for predictable latency; the shared pool can borrow idle premium capacity when premium tenants are quiet, which recovers much of the cost of dedicating it.

Data isolation on shared tables

Two framings, both valid; I would use the first and keep the second for tenants who query through their own tools:

• Row-level security plus resource governance: one set of shared tables with tenant_id; a row access policy (both Snowflake and BigQuery support row-level policies) filters by the session's tenant, and the compute routing above governs resources. One copy of data, one schema to migrate.
• Virtual schema per tenant over the shared warehouse: each tenant sees a schema of secure views (views whose definition and underlying data are hidden from the consumer) filtered to its tenant_id. Tenants who connect their own BI tool (business intelligence tool, such as a dashboard product that queries the warehouse directly) get a clean, tenant-only namespace, and grants are per schema, which auditors find easy to read. Cost: view sprawl across thousands of tenants, handled by generating the views from code.

Clustering here means physically co-locating and ordering the stored rows on disk, the data-warehouse sense of the word, not the machine-learning technique of grouping similar data points. Clustering (Snowflake) or partitioning and clustering (BigQuery) the big tables on tenant_id (plus a date column) means a tenant's query prunes (skips reading the parts of the table that cannot contain matching rows) to its own data instead of scanning everyone's, which is a performance and cost win on its own.

Worked example: cost of dedicating compute

Snowflake bills warehouses in credits per hour of running time, doubling with each size step (X-Small 1, Small 2, Medium 4, Large 8 per hour). Suppose 5 premium tenants each need a Medium warehouse for 10 operating hours a day, every calendar day rather than just weekdays (a global SaaS has tenants active across time zones and weekends too), with auto-suspend (the warehouse pauses itself, and stops billing, after a period with no queries) off-hours:

Compare one Large warehouse running 24 hours a day for everyone, premium and long-tail tenants together: $8 \times 24 \times 30 = 5{,}760$ credits per month. The 6,000 credits for dedicating the five premium tenants is the same order of magnitude as that 5,760-credit baseline, but it is not a full before-and-after comparison: after moving the five premium tenants onto their own warehouses, the long tail still needs its own shared warehouse, so total spend becomes 6,000 plus whatever that right-sized shared warehouse costs, not 6,000 in place of 5,760. Whether total spend ends up close to flat or roughly doubles the 5,760 baseline depends on how much of that original Large warehouse's load actually came from the five premium tenants, which is a figure to size the long-tail warehouse against, not to assume away. The long tail then runs on a multi-cluster Small or Medium warehouse sized from its own measured concurrency and scaling out under load. The deciding figure for whether dedicating compute is worth it at all is each premium tenant's revenue against its 1,200 credits a month; attribution makes that visible.

Keeping cost under control as tenants grow

• Attribution first: join query tags to query history and warehouse metering views (built-in views reporting exactly how much compute each query or warehouse used; Snowflake's ACCOUNT_USAGE views, BigQuery's INFORMATION_SCHEMA.JOBS) to allocate compute per tenant. For a shared warehouse, allocate its credits by each tenant's share of execution time.
• Tier by measured cost, not just revenue: a tenant whose attributed compute exceeds its plan's allowance is moved to a dedicated pool and billed for it, or throttled.
• Result caching and pre-aggregation (materialized views on common dashboard queries) cut repeated heavy scans.
• Review quarterly: which dedicated pools sit idle (merge them back), which shared tenants dominate (promote them).

Trade-offs and pitfalls

• Pitfall: bigger shared warehouse as the fix. Scaling up helps briefly; the next heavy job fills it again. Separation plus limits is what makes latency predictable.
• Pitfall: premium reservations that borrow idle slots. Performance becomes unpredictable when the borrowed capacity disappears. Size the baseline for the real peak.
• Pitfall: row policies that depend on a session variable the router forgets to set. Test that a query with no tenant context returns zero rows, not all rows.
• What would change the design: tenants demanding contractual isolation (regulated industries) move from dedicated compute to a dedicated account or project with their own data copy, at a higher operational cost.

Direct answer

I would design a hybrid tenant model with a single codebase: every tenant starts in a shared, logically separated deployment (the pool), and a tenant graduates to a dedicated single-tenant instance only when a clear trigger fires: compliance, contractual SLA, sustained noisy-neighbour impact, or data-residency. The key architectural move is a control plane (the management layer that knows every tenant: plan, routing, configuration, feature flags, SLA) separate from the data plane (the services that actually serve requests). Pooled and dedicated tenants run the same code, read their differences from the control plane, and are upgraded by the same pipeline, which is what keeps the hybrid model from turning into N forks of the product.

The three models

• Single-tenant (per-customer): each customer gets its own deployment and database. Also called the silo model.
• Shared multi-tenant (logical separation): one deployment serves everyone; data is separated by a tenant_id enforced in the application and ideally by database row-level security (RLS). Also called the pool model.
• Hybrid: pool by default, silo for the tenants that need it. Sometimes called a bridge model when individual layers are mixed (for example shared application servers but a per-tenant database).

Control plane: routing, configuration and policy per tenant

flowchart LR
  CP[Control plane: tenant registry, config, flags, SLA, routing] --> R[Edge router]
  CP --> P[Pool deployment]
  CP --> S[Dedicated instance]
  R -->|tenant in pool| P
  R -->|tenant graduated| S

• Per-tenant routing rules. The router maps each tenant (by subdomain, API key or token claim, a field embedded in an authentication token such as a JWT that identifies which tenant the caller belongs to) to its deployment. Graduation or migration is a routing change here, never a DNS rework by hand or a code change.
• Configuration and policy management. One versioned record per tenant: plan, limits (rate, storage, seats), data region, retention policy, authentication settings (for example single sign-on required), allowed integrations. Services read it through a cached client; every change is audited, so "who raised this tenant's rate limit" has an answer.
• Per-tenant feature flags. A feature flag is a runtime switch that turns code paths on or off without a deploy. Targeted per tenant, flags do two different jobs, and you should keep them apart:
  • Release flags (temporary): roll a new feature to 5% of tenants, then all, then delete the flag.
  • Entitlement flags (permanent): which features a plan includes. These belong in the tenant's plan configuration, not in a pile of never-deleted release flags.
• Customization without forks. Custom fields, workflows, branding and integrations are data in the tenant's configuration and extension points in the code (specific places deliberately built to let behavior be customized through configuration or plugins, without editing the core code itself). The rule: a customer's requirement that needs a code branch becomes a generic, configurable feature or it is declined.

Custom per-tenant SLAs

An SLA (service-level agreement) is the contractual promise, such as 99.95% monthly availability with credits if missed; an SLO (service-level objective) is the internal target you run to. Custom SLAs affect architecture:

• Store each tenant's SLA tier in the control plane and measure availability per tenant, not just fleet-wide (a fleet at 99.99% can still fail one tenant's 99.95%).
• A 99.95% monthly SLA allows 0.05% of a 30-day month: 0.0005 x 43,200 minutes = 21.6 minutes of downtime. Our pooled maintenance windows alone might exceed that, which is itself a reason to graduate such a tenant or schedule zero-downtime upgrades.
• Higher tiers get priority in incident response, dedicated capacity, or a cross-region standby.

When a tenant graduates to a dedicated instance

Four triggers, each measurable:

1. Compliance: the contract or regulator requires a separate data store, customer-managed encryption keys, or a dedicated audit scope.
2. Cost and contract: they will pay the dedicated minimum, or their load is large enough that a silo is cheaper than their share of the pool.
3. Noisy-neighbour prevention: the tenant's sustained load makes them a risk to others. Example trigger: a tenant uses more than 15% of a pool shard's (one of several separate database instances the shared pool is split across, so no single database serves every pooled tenant) database CPU for 7 consecutive days while other tenants' p99 latency (99th-percentile latency) on that shard rises. Throttling them further would break their SLA; isolating them protects everyone.
4. SLA or residency: a custom SLA or a data-location requirement the pool cannot meet.

The graduation itself: provision the dedicated instance from templates, copy the tenant's data (a snapshot, a copy taken at one instant, plus change-data-capture catch-up: streaming just the rows that changed since that snapshot, by reading the database's own change log, rather than re-copying everything), briefly freeze writes, flip the routing rule. Reverse graduation (back to the pool) must also be supported for contract downgrades.

Upgrade paths

• Ring deployment: ring 0 is internal tenants, ring 1 a small share of pooled tenants, ring 2 the rest of the pool, ring 3 dedicated instances in each customer's maintenance window.
• Bounded version skew (how many releases behind the pool a dedicated instance is allowed to lag): dedicated instances may lag the pool by at most one release; the control plane refuses to let them fall further behind. Without that rule, silos become forks.
• Backward-compatible schema changes (add first, remove later) so pool and silos can run adjacent versions during a rollout.

Per-tenant billing

• Pooled tenants: meter usage per tenant (requests, storage bytes, seats, compute-seconds for heavy jobs) and bill by plan plus overage. Shared infrastructure is allocated by a documented rule so gross margin per tenant can be computed.
• Dedicated tenants: bill the deployment's full cost, with a floor, since its idle capacity cannot be resold. Tag every resource of the instance with the tenant ID so the cloud bill maps directly.

Worked example

A B2B project-management SaaS with 3,000 tenants: 2,960 pooled; 25 dedicated because of regulated-industry contracts; 15 dedicated after tripping the noisy-neighbour trigger. A pooled tenant asks for "SSO required, custom approval workflow, EU data residency". SSO is a configuration setting; the approval workflow ships as a configurable workflow feature behind a release flag; EU residency means routing them to the EU pool deployment, not a dedicated instance. None of that forks the code.

Trade-offs and pitfalls

• Customization by fork is the classic single-tenant trap: every silo diverges and upgrades stop.
• Flag debt: release flags never removed turn into an untested combinatorial space.
• Dedicated instances outside the pipeline drift and become security liabilities.
• What would change my recommendation: a product with a few hundred very large enterprise customers and heavy regulation could default to single-tenant, but only with the same control plane and fully automated upgrades.