Multi-Tenancy and Isolation Questions

Direct answer

I would split the cluster into three service classes with different promises: real-time inference gets a reserved, non-preemptible slice sized from its peak load plus headroom; interactive experiments get small, time-boxed, guaranteed quotas per team; batch training gets everything else, on a preemptible basis, with mandatory checkpointing. Teams get quotas they are guaranteed, can borrow each other's idle GPUs, and lose borrowed GPUs first when the owner needs them back. Batch training is deliberately the shock absorber: when inference bursts, batch is what gets preempted, which is why batch jobs must be able to resume.

The forces I am balancing

• Cost: GPUs are the most expensive thing in the cluster, so idle GPUs are the main waste. Anything reserved and unused is money lost.
• Latency guarantees for inference: a user is waiting on every inference request. Its SLO (service-level objective, e.g. "99% of requests answer within 300 ms") cannot depend on whether someone launched a big training run.
• Fairness across teams: no team should be able to take the whole cluster just by submitting first.
• Bursty demand: inference traffic spikes (a product launch, a daily peak), and experiments pile up before deadlines.

These pull against each other: the cheapest cluster is 100% packed, the safest inference is 100% reserved. The policy below reserves only for the class that cannot wait and lets everything else share.

The policy, class by class

Priority classes. On Kubernetes this is a PriorityClass per tier. The inference class has the highest value; batch has the lowest. The interactive class can set preemptionPolicy: Never so it waits for capacity instead of evicting batch jobs, which keeps a single notebook from killing a multi-day training run; only inference is allowed to preempt.

Quotas and borrowing. Plain Kubernetes ResourceQuota caps a namespace (a Kubernetes boundary that groups one team's resources and limits) but cannot lend idle GPUs. A queueing layer such as Kueue (a Kubernetes-native job queueing controller) adds that: each team gets a ClusterQueue with a nominalQuota of GPUs (its guaranteed share); queues in the same cohort can borrow each other's unused quota up to a borrowingLimit; a lendingLimit stops a team lending away the part it needs to get back instantly. Preemption is set so that reclaimWithinCohort: LowerPriority lets an owner reclaim lent GPUs from lower-priority borrowers. Borrowed capacity is, by design, the first to go.

For example: team A's queue has a nominalQuota of 10 GPUs and is using 4, so 6 sit idle. Team B's queue also has a nominalQuota of 10 but needs 14, meaning it must borrow 4 GPUs beyond its own share. Because both queues are in the same cohort, and Kueue only admits a request when it fits both the requester's own borrowingLimit and the cohort's actual unused quota, B's borrow of 4 clears its borrowingLimit (say 8) and fits inside A's 6 idle GPUs, so B is admitted at 14. If B instead needed 20, it could still borrow at most 6, the smaller of its borrowingLimit and what A actually has idle, landing at 16; the remaining 4 queues until more capacity is idle, because borrowingLimit caps what a queue may ask for, it does not conjure GPUs no one has free. When A later submits work that needs its full 10 back, reclaimWithinCohort: LowerPriority preempts B's lowest-priority borrowed jobs first, returning A's GPUs. A lendingLimit on A's queue (say 8) would have capped how much of A's own quota could ever be lent out in the first place, protecting a minimum A always keeps for itself; it does not bind in this example, because A's actual idle capacity (6) never reaches its 8-GPU lendingLimit. Day to day, nominalQuota and cohort membership are what decide behaviour; borrowingLimit, lendingLimit and reclaimWithinCohort are the safety valves you tune once interference shows up.

Preemption rules. A preempted batch job must lose little work, so the policy requires checkpointing (saving model state to storage) at a fixed interval, and the scheduler gives a grace period (e.g. 2 minutes) for a final checkpoint before the pod (Kubernetes's smallest deployable unit: one running container or group of containers, here one job's worker) is killed. Jobs that cannot checkpoint are admitted only into a team's guaranteed quota, never onto borrowed GPUs.

Fairness. Within the batch tier I would use a fair-share ordering (the team furthest below its guaranteed share goes next) rather than first-come-first-served. Because jobs ask for GPUs, CPU and memory together, dominant resource fairness (compare teams by whichever resource they use the largest fraction of) is the right generalisation when some teams are CPU-heavy and others GPU-heavy.

Worked example: 64 GPUs, four teams

Assume 8 nodes with 8 GPUs each (64 GPUs). Inference peaks at 2,400 requests per second, and load testing shows one GPU serves 150 requests per second while meeting the p99 (99th-percentile) latency target.

Four teams each get a nominal quota of 44 / 4 = 11 GPUs, all in one cohort so an idle team's GPUs are borrowable. Interactive work is carved out of each team's 11 (for example 2 GPUs each, partitioned into MIG slices).

Bursty inference beyond the reserve. The 25% headroom covers normal bursts. For a spike beyond 20 GPUs, the inference autoscaler adds replicas; those replicas have the top priority class, so the scheduler preempts batch pods (borrowed ones first) to make room. The cost of that preemption is bounded by the checkpoint interval. If a job saves a final checkpoint in its grace period, it loses almost nothing; if it cannot (the save takes longer than the grace period, or the job ignores the signal), it falls back to its last periodic checkpoint. With checkpoints every 30 minutes and preemptions landing at a random point in the interval, that fallback costs on average 30 / 2 = 15 minutes of the job's GPUs.

Cost trade-off made explicit. The 4 headroom GPUs (20 minus 16) are idle most of the day. The alternative is letting batch run on them and preempting on every spike. I would keep the headroom because inference scale-up must beat a pod start plus model load, which takes minutes for a large model, while a latency SLO is judged in seconds. What would flip it: a model small enough to load in seconds, or a traffic pattern predictable enough to pre-scale on a schedule.

Noisy-neighbour interference: measure, then mitigate

A noisy neighbour is a workload that degrades others through a shared resource. On GPUs, the shared resource is not only the GPU itself.

How to measure it. Export per-GPU metrics with NVIDIA DCGM (Data Center GPU Manager) via its Prometheus exporter: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_ACTIVE (SM: streaming multiprocessor, the GPU's core compute unit; the metric is the fraction of time they are busy) and DCGM_FI_DEV_FB_USED (GPU memory used), labelled by pod and team. Then compare each inference replica's p99 latency against its own baseline and correlate the degradations with what was co-scheduled on the node at the time. The signal to alert on is "inference p99 rose while a non-inference workload was placed on the same node or GPU", which is only possible if the placement log is kept with the metrics.

Mitigation ladder, cheapest first: move the offender (reschedule with anti-affinity), cap it (lower its quota or priority), isolate it (MIG or dedicated nodes), and only then buy capacity.

Trade-offs and pitfalls

• Reserving too much for inference looks safe and wastes the most money. Size the reserve from measured peak throughput per GPU, not from a guess.
• Letting interactive sessions preempt batch sounds fair but destroys long training runs for short experiments. Only the latency-critical class should preempt.
• Quotas without borrowing leave GPUs idle while other teams queue. Quotas with borrowing but without reclaim make the guarantee meaningless.
• Preemption without checkpointing turns every inference spike into hours of lost training.
• Measuring utilisation instead of interference: a GPU at 95% utilisation can be perfectly healthy. The metric that matters is whether the latency-sensitive workload got worse because of its neighbours.

Direct answer

I would build it as a global control plane plus regional data planes: one central system (the control plane) that manages configuration, model versions and tenant routing rules, and independent request-serving infrastructure in each region (the data planes) that keeps serving traffic on its own even if the control plane is briefly unreachable. Each region runs a stateless edge gateway that authenticates the tenant, applies per-tenant rate limits, and routes to shared model-server pools grouped by model family, with dedicated pools only for tenants that buy guaranteed capacity. A central model registry records which model version each tenant is pinned to, so per-tenant versions and rolling upgrades are just routing changes. Isolation comes from per-tenant quotas and fair-share queues on shared GPUs, hard partitioning (dedicated pools or GPU partitions) for premium tiers, per-tenant credentials, and caches keyed by tenant. Every request is metered with tenant, model version, and GPU-seconds, which drives both SLA monitoring and cost allocation.

Requirements I am designing to

• 1,000,000 requests per second (RPS) aggregate, spread over 4 regions.
• Per-tenant SLAs (service-level agreements), for example p99 (99th percentile) latency under 100 ms for a Premium tier.
• Per-tenant throttling and fair-share, per-tenant model versions, rolling upgrades with no cross-tenant impact.
• Assumption for sizing, stated not measured: a mid-size model served on one GPU handles 2,500 RPS at the target latency with batching. The real figure must come from a load test of your model.

Capacity arithmetic

• Evenly spread, each region serves 1,000,000 / 4 = 250,000 RPS, which is 250,000 / 2,500 = 100 GPUs of pure load.
• To survive losing a whole region, the remaining 3 must absorb 1,000,000 / 3 = 333,333 RPS each, which needs 134 GPUs (333,333 / 2,500 = 133.3, rounded up).
• Provision 140 GPUs per region (560 total): 140 x 2,500 = 350,000 RPS capacity. Normal utilization is 250,000 / 350,000 = 71%; after a region failure it is 333,333 / 350,000 = 95%. Tight, so during a regional failover the Free tier is shed first.

Architecture

flowchart LR
  C[Client] --> DNS[Geo DNS / anycast]
  DNS --> GW[Regional gateway: auth, tenant rate limit]
  GW --> RT[Router: tenant to model version to pool]
  RT --> CACHE[(Per-tenant response cache)]
  RT --> SP[Shared GPU pools]
  RT --> DP[Dedicated tenant pools]
  REG[Model registry + tenant config] --> RT
  SP --> MET[Metering + tracing]
  DP --> MET
  MET --> BILL[Cost allocation]

The client reaches the nearest region through Geo DNS or anycast routing (mechanisms that automatically direct each client to the closest healthy region, without the client choosing one itself).

Per-tenant throttling and fair-share

• Token bucket per tenant per region: each gateway keeps local buckets and syncs usage with a regional counter store (for example Redis) every 100 ms or so. A central check on every request would add a network hop to 1M RPS; local buckets with periodic reconciliation allow a small, bounded overshoot instead. A tenant's global limit is split across regions by recent traffic share and rebalanced every few seconds.
• Fair-share on the GPU queue: each model server keeps one queue per tenant and pulls from them with weighted round-robin (Premium weight 4, Standard 2, Free 1). Without this a tenant sending a burst of 50,000 requests fills the batch queue and everyone else's p99 explodes.
• Load shedding by tier: when queue wait exceeds the budget, reject Free-tier requests with HTTP 429 first, then Standard.

Per-tenant model versions and rolling upgrades

• The model registry stores immutable versions (fraud-model:v14) with an artifact hash (a cryptographic fingerprint of the exact model file, so you can tell if the bytes ever changed); tenant config pins each tenant to a version or to a channel such as stable (a channel is a rolling alias that always points at whichever version has most recently been promoted as safe, unlike a version pin, which never moves on its own).
• The router resolves (tenant, model) to a pool that has that version loaded. Two versions can serve side by side; there is no "upgrade the fleet in place" moment.
• Rolling upgrade: deploy v15 to new replicas, send shadow traffic (copies of real requests whose results are discarded) to compare outputs and latency, then move tenants in waves: internal tenants, 1% of Standard, 10%, then all. Premium tenants move only at their chosen window or stay pinned. Any SLO (service-level objective: the target value set for a metric, such as p99 latency) regression for a wave triggers automatic rollback of just that wave's routing entries, so other tenants are untouched.

Tenant isolation

GPU partitioning and model pooling (keeping cost down). Most tenants' models are small and idle much of the time, so dedicating a GPU per model wastes money. Options: MIG slices (strong isolation, fixed sizes), time-slicing (processes take turns on a GPU; cheap but no memory isolation, so one tenant's out-of-memory can affect another), and multi-model servers (for example NVIDIA Triton or KServe ModelMesh) that load and evict many models on one GPU by least-recently-used order.

Low cold start. A cold start is the delay when a request arrives for a model that is not loaded. Keep a warm floor of replicas for every Premium model, preload popular models on node start, keep weights on local NVMe (a fast local solid-state drive, far quicker to read from than fetching over the network) or a node cache rather than pulling multi-GB files from object storage per start, and scale on queue depth before it becomes latency.

Hosted third-party models, a secondary case layered on top of the core design above. If part of the fleet proxies a vendor-hosted model (for example Azure OpenAI in Microsoft Foundry, Microsoft's current branding for its hosted OpenAI-model offering (formerly marketed as Azure OpenAI Service)) instead of serving your own weights, isolation moves to credentials and quota. Give each tenant tier (or each large tenant) its own deployment or resource so one tenant's tokens-per-minute quota cannot be exhausted by another, authenticate with Microsoft Entra ID managed identities (the service's own identity, no long-lived keys in code) rather than one shared API key, keep any per-tenant secrets in a per-tenant Key Vault scope (Key Vault: a managed service for storing secrets like API keys, separate from application code), and log the tenant ID with every upstream call so usage can be traced and billed.

Caching

Cache results only where the model is deterministic for the input (classification, embeddings), keyed by (tenant_id, model_version, hash of normalized input). Tenant ID in the key prevents cross-tenant hits; model version in the key means an upgrade cannot serve stale answers from the previous model. Also cache features and tenant config (routing, quotas) in memory at the gateway with short TTLs (time-to-live).

Observability

Tag every metric and trace with tenant, model version, pool, and region. Per-tenant SLIs (service-level indicators: p99 latency, error rate, 429 rate) are compared against that tenant's SLO (service-level objective). With thousands of tenants, keep per-tenant series for paying tiers and aggregate the long tail by cohort (cohort: a group of similar tenants tracked together as one series instead of individually) to keep metric cardinality (the number of distinct label combinations a monitoring system tracks separately; too many blows up its cost and speed) manageable.

Cost allocation

Record GPU-seconds per request from the model server (batch time divided by requests in the batch). Shared pool cost is split by GPU-seconds share; dedicated pools are charged to their tenant in full; cache hits cost almost nothing and should be priced accordingly.

Reproducibility and packaging formats

This is implementation detail underneath the core design above (rate limiting, fair-share, version routing): the specific formats matter less than the discipline of pinning everything. Package every model as an immutable artifact: weights in a portable format (ONNX, TorchScript, TensorFlow SavedModel, or an MLflow model directory, four common ways to save a trained model's weights and structure to disk), plus the exact serving container image pinned by digest (referenced by its exact cryptographic hash, not a mutable tag like latest, so it can never silently change underneath you), the preprocessing code version, and the dependency lockfile (a file recording the exact version of every software dependency, so a rebuild uses identical code). The registry stores all of these hashes, so "tenant X on v14" means the same bytes in every region and can be rebuilt for an incident investigation.

Trade-offs and pitfalls

• Dedicated pools for everyone gives perfect isolation and a GPU bill several times larger; reserve it for tenants whose contracts pay for it.
• Time-slicing without memory limits is not isolation: one tenant's memory spike crashes a neighbour.
• Upgrading models fleet-wide couples every tenant's risk together; routing-based per-tenant waves keep failures local.
• A cache key without model version silently serves old-model answers after an upgrade.

Direct answer

On Kubernetes I would give each tenant a namespace (a named, walled-off slice of the cluster: its own quotas, permissions and objects) with quotas enforced at admission, run models on shared GPU node pools packed tightly (a node pool is a group of machines of the same hardware type that the scheduler places pods onto) (whole GPUs for large models, MIG partitions for small ones), put an inference layer such as KServe (a Kubernetes add-on that turns a trained model into a running, autoscaled web service, one InferenceService object per model) behind a Gateway API ingress (Kubernetes's traffic-routing layer: the front door that decides which backend service a request reaches) that routes by tenant and model version, and autoscale on in-flight requests or queue depth rather than CPU. Tenants who need hard isolation get tainted, dedicated node pools: a taint is a marker on a node that repels every pod except the ones with a matching toleration, so only that tenant's pods land there. Billing comes from GPU and CPU usage attributed to tenant namespaces and pods, plus per-request metering.

Building blocks and what each does

Start with the first three rows: tenant boundary, quotas and placement, plus standard autoscaling, already give you a working design. Kueue, custom admission policies, canary rollouts and cost tooling are refinements layered on once that base runs, roughly in the order listed.

GPU and CPU packing

GPUs are the expensive unit, and Kubernetes by default schedules a GPU as an indivisible integer. Three ways to share one:

• Whole GPU per replica: simplest, strongest isolation, wasteful for small models.
• MIG (Multi-Instance GPU): on an A100 80GB you can create up to 7 1g.10gb slices, each with its own memory and compute. The NVIDIA device plugin exposes them as separate schedulable resources. Hardware isolation, but fixed slice sizes.
• Time-slicing: the device plugin advertises one GPU as several; pods take turns. Cheap, but no memory isolation, so reserve it for dev or internal tenants.

Worked example. 12 small tenant models, each needing about 8 GB of GPU memory at modest traffic. Whole GPUs: 12 A100s. With 1g.10gb MIG slices (7 per GPU): 12 / 7 = 1.7, so 2 GPUs, a 6x reduction, and each model keeps its own memory boundary.

For CPU pods (pre/post-processing, small models), use the scheduler's NodeResourcesFit plugin with the MostAllocated scoring strategy so pods pack onto fewer nodes and the autoscaler can remove empty ones. Pair it with accurate requests; packing on inflated requests just packs air.

Admission control

Two meanings, both needed:

1. Kubernetes admission (at deploy time): a tenant cannot create a pod that exceeds its namespace quota, lacks requests/limits, or targets another tenant's dedicated pool. Kueue holds GPU workloads until quota is available instead of letting them sit Pending (Kubernetes's status for a pod that is accepted but cannot yet be placed on any node) and fragment nodes.
2. Request admission (at run time): the model server or gateway bounds in-flight requests per tenant. When a tenant's queue exceeds its share, return HTTP 429 quickly rather than letting latency climb for everyone.

Routing and versioned deployments

flowchart LR
  R[Request with tenant token] --> G[Gateway: auth, rate limit]
  G --> H{Route by tenant + model}
  H -->|stable 90%| V1[InferenceService v1]
  H -->|canary 10%| V2[InferenceService v2]
  H -->|dedicated tenant| D[Tenant pool on tainted nodes]

Each model version is a separate revision. Setting canaryTrafficPercent: 10 on the KServe InferenceService sends 10% of that model's traffic to the new revision; promote or roll back by changing the number. Because each tenant's model is its own service, one tenant's rollout never touches another's pods.

Autoscaling

CPU utilization is a poor signal for GPU inference (the CPU is often idle while the GPU is saturated). Scale on concurrency per replica or queue depth. Scale-to-zero saves money for rarely used tenant models but brings a cold start: pulling a multi-GB image and loading weights can take tens of seconds to minutes, so keep a minimum of one replica for any model with a latency SLO (service-level objective: the latency target that model's callers were promised), and cache weights on the node.

Tenant billing

• Dedicated pools: bill the node cost to the tenant.
• Shared pools: attribute usage per pod from the resource metrics (CPU and memory from cAdvisor; GPU utilization and memory from NVIDIA's DCGM exporter, DCGM being the Data Center GPU Manager), aggregated by namespace. Tools such as OpenCost (an open-source cost-allocation tool) turn those raw metrics into a namespace-level cost breakdown. A MIG slice is billed as its fraction of the GPU (a 1g.10gb slice is one seventh).
• Add per-request metering (tokens or GPU-milliseconds) when several tenants share one model replica, because pod-level metrics cannot split a shared process.

Trade-offs and pitfalls

• Namespaces are a soft boundary. They share the kernel (the core operating-system program every process on the node goes through to touch CPU, memory, disk and network) and the control plane (the cluster's own management services, separate from the node running your pods), plus the node itself. A tenant that can run arbitrary code (custom model containers) needs a sandboxed runtime (for example gVisor or Kata Containers) or dedicated nodes.
• GPU fragmentation: mixed MIG layouts and whole-GPU pods on the same nodes leave stranded capacity. Keep separate node pools per layout.
• Quota without borrowing leaves GPUs idle while another tenant queues; Kueue cohorts fix that, with preemption to reclaim.
• Autoscaling on CPU under-scales GPU services; use request-level metrics.

Direct answer

A multi-tenant AI service shares expensive machinery (GPUs, base models, vector indexes) across customers, so the architecture question is at which layer each tenant gets its own copy. There are three levels: data-level tenancy (one shared model, each tenant's data and context kept separate), model-level tenancy (per-tenant fine-tuned weights or adapters on a shared base model), and infrastructure-level tenancy (dedicated GPUs, nodes, or accounts). Most platforms mix them by tier: data-level for everyone, model-level for tenants who customize, infra-level for tenants who pay for guaranteed capacity or need hard compliance separation. On top of that you need token-based quotas with fair-share scheduling, per-tenant cost metering, and security controls aimed at AI-specific leaks such as shared caches and prompt injection.

The three levels of tenancy

A useful mental model: each step down the table buys stronger isolation and costs more, because a GPU running one tenant's workload is idle whenever that tenant is quiet.

Isolation strategies at each level

• Data-level: every retrieval query is filtered by tenant, enforced by the index itself (a separate namespace or collection per tenant, or a mandatory metadata filter the application cannot omit). Embeddings are data too: an inversion attack can approximately reconstruct the original text from its embedding vector, given enough queries or a similar model, so treat them with the same access rules as the source documents. Conversation history and any response or prompt caches are keyed by tenant_id.
• Model-level: the router maps (tenant, model name) to a specific adapter or weights version. Serving engines such as vLLM (a widely used open-source engine for running LLM inference efficiently) can hold many LoRA adapters on one base model and select one per request, so model-level tenancy does not require one GPU per tenant. Training pipelines must never mix tenants' data unless contracts allow it.
• Infra-level: separate node pools (groups of machines in a Kubernetes cluster, the system that schedules and runs containerized workloads across many machines, set aside for one purpose) using Kubernetes taints (a marker on a node that repels other workloads unless explicitly allowed, used here to reserve nodes for one tenant) so only that tenant's pods (a pod: the smallest deployable unit in Kubernetes, one or more containers running together) land there, separate clusters, or separate accounts with their own keys and network boundaries.

Resource quotas and fair-share

Request counts are a poor quota unit for large language models because a request can cost 100 or 100,000 tokens. Quota on tokens per minute (TPM) and requests per minute (RPM), and on concurrent requests.

• Token bucket per tenant: a bucket refills at the tenant's TPM rate; a request reserves its estimated tokens (prompt tokens plus max_tokens) up front and the difference is refunded when it finishes.
• Fair-share scheduling: when GPUs are saturated, the scheduler serves tenants in proportion to their weight (for example Enterprise weight 4, Pro 2, Free 1) rather than in arrival order, so one tenant's burst cannot fill the whole queue.
• Guaranteed vs burst capacity: a tenant can have a reserved share it always gets plus the right to borrow idle capacity, which is reclaimed when others need it.

Why noisy neighbours are worse for AI: a worked example

On a GPU, the scarce resource during generation is often memory for the KV cache. As the model reads a prompt and generates each new token, its attention mechanism computes a "key" and "value" vector per token in every layer; a layer is one of the model's stacked processing blocks (more layers mean more sequential computation per token), and within each layer the computation is split into several heads that each attend to the input differently, with head dimension the size of the vector each head produces. Caching these keys and values means the model does not have to redo that work for every earlier token on every new step, but the cache must sit in GPU memory for as long as that conversation stays active, which is what makes it scarce. For a Llama-3-8B-class model (32 layers, 8 key/value heads, head dimension 128, 16-bit values):

• KV cache per token = 2 (keys and values) x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KiB.
• One request with a 32,768-token context therefore holds 32,768 x 128 KiB = 4 GiB; a 4,096-token request holds 0.5 GiB.
• On an 80 GiB GPU with roughly 16 GiB taken by the weights, about 64 GiB is left for KV cache: about 16 concurrent 32k-context requests, or about 128 concurrent 4k-context requests.

So one tenant sending long-context requests can cut everyone else's concurrency by 8x even if its request count is modest. Mitigations: cap max_tokens and context length per tier, count context tokens (not requests) against quota, give long-context traffic its own replicas, and use a scheduler that bounds each tenant's share of in-flight KV memory.

Cost allocation

Meter per request: tenant, model, adapter, prompt tokens, output tokens, and GPU-seconds (from the serving engine's timing). For shared GPUs allocate the pool's cost by each tenant's share of tokens (or GPU-seconds, which captures long-context cost better). For dedicated infra, the tenant carries the full node cost whether it is busy or idle. Charge embedding storage and retrieval separately, since a tenant with 50 million document chunks has a very different storage bill from one with 5,000.

Security implications specific to AI

• Cross-tenant cache leaks: prompt or prefix caches (a cache that stores the computed result for a prompt's shared beginning, so a repeated prefix does not have to be recomputed from scratch) shared across tenants can reveal another tenant's prompt through a cache hit or, more subtly, through response-time differences (a timing side channel). Scope caches per tenant.
• Prompt injection across the boundary: prompt injection is malicious or misleading text hidden inside content the model reads, written to trick it into ignoring its real instructions. A document uploaded by tenant A must never be retrievable in tenant B's context, since it could carry exactly this kind of payload. Retrieval filters enforced by the store, not by prompt instructions, are the control.
• Training-data bleed: fine-tuning on pooled data can make a model repeat one tenant's content to another. Keep per-tenant training sets separate, and make shared-model training opt-in by contract.
• Credential and key separation: per-tenant API keys, per-tenant encryption keys for stored prompts and documents, and audit logs that record which tenant's data each request touched.

Trade-offs and pitfalls

• Treating AI tenancy exactly like web tenancy (quota by requests) lets long-context tenants starve others, as the numbers above show.
• Jumping to infra-level isolation for everyone multiplies GPU cost; most tenants are well served by data-level plus model-level isolation on shared GPUs.
• A mandatory tenant filter that lives only in application code will eventually be forgotten in one query path; push it into the retrieval store's access layer.