Multi-Tenancy and Isolation Questions

Direct answer

A multi-tenant AI service shares expensive machinery (GPUs, base models, vector indexes) across customers, so the architecture question is at which layer each tenant gets its own copy. There are three levels: data-level tenancy (one shared model, each tenant's data and context kept separate), model-level tenancy (per-tenant fine-tuned weights or adapters on a shared base model), and infrastructure-level tenancy (dedicated GPUs, nodes, or accounts). Most platforms mix them by tier: data-level for everyone, model-level for tenants who customize, infra-level for tenants who pay for guaranteed capacity or need hard compliance separation. On top of that you need token-based quotas with fair-share scheduling, per-tenant cost metering, and security controls aimed at AI-specific leaks such as shared caches and prompt injection.

The three levels of tenancy

A useful mental model: each step down the table buys stronger isolation and costs more, because a GPU running one tenant's workload is idle whenever that tenant is quiet.

Isolation strategies at each level

• Data-level: every retrieval query is filtered by tenant, enforced by the index itself (a separate namespace or collection per tenant, or a mandatory metadata filter the application cannot omit). Embeddings are data too: an inversion attack can approximately reconstruct the original text from its embedding vector, given enough queries or a similar model, so treat them with the same access rules as the source documents. Conversation history and any response or prompt caches are keyed by tenant_id.
• Model-level: the router maps (tenant, model name) to a specific adapter or weights version. Serving engines such as vLLM (a widely used open-source engine for running LLM inference efficiently) can hold many LoRA adapters on one base model and select one per request, so model-level tenancy does not require one GPU per tenant. Training pipelines must never mix tenants' data unless contracts allow it.
• Infra-level: separate node pools (groups of machines in a Kubernetes cluster, the system that schedules and runs containerized workloads across many machines, set aside for one purpose) using Kubernetes taints (a marker on a node that repels other workloads unless explicitly allowed, used here to reserve nodes for one tenant) so only that tenant's pods (a pod: the smallest deployable unit in Kubernetes, one or more containers running together) land there, separate clusters, or separate accounts with their own keys and network boundaries.

Resource quotas and fair-share

Request counts are a poor quota unit for large language models because a request can cost 100 or 100,000 tokens. Quota on tokens per minute (TPM) and requests per minute (RPM), and on concurrent requests.

• Token bucket per tenant: a bucket refills at the tenant's TPM rate; a request reserves its estimated tokens (prompt tokens plus max_tokens) up front and the difference is refunded when it finishes.
• Fair-share scheduling: when GPUs are saturated, the scheduler serves tenants in proportion to their weight (for example Enterprise weight 4, Pro 2, Free 1) rather than in arrival order, so one tenant's burst cannot fill the whole queue.
• Guaranteed vs burst capacity: a tenant can have a reserved share it always gets plus the right to borrow idle capacity, which is reclaimed when others need it.

Why noisy neighbours are worse for AI: a worked example

On a GPU, the scarce resource during generation is often memory for the KV cache. As the model reads a prompt and generates each new token, its attention mechanism computes a "key" and "value" vector per token in every layer; a layer is one of the model's stacked processing blocks (more layers mean more sequential computation per token), and within each layer the computation is split into several heads that each attend to the input differently, with head dimension the size of the vector each head produces. Caching these keys and values means the model does not have to redo that work for every earlier token on every new step, but the cache must sit in GPU memory for as long as that conversation stays active, which is what makes it scarce. For a Llama-3-8B-class model (32 layers, 8 key/value heads, head dimension 128, 16-bit values):

• KV cache per token = 2 (keys and values) x 32 x 8 x 128 x 2 bytes = 131,072 bytes = 128 KiB.
• One request with a 32,768-token context therefore holds 32,768 x 128 KiB = 4 GiB; a 4,096-token request holds 0.5 GiB.
• On an 80 GiB GPU with roughly 16 GiB taken by the weights, about 64 GiB is left for KV cache: about 16 concurrent 32k-context requests, or about 128 concurrent 4k-context requests.

So one tenant sending long-context requests can cut everyone else's concurrency by 8x even if its request count is modest. Mitigations: cap max_tokens and context length per tier, count context tokens (not requests) against quota, give long-context traffic its own replicas, and use a scheduler that bounds each tenant's share of in-flight KV memory.

Cost allocation

Meter per request: tenant, model, adapter, prompt tokens, output tokens, and GPU-seconds (from the serving engine's timing). For shared GPUs allocate the pool's cost by each tenant's share of tokens (or GPU-seconds, which captures long-context cost better). For dedicated infra, the tenant carries the full node cost whether it is busy or idle. Charge embedding storage and retrieval separately, since a tenant with 50 million document chunks has a very different storage bill from one with 5,000.

Security implications specific to AI

• Cross-tenant cache leaks: prompt or prefix caches (a cache that stores the computed result for a prompt's shared beginning, so a repeated prefix does not have to be recomputed from scratch) shared across tenants can reveal another tenant's prompt through a cache hit or, more subtly, through response-time differences (a timing side channel). Scope caches per tenant.
• Prompt injection across the boundary: prompt injection is malicious or misleading text hidden inside content the model reads, written to trick it into ignoring its real instructions. A document uploaded by tenant A must never be retrievable in tenant B's context, since it could carry exactly this kind of payload. Retrieval filters enforced by the store, not by prompt instructions, are the control.
• Training-data bleed: fine-tuning on pooled data can make a model repeat one tenant's content to another. Keep per-tenant training sets separate, and make shared-model training opt-in by contract.
• Credential and key separation: per-tenant API keys, per-tenant encryption keys for stored prompts and documents, and audit logs that record which tenant's data each request touched.

Trade-offs and pitfalls

• Treating AI tenancy exactly like web tenancy (quota by requests) lets long-context tenants starve others, as the numbers above show.
• Jumping to infra-level isolation for everyone multiplies GPU cost; most tenants are well served by data-level plus model-level isolation on shared GPUs.
• A mandatory tenant filter that lives only in application code will eventually be forgotten in one query path; push it into the retrieval store's access layer.

Direct answer

I would build it as a global control plane plus regional data planes: one central system (the control plane) that manages configuration, model versions and tenant routing rules, and independent request-serving infrastructure in each region (the data planes) that keeps serving traffic on its own even if the control plane is briefly unreachable. Each region runs a stateless edge gateway that authenticates the tenant, applies per-tenant rate limits, and routes to shared model-server pools grouped by model family, with dedicated pools only for tenants that buy guaranteed capacity. A central model registry records which model version each tenant is pinned to, so per-tenant versions and rolling upgrades are just routing changes. Isolation comes from per-tenant quotas and fair-share queues on shared GPUs, hard partitioning (dedicated pools or GPU partitions) for premium tiers, per-tenant credentials, and caches keyed by tenant. Every request is metered with tenant, model version, and GPU-seconds, which drives both SLA monitoring and cost allocation.

Requirements I am designing to

• 1,000,000 requests per second (RPS) aggregate, spread over 4 regions.
• Per-tenant SLAs (service-level agreements), for example p99 (99th percentile) latency under 100 ms for a Premium tier.
• Per-tenant throttling and fair-share, per-tenant model versions, rolling upgrades with no cross-tenant impact.
• Assumption for sizing, stated not measured: a mid-size model served on one GPU handles 2,500 RPS at the target latency with batching. The real figure must come from a load test of your model.

Capacity arithmetic

• Evenly spread, each region serves 1,000,000 / 4 = 250,000 RPS, which is 250,000 / 2,500 = 100 GPUs of pure load.
• To survive losing a whole region, the remaining 3 must absorb 1,000,000 / 3 = 333,333 RPS each, which needs 134 GPUs (333,333 / 2,500 = 133.3, rounded up).
• Provision 140 GPUs per region (560 total): 140 x 2,500 = 350,000 RPS capacity. Normal utilization is 250,000 / 350,000 = 71%; after a region failure it is 333,333 / 350,000 = 95%. Tight, so during a regional failover the Free tier is shed first.

Architecture

flowchart LR
  C[Client] --> DNS[Geo DNS / anycast]
  DNS --> GW[Regional gateway: auth, tenant rate limit]
  GW --> RT[Router: tenant to model version to pool]
  RT --> CACHE[(Per-tenant response cache)]
  RT --> SP[Shared GPU pools]
  RT --> DP[Dedicated tenant pools]
  REG[Model registry + tenant config] --> RT
  SP --> MET[Metering + tracing]
  DP --> MET
  MET --> BILL[Cost allocation]

The client reaches the nearest region through Geo DNS or anycast routing (mechanisms that automatically direct each client to the closest healthy region, without the client choosing one itself).

Per-tenant throttling and fair-share

• Token bucket per tenant per region: each gateway keeps local buckets and syncs usage with a regional counter store (for example Redis) every 100 ms or so. A central check on every request would add a network hop to 1M RPS; local buckets with periodic reconciliation allow a small, bounded overshoot instead. A tenant's global limit is split across regions by recent traffic share and rebalanced every few seconds.
• Fair-share on the GPU queue: each model server keeps one queue per tenant and pulls from them with weighted round-robin (Premium weight 4, Standard 2, Free 1). Without this a tenant sending a burst of 50,000 requests fills the batch queue and everyone else's p99 explodes.
• Load shedding by tier: when queue wait exceeds the budget, reject Free-tier requests with HTTP 429 first, then Standard.

Per-tenant model versions and rolling upgrades

• The model registry stores immutable versions (fraud-model:v14) with an artifact hash (a cryptographic fingerprint of the exact model file, so you can tell if the bytes ever changed); tenant config pins each tenant to a version or to a channel such as stable (a channel is a rolling alias that always points at whichever version has most recently been promoted as safe, unlike a version pin, which never moves on its own).
• The router resolves (tenant, model) to a pool that has that version loaded. Two versions can serve side by side; there is no "upgrade the fleet in place" moment.
• Rolling upgrade: deploy v15 to new replicas, send shadow traffic (copies of real requests whose results are discarded) to compare outputs and latency, then move tenants in waves: internal tenants, 1% of Standard, 10%, then all. Premium tenants move only at their chosen window or stay pinned. Any SLO (service-level objective: the target value set for a metric, such as p99 latency) regression for a wave triggers automatic rollback of just that wave's routing entries, so other tenants are untouched.

Tenant isolation

GPU partitioning and model pooling (keeping cost down). Most tenants' models are small and idle much of the time, so dedicating a GPU per model wastes money. Options: MIG slices (strong isolation, fixed sizes), time-slicing (processes take turns on a GPU; cheap but no memory isolation, so one tenant's out-of-memory can affect another), and multi-model servers (for example NVIDIA Triton or KServe ModelMesh) that load and evict many models on one GPU by least-recently-used order.

Low cold start. A cold start is the delay when a request arrives for a model that is not loaded. Keep a warm floor of replicas for every Premium model, preload popular models on node start, keep weights on local NVMe (a fast local solid-state drive, far quicker to read from than fetching over the network) or a node cache rather than pulling multi-GB files from object storage per start, and scale on queue depth before it becomes latency.

Hosted third-party models, a secondary case layered on top of the core design above. If part of the fleet proxies a vendor-hosted model (for example Azure OpenAI in Microsoft Foundry, Microsoft's current branding for its hosted OpenAI-model offering (formerly marketed as Azure OpenAI Service)) instead of serving your own weights, isolation moves to credentials and quota. Give each tenant tier (or each large tenant) its own deployment or resource so one tenant's tokens-per-minute quota cannot be exhausted by another, authenticate with Microsoft Entra ID managed identities (the service's own identity, no long-lived keys in code) rather than one shared API key, keep any per-tenant secrets in a per-tenant Key Vault scope (Key Vault: a managed service for storing secrets like API keys, separate from application code), and log the tenant ID with every upstream call so usage can be traced and billed.

Caching

Cache results only where the model is deterministic for the input (classification, embeddings), keyed by (tenant_id, model_version, hash of normalized input). Tenant ID in the key prevents cross-tenant hits; model version in the key means an upgrade cannot serve stale answers from the previous model. Also cache features and tenant config (routing, quotas) in memory at the gateway with short TTLs (time-to-live).

Observability

Tag every metric and trace with tenant, model version, pool, and region. Per-tenant SLIs (service-level indicators: p99 latency, error rate, 429 rate) are compared against that tenant's SLO (service-level objective). With thousands of tenants, keep per-tenant series for paying tiers and aggregate the long tail by cohort (cohort: a group of similar tenants tracked together as one series instead of individually) to keep metric cardinality (the number of distinct label combinations a monitoring system tracks separately; too many blows up its cost and speed) manageable.

Cost allocation

Record GPU-seconds per request from the model server (batch time divided by requests in the batch). Shared pool cost is split by GPU-seconds share; dedicated pools are charged to their tenant in full; cache hits cost almost nothing and should be priced accordingly.

Reproducibility and packaging formats

This is implementation detail underneath the core design above (rate limiting, fair-share, version routing): the specific formats matter less than the discipline of pinning everything. Package every model as an immutable artifact: weights in a portable format (ONNX, TorchScript, TensorFlow SavedModel, or an MLflow model directory, four common ways to save a trained model's weights and structure to disk), plus the exact serving container image pinned by digest (referenced by its exact cryptographic hash, not a mutable tag like latest, so it can never silently change underneath you), the preprocessing code version, and the dependency lockfile (a file recording the exact version of every software dependency, so a rebuild uses identical code). The registry stores all of these hashes, so "tenant X on v14" means the same bytes in every region and can be rebuilt for an incident investigation.

Trade-offs and pitfalls

• Dedicated pools for everyone gives perfect isolation and a GPU bill several times larger; reserve it for tenants whose contracts pay for it.
• Time-slicing without memory limits is not isolation: one tenant's memory spike crashes a neighbour.
• Upgrading models fleet-wide couples every tenant's risk together; routing-based per-tenant waves keep failures local.
• A cache key without model version silently serves old-model answers after an upgrade.

Direct answer

On Kubernetes I would give each tenant a namespace (a named, walled-off slice of the cluster: its own quotas, permissions and objects) with quotas enforced at admission, run models on shared GPU node pools packed tightly (a node pool is a group of machines of the same hardware type that the scheduler places pods onto) (whole GPUs for large models, MIG partitions for small ones), put an inference layer such as KServe (a Kubernetes add-on that turns a trained model into a running, autoscaled web service, one InferenceService object per model) behind a Gateway API ingress (Kubernetes's traffic-routing layer: the front door that decides which backend service a request reaches) that routes by tenant and model version, and autoscale on in-flight requests or queue depth rather than CPU. Tenants who need hard isolation get tainted, dedicated node pools: a taint is a marker on a node that repels every pod except the ones with a matching toleration, so only that tenant's pods land there. Billing comes from GPU and CPU usage attributed to tenant namespaces and pods, plus per-request metering.

Building blocks and what each does

Start with the first three rows: tenant boundary, quotas and placement, plus standard autoscaling, already give you a working design. Kueue, custom admission policies, canary rollouts and cost tooling are refinements layered on once that base runs, roughly in the order listed.

GPU and CPU packing

GPUs are the expensive unit, and Kubernetes by default schedules a GPU as an indivisible integer. Three ways to share one:

• Whole GPU per replica: simplest, strongest isolation, wasteful for small models.
• MIG (Multi-Instance GPU): on an A100 80GB you can create up to 7 1g.10gb slices, each with its own memory and compute. The NVIDIA device plugin exposes them as separate schedulable resources. Hardware isolation, but fixed slice sizes.
• Time-slicing: the device plugin advertises one GPU as several; pods take turns. Cheap, but no memory isolation, so reserve it for dev or internal tenants.

Worked example. 12 small tenant models, each needing about 8 GB of GPU memory at modest traffic. Whole GPUs: 12 A100s. With 1g.10gb MIG slices (7 per GPU): 12 / 7 = 1.7, so 2 GPUs, a 6x reduction, and each model keeps its own memory boundary.

For CPU pods (pre/post-processing, small models), use the scheduler's NodeResourcesFit plugin with the MostAllocated scoring strategy so pods pack onto fewer nodes and the autoscaler can remove empty ones. Pair it with accurate requests; packing on inflated requests just packs air.

Admission control

Two meanings, both needed:

1. Kubernetes admission (at deploy time): a tenant cannot create a pod that exceeds its namespace quota, lacks requests/limits, or targets another tenant's dedicated pool. Kueue holds GPU workloads until quota is available instead of letting them sit Pending (Kubernetes's status for a pod that is accepted but cannot yet be placed on any node) and fragment nodes.
2. Request admission (at run time): the model server or gateway bounds in-flight requests per tenant. When a tenant's queue exceeds its share, return HTTP 429 quickly rather than letting latency climb for everyone.

Routing and versioned deployments

flowchart LR
  R[Request with tenant token] --> G[Gateway: auth, rate limit]
  G --> H{Route by tenant + model}
  H -->|stable 90%| V1[InferenceService v1]
  H -->|canary 10%| V2[InferenceService v2]
  H -->|dedicated tenant| D[Tenant pool on tainted nodes]

Each model version is a separate revision. Setting canaryTrafficPercent: 10 on the KServe InferenceService sends 10% of that model's traffic to the new revision; promote or roll back by changing the number. Because each tenant's model is its own service, one tenant's rollout never touches another's pods.

Autoscaling

CPU utilization is a poor signal for GPU inference (the CPU is often idle while the GPU is saturated). Scale on concurrency per replica or queue depth. Scale-to-zero saves money for rarely used tenant models but brings a cold start: pulling a multi-GB image and loading weights can take tens of seconds to minutes, so keep a minimum of one replica for any model with a latency SLO (service-level objective: the latency target that model's callers were promised), and cache weights on the node.

Tenant billing

• Dedicated pools: bill the node cost to the tenant.
• Shared pools: attribute usage per pod from the resource metrics (CPU and memory from cAdvisor; GPU utilization and memory from NVIDIA's DCGM exporter, DCGM being the Data Center GPU Manager), aggregated by namespace. Tools such as OpenCost (an open-source cost-allocation tool) turn those raw metrics into a namespace-level cost breakdown. A MIG slice is billed as its fraction of the GPU (a 1g.10gb slice is one seventh).
• Add per-request metering (tokens or GPU-milliseconds) when several tenants share one model replica, because pod-level metrics cannot split a shared process.

Trade-offs and pitfalls

• Namespaces are a soft boundary. They share the kernel (the core operating-system program every process on the node goes through to touch CPU, memory, disk and network) and the control plane (the cluster's own management services, separate from the node running your pods), plus the node itself. A tenant that can run arbitrary code (custom model containers) needs a sandboxed runtime (for example gVisor or Kata Containers) or dedicated nodes.
• GPU fragmentation: mixed MIG layouts and whole-GPU pods on the same nodes leave stranded capacity. Keep separate node pools per layout.
• Quota without borrowing leaves GPUs idle while another tenant queues; Kueue cohorts fix that, with preemption to reclaim.
• Autoscaling on CPU under-scales GPU services; use request-level metrics.

Direct answer

I would split the cluster into three service classes with different promises: real-time inference gets a reserved, non-preemptible slice sized from its peak load plus headroom; interactive experiments get small, time-boxed, guaranteed quotas per team; batch training gets everything else, on a preemptible basis, with mandatory checkpointing. Teams get quotas they are guaranteed, can borrow each other's idle GPUs, and lose borrowed GPUs first when the owner needs them back. Batch training is deliberately the shock absorber: when inference bursts, batch is what gets preempted, which is why batch jobs must be able to resume.

The forces I am balancing

• Cost: GPUs are the most expensive thing in the cluster, so idle GPUs are the main waste. Anything reserved and unused is money lost.
• Latency guarantees for inference: a user is waiting on every inference request. Its SLO (service-level objective, e.g. "99% of requests answer within 300 ms") cannot depend on whether someone launched a big training run.
• Fairness across teams: no team should be able to take the whole cluster just by submitting first.
• Bursty demand: inference traffic spikes (a product launch, a daily peak), and experiments pile up before deadlines.

These pull against each other: the cheapest cluster is 100% packed, the safest inference is 100% reserved. The policy below reserves only for the class that cannot wait and lets everything else share.

The policy, class by class

Priority classes. On Kubernetes this is a PriorityClass per tier. The inference class has the highest value; batch has the lowest. The interactive class can set preemptionPolicy: Never so it waits for capacity instead of evicting batch jobs, which keeps a single notebook from killing a multi-day training run; only inference is allowed to preempt.

Quotas and borrowing. Plain Kubernetes ResourceQuota caps a namespace (a Kubernetes boundary that groups one team's resources and limits) but cannot lend idle GPUs. A queueing layer such as Kueue (a Kubernetes-native job queueing controller) adds that: each team gets a ClusterQueue with a nominalQuota of GPUs (its guaranteed share); queues in the same cohort can borrow each other's unused quota up to a borrowingLimit; a lendingLimit stops a team lending away the part it needs to get back instantly. Preemption is set so that reclaimWithinCohort: LowerPriority lets an owner reclaim lent GPUs from lower-priority borrowers. Borrowed capacity is, by design, the first to go.

For example: team A's queue has a nominalQuota of 10 GPUs and is using 4, so 6 sit idle. Team B's queue also has a nominalQuota of 10 but needs 14, meaning it must borrow 4 GPUs beyond its own share. Because both queues are in the same cohort, and Kueue only admits a request when it fits both the requester's own borrowingLimit and the cohort's actual unused quota, B's borrow of 4 clears its borrowingLimit (say 8) and fits inside A's 6 idle GPUs, so B is admitted at 14. If B instead needed 20, it could still borrow at most 6, the smaller of its borrowingLimit and what A actually has idle, landing at 16; the remaining 4 queues until more capacity is idle, because borrowingLimit caps what a queue may ask for, it does not conjure GPUs no one has free. When A later submits work that needs its full 10 back, reclaimWithinCohort: LowerPriority preempts B's lowest-priority borrowed jobs first, returning A's GPUs. A lendingLimit on A's queue (say 8) would have capped how much of A's own quota could ever be lent out in the first place, protecting a minimum A always keeps for itself; it does not bind in this example, because A's actual idle capacity (6) never reaches its 8-GPU lendingLimit. Day to day, nominalQuota and cohort membership are what decide behaviour; borrowingLimit, lendingLimit and reclaimWithinCohort are the safety valves you tune once interference shows up.

Preemption rules. A preempted batch job must lose little work, so the policy requires checkpointing (saving model state to storage) at a fixed interval, and the scheduler gives a grace period (e.g. 2 minutes) for a final checkpoint before the pod (Kubernetes's smallest deployable unit: one running container or group of containers, here one job's worker) is killed. Jobs that cannot checkpoint are admitted only into a team's guaranteed quota, never onto borrowed GPUs.

Fairness. Within the batch tier I would use a fair-share ordering (the team furthest below its guaranteed share goes next) rather than first-come-first-served. Because jobs ask for GPUs, CPU and memory together, dominant resource fairness (compare teams by whichever resource they use the largest fraction of) is the right generalisation when some teams are CPU-heavy and others GPU-heavy.

Worked example: 64 GPUs, four teams

Assume 8 nodes with 8 GPUs each (64 GPUs). Inference peaks at 2,400 requests per second, and load testing shows one GPU serves 150 requests per second while meeting the p99 (99th-percentile) latency target.

Four teams each get a nominal quota of 44 / 4 = 11 GPUs, all in one cohort so an idle team's GPUs are borrowable. Interactive work is carved out of each team's 11 (for example 2 GPUs each, partitioned into MIG slices).

Bursty inference beyond the reserve. The 25% headroom covers normal bursts. For a spike beyond 20 GPUs, the inference autoscaler adds replicas; those replicas have the top priority class, so the scheduler preempts batch pods (borrowed ones first) to make room. The cost of that preemption is bounded by the checkpoint interval. If a job saves a final checkpoint in its grace period, it loses almost nothing; if it cannot (the save takes longer than the grace period, or the job ignores the signal), it falls back to its last periodic checkpoint. With checkpoints every 30 minutes and preemptions landing at a random point in the interval, that fallback costs on average 30 / 2 = 15 minutes of the job's GPUs.

Cost trade-off made explicit. The 4 headroom GPUs (20 minus 16) are idle most of the day. The alternative is letting batch run on them and preempting on every spike. I would keep the headroom because inference scale-up must beat a pod start plus model load, which takes minutes for a large model, while a latency SLO is judged in seconds. What would flip it: a model small enough to load in seconds, or a traffic pattern predictable enough to pre-scale on a schedule.

Noisy-neighbour interference: measure, then mitigate

A noisy neighbour is a workload that degrades others through a shared resource. On GPUs, the shared resource is not only the GPU itself.

How to measure it. Export per-GPU metrics with NVIDIA DCGM (Data Center GPU Manager) via its Prometheus exporter: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_ACTIVE (SM: streaming multiprocessor, the GPU's core compute unit; the metric is the fraction of time they are busy) and DCGM_FI_DEV_FB_USED (GPU memory used), labelled by pod and team. Then compare each inference replica's p99 latency against its own baseline and correlate the degradations with what was co-scheduled on the node at the time. The signal to alert on is "inference p99 rose while a non-inference workload was placed on the same node or GPU", which is only possible if the placement log is kept with the metrics.

Mitigation ladder, cheapest first: move the offender (reschedule with anti-affinity), cap it (lower its quota or priority), isolate it (MIG or dedicated nodes), and only then buy capacity.

Trade-offs and pitfalls

• Reserving too much for inference looks safe and wastes the most money. Size the reserve from measured peak throughput per GPU, not from a guess.
• Letting interactive sessions preempt batch sounds fair but destroys long training runs for short experiments. Only the latency-critical class should preempt.
• Quotas without borrowing leave GPUs idle while other teams queue. Quotas with borrowing but without reclaim make the guarantee meaningless.
• Preemption without checkpointing turns every inference spike into hours of lost training.
• Measuring utilisation instead of interference: a GPU at 95% utilisation can be perfectly healthy. The metric that matters is whether the latency-sensitive workload got worse because of its neighbours.

Direct answer

I would build a shared control plane in one platform project and per-tenant data boundaries in per-tenant Google Cloud projects, grouped under a folder, with the shared inference path reaching tenant data only through tenant-scoped identities. Every request is authenticated to a tenant at the edge, metered in tokens against that tenant's budget, screened on the way in and out, and served either by a shared base model (with the tenant's own documents and adapters attached only for that tenant) or, for premium tenants, by dedicated capacity. Model governance means one approved-model catalogue, versioned prompts and adapters that belong to exactly one tenant, and evaluation gates before any model change reaches tenants. The operational playbook covers staged model rollouts with per-tenant pinning and a small set of rehearsed incident runbooks, the most important being a suspected cross-tenant leak.

Requirements

• Enterprise tenants use the offering for chat, summarisation and retrieval over their own documents (RAG, retrieval-augmented generation: fetch relevant passages from the tenant's documents and put them in the prompt).
• Tenants may customise: system prompts, retrieval sources, and optionally fine-tuned adapters (small add-on weights, trained on the tenant's data, applied on top of a shared base model).
• Hard requirements: no tenant's data, prompts, outputs or customisations can reach another tenant; cost per tenant is bounded and attributable; every model change is controlled and auditable.

Architecture components

flowchart TB
  U[Tenant users and apps] --> AG[Apigee: auth, per-tenant quotas]
  AG --> ORC[Orchestrator on GKE or Cloud Run]
  ORC --> MA[Model Armor: prompt and response screening]
  ORC --> RET[Retrieval: per-tenant index]
  ORC --> VX[Vertex AI endpoints: shared base models]
  ORC --> DED[Dedicated capacity for premium tenants]
  RET --> TP[(Tenant project: documents, index, adapters, CMEK key)]
  ORC --> MET[Usage metering to BigQuery]
  REG[Model registry and eval gates] -.-> VX

Tiering. Small tenants share the orchestrator and base-model endpoints and get a lightweight tenant project (documents, index, key). Regulated or very large tenants get dedicated endpoints or provisioned throughput, and optionally a dedicated project for the whole data plane in a region they choose (for residency, Assured Workloads, a Google Cloud control that locks a folder of projects to an approved list of regions, personnel and services to meet a specific compliance regime, can constrain which regions and services a tenant's folder may use).

Security controls

1. Tenant binding end to end. The orchestrator derives the tenant from the token, then calls tenant resources using that tenant's service account. Even a bug that passes the wrong document ID fails at IAM (Identity and Access Management, Google Cloud's system for deciding which identity may call which API on which resource), because tenant A's service account has no IAM grant on tenant B's project at all; the call is rejected before the wrong document ID is even evaluated, regardless of what the application code asked for.
2. Retrieval isolation. One index per tenant (or, if shared for cost, a mandatory tenant filter applied by the retrieval service, not by prompt logic). Retrieval that is filtered only by an instruction in the prompt is not isolation.
3. No shared caches without tenant keys. Prompt caches, semantic caches (caches that match on meaning, reusing a cached answer for a differently-worded but equivalent question, not just an exact repeat) and conversation stores are keyed by tenant ID; a cache hit across tenants would return one tenant's answer to another.
4. No training on tenant data into shared weights. Tenant data may train only that tenant's adapter, stored in that tenant's project under its CMEK key.
5. Screening. Model Armor on inputs (prompt injection, including instructions hidden in retrieved documents) and outputs (sensitive data leaving). Tool calls, if the assistant can act, run with the tenant's scoped permissions only.
6. Audit. Cloud Audit Logs on tenant projects; per-request records of tenant, model version, adapter version, prompt template version and token counts, with the content itself stored only in the tenant's project.

Cost control

Cost in generative AI is dominated by tokens, and one tenant's runaway integration can spend more than everyone else combined. Controls, in order:

• Per-tenant token budgets (daily and monthly) enforced in the orchestrator before the model call, with soft alerts at 80% and hard stops or degradation (a cheaper model) at 100%, per the tenant's contract.
• Per-tenant rate limits in Apigee to stop request floods before they reach the models.
• Model routing by task: a small model for classification and short answers, the large model only where quality requires it.
• Budgets in Cloud Billing as a backstop alarm on each tenant project; they notify, they do not cap spend on their own, so the orchestrator's budget is the real control.

Worked example (illustrative prices; use your contracted rates). Assume $0.30 per million input tokens and $2.50 per million output tokens. A tenant with 1,000 users, 50 requests per user per day, 2,500 input tokens (prompt plus retrieved passages) and 500 output tokens per request:

Now a misconfigured integration at that tenant loops at 20 requests per second all day: 20 × 86,400 = 1,728,000 requests, which is 4,320M input and 864M output tokens, or 4,320 × 0.30 + 864 × 2.50 = $3,456 in one day, about 35 times the tenant's normal spend. A daily token budget stops this within minutes; a monthly invoice review finds it weeks later.

Model governance

• Approved-model catalogue in the model registry: which base models and versions are allowed, for which tiers and regions, with an owner and an evaluation record.
• Evaluation gates: a new model or version must pass quality, safety and regression suites, including each tenant's own golden test set (a fixed set of inputs with known-correct expected outputs, used to check a model change has not broken something that used to work) if the tenant provided one, before it is offered.
• Tenant customisations are tenant-owned artifacts: system prompts, retrieval configuration and adapters are versioned per tenant, stored in the tenant's project, and loaded only for that tenant. A tenant's system prompt cannot change another tenant's behaviour because there is no shared mutable prompt; platform-wide safety instructions are a separate, platform-owned layer that tenants cannot override.
• Pinning: each tenant is pinned to a model version; upgrades are explicit, announced and reversible.
• Lineage: for every response, which base model, adapter, prompt template and retrieval sources produced it, for audits and disputes.

Operational playbook

Updates (model or platform):

1. Evaluate offline against the golden sets; publish results.
2. Canary (release to a small slice of traffic first and watch it, before rolling out to everyone) to internal tenants, then 1% of opted-in tenants, watching quality signals (thumbs-down rate, refusal rate), latency and cost per request.
3. Roll forward by tenant cohort; tenants with pinned versions move only on their chosen date.
4. Keep the previous version deployable; rollback is a routing change, not a redeploy.
5. Retire old versions on a published schedule, with notice before any provider-side model retirement.

Incident response:

Trade-offs and pitfalls

• Project per tenant gives strong IAM, key and billing boundaries but adds provisioning work; automate it fully (infrastructure as code) or it becomes the bottleneck. For tens of thousands of tiny tenants, pool them in shared projects with per-tenant indexes and keys, and keep project-per-tenant for paying tiers.
• Shared endpoints are far cheaper than dedicated ones but share capacity; premium tenants with latency guarantees need provisioned capacity.
• Isolation through prompts ("only answer from tenant X's documents") is not a control. Enforce it in IAM, retrieval filters and cache keys.
• Budget alerts are not caps; enforce budgets where the tokens are spent.