Retrieval-Augmented Generation (RAG) Questions

Direct answer

Run one write path and many read paths. Documents are ingested once into a durable, ordered change log in the primary region; every region consumes that log and builds its own searchable copy of the vector index, so users are served by a replica in their own region and never wait on a cross-ocean search. Freshness is the price: a replica is behind the primary by the replication lag plus the index refresh interval, so you state a freshness target (for example "99 percent of new documents searchable everywhere within a minute"), measure it, and offer a stronger read-your-writes option for the few requests that need it. Tail latency is kept low by local replicas, spare replicas to hedge against slow searches (a hedge is a duplicate request sent to a second replica when the first has not answered within a set time, using whichever answers first), and degraded-but-fast fallbacks.

Terms

A vector index stores one embedding vector per text chunk and answers "which stored vectors are nearest to this query vector" approximately; HNSW (a layered neighbour-graph index) is a common choice and keeps the vectors and the graph in memory. A shard is a slice of the index held by one node. A replica is a copy of a shard. Tail latency (p99) is the time within which 99 percent of requests finish. Compaction rewrites an index into a fresh, tidy one, merging small recent additions and dropping deleted entries for good. A monotonic version number is a counter that only ever increases, so a larger number always means a newer state. Anycast advertises one network address from several locations so that each user's traffic reaches the nearest one. A lognormal distribution is a right-skewed one for positive values (its logarithm is bell-shaped): most delays sit near the median and a few are much longer, which is why it models latencies. Read-your-writes means a user who just added a document sees it in their next query.

Design, in the order the data moves

1. Partitioning. Split the index into shards by a key that matches the query pattern. If queries are scoped to a tenant (a customer organisation), shard by tenant: each query touches one shard, and data-residency rules (this tenant's data stays in this geography) become a placement rule. If queries search everything, shard by hash of the chunk id and scatter each query to all shards, then merge the top results; this costs one parallel fan-out per query (a scatter to all shards, followed by a gather) and makes the slowest shard set the latency. That is tail amplification: if each shard is slow on 1 percent of requests, a query that waits for 20 shards is slowed by at least one of them with probability 1 - 0.99^20, about 18 percent. This is why shards each get replicas.
2. Write path. Ingestion (parse, chunk, embed) runs in the primary region and appends records (add, update, delete, plus access labels) to an ordered log with a monotonic version number. Regions subscribe. Each region applies the records to its own index. Shipping the log, not index files, keeps every region independent, lets each rebuild from the log, and gives you a version number to reason about freshness.
3. Index structure for freshness. Graph indexes are slow to update in place and degrade after many deletes. A common pattern is a large immutable main index rebuilt or compacted periodically, plus a small in-memory "recent" index for documents added since the last build; a query searches both and merges results. Deletes are tombstones (a marker that hides the chunk) applied immediately and physically removed at compaction.
4. Replication. Choose per region between a full replica and a partial one. A full replica gives local reads for everything at the cost of the storage in the worked example. A partial one (hot subset or tenants homed in that region) saves memory and sends cold queries to a remote region, so decide from the traffic distribution, not by default. Quantizing vectors (storing each number in 1 byte instead of 4, a coarser but smaller representation) cuts the memory about fourfold at some recall cost that you measure on your own data.
5. Routing. Geo-aware DNS or an anycast edge sends each request to the nearest healthy region; the retrieval and generation services run in that same region, so the whole RAG path (embed the query, search, rerank, call the language model) stays local. On a regional failure, route to the next-nearest region and accept the latency hit.
6. Consistency options per request. The default is eventual: search the local replica. For read-your-writes, the client sends the version number returned by its upload; the replica compares it with the version it has applied, waits up to a small bound (say 200 ms) if it is behind, and otherwise sends that one query to the primary region. A traced case: the upload returns version 1042 and the user asks a question straight away; the local replica has applied version 1040, so it holds the query. If version 1042 arrives within 200 ms it answers locally; if the replica is still at 1041 when the 200 ms are up, that one query goes to the primary, which has version 1042. The few users who need it pay the latency; everyone else does not.
7. Deletes and permission changes must propagate faster than additions: a revoked document or a removed user must stop being returned everywhere within the security target. Give them priority in the log or enforce access checks against a central, quickly replicated authorization service at query time.

Worked example

All inputs are assumed values for illustration (50 million chunks, 768-dimensional vectors, HNSW with M = 16, which allows up to 2 x 16 = 32 links per node in the bottom layer, each link a 4-byte id, so the bottom-layer graph is 50 million x 32 x 4 bytes = 6.4 GB; lognormal replication delays; and a 2 percent chance a search runs six times slower, as when a disk read stalls or a garbage-collection pause hits that one machine); none is a measurement of a real system. The program prints memory arithmetic, a freshness simulation, and a tail-latency simulation with 200,000 samples each.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np

# Assumed inputs (stated, not measured): 50 million chunks, 768-dim vectors, HNSW with M=16 and 4-byte ids.
N, DIM, M_LINKS = 50_000_000, 768, 16
GB = 1e9
vec_f32 = N * DIM * 4
vec_i8 = N * DIM * 1
links0 = 2 * M_LINKS                   # HNSW's bottom layer allows up to 2 x M links per node
graph = N * links0 * 4                 # layer-0 neighbour lists only (upper layers are a small extra)
print(f"vectors float32 : {vec_f32 / GB:7.1f} GB")
print(f"vectors int8    : {vec_i8 / GB:7.1f} GB")
print(f"graph layer 0   : {graph / GB:7.1f} GB  ({N:,} x {links0} links x 4 B)")
full_f32 = (vec_f32 + graph) / GB
full_i8 = (vec_i8 + graph) / GB
print(f"one full copy   : {full_f32:7.1f} GB (float32)  {full_i8:7.1f} GB (int8, originals kept on disk)")
print(f"3 regions x 1 full copy x 2 replicas each: {3 * 2 * full_f32:7.1f} GB of RAM (float32)")
print(f"4 shards, 3 regions, 2 replicas per shard : {full_f32 / 4:7.1f} GB per node, {4 * 3 * 2} nodes")

# Freshness: time from "document written at the primary" to "searchable in a region".
# Assumed lag model: ship delay lognormal (median 6 s, sigma 0.7) plus a refresh wait uniform in [0, 5] s.
rng = np.random.default_rng(0)
n = 200_000
def visible_after(median_s, extra_s=0.0):
    ship = rng.lognormal(np.log(median_s), 0.7, n)
    refresh = rng.uniform(0, 5.0, n)
    return ship + refresh + extra_s
regions = {"primary": 0.0, "region_b": 6.0, "region_c": 20.0}   # median ship delay, seconds; primary has none
print("freshness (seconds until searchable):")
for name, med in regions.items():
    t = rng.uniform(0, 5.0, n) if med == 0 else visible_after(med)
    print(f"  {name:9s} p50={np.percentile(t, 50):5.1f} p99={np.percentile(t, 99):5.1f} P(<=10s)={np.mean(t <= 10):.3f} P(<=30s)={np.mean(t <= 30):.3f}")

# Read-your-writes: a user uploads, then asks 3 s later. Share of such queries that would miss the new document.
t_b = visible_after(6.0)
print(f"region_b: queries 3 s after a write that would miss it = {np.mean(t_b > 3):.3f}")

# Tail latency. Assumed: RTT 150 ms to the far primary, 10 ms to a local replica; search time lognormal
# (median 30 ms, sigma 0.5) and 2 percent of searches are slow (x6). Hedge: send a copy to a second local replica at 60 ms.
def search_ms(size):
    s = rng.lognormal(np.log(30), 0.5, size)
    slow = rng.random(size) < 0.02
    return np.where(slow, s * 6, s)
remote = 150 + search_ms(n)
local = 10 + search_ms(n)
first, second = search_ms(n), search_ms(n)
hedged = 10 + np.minimum(first, 60 + second)
for name, x in [("remote primary", remote), ("local replica", local), ("local + hedge@60ms", hedged)]:
    print(f"{name:20s} p50={np.percentile(x, 50):6.1f} ms  p99={np.percentile(x, 99):6.1f} ms")
extra = np.mean(first > 60)
print(f"hedge sends a second request for {extra:.3f} of queries")
ordinary = np.mean(rng.lognormal(np.log(30), 0.5, n) > 60)
print(f"why: an ordinary search exceeds 60 ms with probability {ordinary:.3f}; the 2% slow searches almost always do")

Output:

vectors float32 :   153.6 GB
vectors int8    :    38.4 GB
graph layer 0   :     6.4 GB  (50,000,000 x 32 links x 4 B)
one full copy   :   160.0 GB (float32)     44.8 GB (int8, originals kept on disk)
3 regions x 1 full copy x 2 replicas each:   960.0 GB of RAM (float32)
4 shards, 3 regions, 2 replicas per shard :    40.0 GB per node, 24 nodes
freshness (seconds until searchable):
  primary   p50=  2.5 p99=  5.0 P(<=10s)=1.000 P(<=30s)=1.000
  region_b  p50=  8.7 p99= 33.2 P(<=10s)=0.611 P(<=30s)=0.985
  region_c  p50= 22.6 p99=104.7 P(<=10s)=0.084 P(<=30s)=0.674
region_b: queries 3 s after a write that would miss it = 0.973
remote primary       p50= 180.3 ms  p99= 329.8 ms
local replica        p50=  40.5 ms  p99= 194.1 ms
local + hedge@60ms   p50=  40.4 ms  p99= 106.1 ms
hedge sends a second request for 0.100 of queries
why: an ordinary search exceeds 60 ms with probability 0.082; the 2% slow searches almost always do

Reading it:

• Storage drives the replication choice. One full copy is about 160 GB in 32-bit floats; three regions with two replicas each is 960 GB held in RAM. Splitting into 4 shards gives 40 GB per node, which fits ordinary hosts, at 24 nodes. Quantizing to 1 byte per dimension brings one copy to 44.8 GB if the full-precision vectors stay on disk for rescoring.
• Freshness differs per region. In this model the primary is searchable within about 5 seconds (the refresh wait alone), region_b has a p99 of 33.2 seconds and region_c of 104.7 seconds. Against a "within 30 seconds" target, region_b meets it for 98.5 percent of documents and region_c for 67.4 percent, so region_c either needs a faster link or a looser target. Because the target is stated per region, the dashboard can alarm on the right number.
• Read-your-writes matters in the first seconds. In region_b, 97.3 percent of queries issued 3 seconds after a write would miss the new document without the version-token mechanism.
• Local replicas cut the median and the tail; hedging cuts the tail further. The remote primary has p99 of 329.8 ms, the local replica 194.1 ms, and the hedged local pair 106.1 ms. The hedge (a duplicate request sent to a second replica if the first has not answered after 60 ms) fires on 10 percent of queries here: 60 ms is twice the 30 ms median, an ordinary search exceeds it about 8 percent of the time (the second printed line), and nearly all of the 2 percent slow searches do too. So it costs about 10 percent more search load for this latency gain. Hedging only helps when the slowness is independent per replica, as it is in this simulation (a slow disk, a garbage-collection pause, a noisy neighbour), not when all replicas are slow together (a bad query, an overloaded shard).

Freshness versus latency, stated as a trade-off

Waiting for a replica to catch up buys freshness at the price of latency; returning immediately buys latency at the price of staleness. Commit to eventual consistency as the default because almost all RAG questions concern documents that have existed for hours or days, expose the strong option on request, show the index version or "as of" time in the response so users and logs can see staleness, and set a per-request wait bound so the strong option cannot block forever. What would flip the choice: an interactive "upload then ask about it" product makes read-your-writes the default; a compliance product makes deletes the strongest-consistency path.

Failure modes and mistakes

• Replication lag grows silently after a regional incident; alert on the gap between the primary's latest version and each region's applied version, not on CPU.
• Partial failures in a fan-out: if one shard times out, return results from the others with a "partial" flag rather than failing the request, and track the rate.
• Cross-region fallbacks on every miss turn a regional outage into a global overload; cap them.
• Single-copy ingestion: an embedding-model change needs a full re-embed and a new index in every region, so version the index and switch regions one at a time.
• Assuming strong consistency from an eventually consistent store and then debugging "missing" documents.
• Hedging without limits doubles load during an incident; budget hedges as a small percentage of requests.

Direct answer

Enforce access control inside retrieval, before any text reaches the language model: authenticate the user once, resolve their groups and clearances into a filter, apply that filter as part of the vector search itself (vector search finds the stored vectors nearest to the query's vector; applying the filter inside that search, before results are cut to the top k, is a pre-filter), and log what was retrieved and shown. The model must never be the thing that decides what a user may see, because anything placed in its prompt can be quoted in its answer. Latency stays low because the identity lookup is cached for a short time and the filter is a metadata predicate evaluated inside the index, not a second pass over results.

The request path, in order

1. Authenticate. Verify the caller's token at the edge (who is this, signed and unexpired). Do not accept a user id or a list of groups from the browser body.
2. Authorize, per request. Resolve the user into the set of groups and clearance level the document ACLs (access-control lists: per-document records of who may read it) are written in. Cache this resolution for a short, bounded time (for example 60 seconds, an illustrative value) so group changes take effect quickly; the cache time is the revocation delay and should be stated.
3. Build the filter and pass it to the search. Every chunk is stored with its ACL as metadata copied from the source document at ingestion. The query carries a predicate such as "chunk ACL intersects user groups AND classification <= user clearance". The search returns only chunks that pass it.
4. Assemble context only from filtered results. Reranking, query rewriting, caching and any follow-up fetch use the same filter. A shared answer cache must be keyed by the access scope too, otherwise one user's answer is served to another.
5. Check the output. Every citation in the answer must be one of the chunks given to the model on this request. An answer citing something else is withheld or repaired. This catches the model naming or quoting material that came from somewhere other than the permitted context, for example from a stale conversation memory.
6. Audit. Write an append-only record per request: request id, user, the groups they were resolved to, the chunk ids given to the model, the chunk ids cited, and any blocked citation. Store ids, not text, so the audit log is not a second copy of the sensitive corpus. Write it asynchronously off the critical path (a queue), with a guarantee that a failed write alarms rather than silently drops.

Why pre-filtering rather than filtering the results afterwards

The program below uses random unit vectors as stand-ins for embeddings (a real system would use an embedding model) and brute-force similarity search. The corpus has 2,000 chunks, each readable by one or two of six groups; each of 300 simulated users belongs to two groups. The point is the selectivity arithmetic, which does not depend on the embedding model. Selectivity here means the share of the corpus a user's filter lets through: a user who may read 47 percent of the chunks has a selectivity of 0.47, and a user who may read 2 percent has a narrow filter.

How to read the code: V holds the 2,000 stand-in vectors and acl[i] is the set of groups allowed to read chunk i. top(qv, allowed_idx, k) scores only the listed chunks against the query vector and returns the best k. In each of the 300 trials a random user with two groups is drawn, ok lists the chunks that user may read, and truth is the best 10 among those, the exact permitted answer. plain is the best 10 over everything, with no access control. Then three strategies are compared against truth: filter plain afterwards (post_ret counts how many results survive, post_recall how many of the true top 10 are among them), fetch 50 then filter (over_ret, over_recall), and search only ok (pre_ret). The lists are averaged at the end to print one line each.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import json
import numpy as np

rng = np.random.default_rng(0)
N, D, K = 2000, 32, 10
V = rng.normal(size=(N, D)); V /= np.linalg.norm(V, axis=1, keepdims=True)   # stand-in embeddings (random unit vectors)
groups = ["hr", "legal", "fin", "eng", "exec", "sales"]
# each chunk is readable by 1 or 2 groups (its access-control list)
acl = [set(rng.choice(groups, size=rng.integers(1, 3), replace=False)) for _ in range(N)]

def top(qv, allowed_idx, k):
    idx = np.array(allowed_idx)
    sc = V[idx] @ qv
    return idx[np.argsort(-sc)[:k]].tolist()

n_trials = 300
share_ok, unauth_share, post_ret, post_recall, over_ret, over_recall, pre_ret = [], [], [], [], [], [], []
for _ in range(n_trials):
    user = set(rng.choice(groups, size=2, replace=False))
    ok = [i for i in range(N) if acl[i] & user]
    share_ok.append(len(ok) / N)
    qv = rng.normal(size=D); qv /= np.linalg.norm(qv)
    truth = top(qv, ok, K)                              # exact answer restricted to what the user may see
    plain = top(qv, range(N), K)                         # no access control at all
    unauth_share.append(np.mean([not (acl[i] & user) for i in plain]))
    post = [i for i in plain if acl[i] & user]           # filter AFTER retrieving k
    post_ret.append(len(post)); post_recall.append(len(set(post) & set(truth)) / K)
    over = [i for i in top(qv, range(N), 5 * K) if acl[i] & user][:K]   # fetch 5k, then filter
    over_ret.append(len(over)); over_recall.append(len(set(over) & set(truth)) / K)
    pre_ret.append(len(top(qv, ok, K)))                  # filter INSIDE the search

print(f"mean share of chunks a user may read: {np.mean(share_ok):.2f}")
print(f"no filter: unauthorized share of top-{K}: {np.mean(unauth_share):.2f}")
print(f"post-filter k={K}:   results {np.mean(post_ret):.2f}, recall vs exact allowed top-{K} {np.mean(post_recall):.2f}")
print(f"candidates needed on average to find {K} allowed chunks: {K / np.mean(share_ok):.1f}")
print(f"overfetch 5x:       results {np.mean(over_ret):.2f}, recall vs exact allowed top-{K} {np.mean(over_recall):.2f}")
print(f"pre-filter:         results {np.mean(pre_ret):.2f}, recall 1.00 by construction")

# Per-request enforcement, citation check and audit record (illustration with fixed values)
user_groups = {"legal", "fin"}
allowed_ids = [i for i in range(N) if acl[i] & user_groups]
shown = top(V[7], allowed_ids, 3)
forbidden = next(i for i in range(N) if i not in set(allowed_ids))
cited_by_model = shown[:2] + [forbidden]                 # model cites a chunk it was never given
violations = [c for c in cited_by_model if c not in set(shown)]
audit = dict(request="r-001", user="u-42", groups=sorted(user_groups), chunks_given=shown, cited=cited_by_model, blocked_citations=violations)
print(json.dumps(audit))

Output:

mean share of chunks a user may read: 0.47
no filter: unauthorized share of top-10: 0.55
post-filter k=10:   results 4.48, recall vs exact allowed top-10 0.45
candidates needed on average to find 10 allowed chunks: 21.5
overfetch 5x:       results 10.00, recall vs exact allowed top-10 1.00
pre-filter:         results 10.00, recall 1.00 by construction
{"request": "r-001", "user": "u-42", "groups": ["fin", "legal"], "chunks_given": [7, 257, 329], "cited": [7, 257, 0], "blocked_citations": [0]}

Reading it. With no access control, 0.55 of the top ten are chunks the user may not read, and those would go straight into the prompt. Filtering after retrieving ten leaves on average 4.48 results and only 0.45 of the exact permitted top ten, because most of the ten slots were wasted on forbidden chunks. Overfetching (retrieve 50, then filter) recovers the full list here, because this average user may read about 0.47 of the corpus, so finding ten permitted chunks takes about 10 / 0.47 = 21 candidates (the printed 21.5 divides by the unrounded share), and 50 candidates is comfortably more than that. The overfetch factor is a guess about selectivity: a user who may read a small share of the corpus needs about K divided by that share candidates (here K = 10), so the fetch size has to grow as the share shrinks, and the cost and the recall both degrade as selectivity falls. A pre-filter does not depend on that guess. The last line is the per-request audit record, and it shows the output check firing: the model cited chunk 0, which was never given to it, and the record lists it under blocked_citations.

Latency

• The authorization lookup is one cached call; the filter is a predicate inside the index, whose cost depends on the engine and on how selective the predicate is. Measure p95 (the latency that 95 percent of requests beat) with your real ACL distribution, because narrow filters are the slow case for approximate indexes. An approximate index finds near neighbours without scanning every vector, typically by walking from one nearby vector to the next; with a filter, it discards each visited vector the user may not read, so when the filter lets through only a few percent of the corpus most visited vectors are discarded and the search has to visit many more of them (or return fewer than k results) to find enough permitted neighbours. How an engine handles this differs, so test it.
• If one tenant or group is tiny, give it its own partition (a physically separate slice of the index holding only that group's chunks) or index; the filter then costs nothing and isolation is stronger.
• Audit logging and the output check are the only additions after generation; keep them cheap (set membership on ids) and the log write asynchronous.

Trade-offs and failure modes

• Stale permissions. Revoking access in the source system must reach the index. Either copy ACL changes through the same incremental update path (with a short lag target and an alarm) or re-check authorization against the source of truth for the top results at query time, which costs a call per request.
• Leakage during generation. The guarantee comes from what is in the prompt: if only permitted chunks are there, the model has nothing restricted to quote. The citation check above is a backstop that catches references to chunks that were not supplied; it does not prove that every sentence is grounded in permitted text, so keep restricted material out of model weights (no fine-tuning on it) and out of any memory shared between users.
• Derived data. Summaries, caches, embeddings of conversation history and fine-tuned weights are copies of documents. Each must inherit the ACL of its inputs or not exist.
• ACL explosion. Per-user ACLs on millions of chunks are expensive; prefer groups and roles, and a classification level, as the stored attribute.
• Common mistakes: trusting groups sent by the client; filtering after retrieval and calling it done; one shared cache across users; logging full chunk text into the audit trail.

Direct answer

A four-week proof of concept (PoC) should answer one question: on our own documents and our own users' questions, does a retrieval-augmented assistant give correct, cited answers more often than what people use today, safely enough to pilot? Fix the evaluation set and the pass/fail rules in week 1, before anyone sees results; use the simplest retriever-plus-generator that can work; measure retrieval and answers separately; compare to the current search on the same questions; and write the stop conditions down in advance.

Four-week plan

Data and privacy requirements

Decide before ingestion: which repositories are in scope, which classifications are excluded, whether personal data is present and redacted, where the data may be processed (inside your boundary or at a service whose retention terms were reviewed), and how the assistant respects each user's existing document permissions (no user may retrieve through the assistant what they cannot open directly). A PoC that bypasses access control proves nothing about production.

Minimal model choices

One embedding model, one generator and optionally one reranker, chosen for being available and permitted, not for being impressive. Keep them behind interfaces so a swap is a config change. Do not tune models in a PoC; if the system needs fine-tuning to work, that is a finding about cost, not something to hide.

Evaluation criteria

• Retrieval: recall@5 of the gold passages (is the evidence found).
• Answer quality: correctness against the gold answer, and citation support (is each claim backed by a cited passage). Use human labels on the full set or on a large random sample, and calibrate any model judge against them.
• Safety: unsupported-claim rate, refusal on out-of-corpus questions, zero access or privacy leaks in a planted-canary test (plant a document holding a unique made-up secret string that a given test user is not allowed to open, then check that no answer for that user ever contains it).
• Comparison: same questions, same graders, against the baseline.
• Practical: latency and cost per question, extrapolated to pilot volume.

Worked example: how big a test set, and what counts as a result

Sample size sets how precisely you can state accuracy. Four terms first. A 95 percent confidence interval is the range of true accuracies that the sample is compatible with; the Wilson interval is one standard formula for it that behaves well at small n. A paired comparison scores two systems on the same questions. The discordant questions are those where exactly one system is right; questions both get right or both get wrong say nothing about which is better. The exact sign test asks: if the two systems were equally good, each discordant question would favour either one with chance 1/2, like a fair coin, so how likely is a split at least as lopsided as the one seen (in either direction)? That likelihood is the p-value, and a small one (conventionally under 0.05) means luck is an unlikely explanation. The program computes Wilson intervals (checked against scipy's implementation), the exact sign test, the same p-value by hand from binomial counts, and the stop rules as a function that is called once on inputs derived from counts and four times on hand-set inputs that each exercise a different branch. The 100-question outcome is an invented example of the calculation, not a result.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import math
from scipy.stats import binomtest

def wilson(k, n, z=1.959964):
    p = k / n
    c = (p + z * z / (2 * n)) / (1 + z * z / n)
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n)
    return c - h, c + h

# Reference check against scipy's Wilson interval
lo, hi = wilson(70, 100)
ref = binomtest(70, 100).proportion_ci(confidence_level=0.95, method="wilson")
assert abs(lo - ref.low) < 1e-4 and abs(hi - ref.high) < 1e-4
print(f"wilson 70/100: {lo:.3f}-{hi:.3f} (matches scipy: {ref.low:.3f}-{ref.high:.3f})")

print("n   observed 70%: 95% interval")
for n in [50, 100, 200, 400]:
    lo, hi = wilson(round(0.7 * n), n)
    print(f"{n:<4}{lo:.3f}-{hi:.3f}  width {hi - lo:.3f}")

# Paired comparison on the SAME questions: new RAG assistant vs the existing keyword search + reading.
both_right, new_only, old_only, both_wrong = 55, 20, 6, 19
n = both_right + new_only + old_only + both_wrong
p = binomtest(new_only, new_only + old_only, 0.5).pvalue
print(f"n={n}  new correct {both_right + new_only}, old correct {both_right + old_only}, discordant {new_only}+{old_only}, exact sign-test p={p:.4f}")
print(f"same test with discordant 12 vs 8: p={binomtest(12, 20, 0.5).pvalue:.4f}")
# The same p-value by hand: 26 discordant questions, each a fair coin flip if the systems are equally good.
# Chance of 20 or more heads (or 20 or more tails) out of 26 = 2 * (sum of C(26, j) for j = 20..26) / 2^26.
by_hand = 2 * sum(math.comb(26, j) for j in range(20, 27)) / 2 ** 26
print(f"by hand: 2 * {sum(math.comb(26, j) for j in range(20, 27))} / {2 ** 26} = {by_hand:.4f}")

# Where the decide() inputs come from: counts from the week-3 scoring (invented, like the paired outcome above).
recall_lo, _ = wilson(82, 100)          # gold passage found in the top 5 for 82 of 100 questions
answer_lo, _ = wilson(75, 100)          # answers judged correct: the 75 'new correct' questions above
_, unsupported_hi = wilson(3, 100)      # 3 of 100 answers contained an unsupported claim
print(f"recall@5 lower bound {recall_lo:.2f}, answer-correct lower bound {answer_lo:.2f}, "
      f"unsupported upper bound {unsupported_hi:.2f}, p {p:.4f}")

def decide(recall_at5_lo, answer_correct_lo, unsupported_hi, p_value, leaks):
    if leaks > 0: return "STOP: access or privacy leak"
    if recall_at5_lo < 0.60: return "STOP: retrieval too weak for generation to fix"
    if unsupported_hi > 0.10: return "STOP: too many unsupported claims"
    if answer_correct_lo < 0.60 or p_value > 0.05: return "NO-GO: not shown better than baseline"
    return "GO to pilot"

print(decide(recall_lo, answer_lo, unsupported_hi, p, 0))
print(decide(0.71, 0.62, 0.14, 0.003, 0))
print(decide(0.52, 0.80, 0.04, 0.001, 0))
print(decide(0.71, 0.62, 0.06, 0.003, 1))
print(decide(0.71, 0.55, 0.06, 0.20, 0))

Output:

wilson 70/100: 0.604-0.781 (matches scipy: 0.604-0.781)
n   observed 70%: 95% interval
50  0.562-0.809  width 0.246
100 0.604-0.781  width 0.177
200 0.633-0.759  width 0.126
400 0.653-0.743  width 0.089
n=100  new correct 75, old correct 61, discordant 20+6, exact sign-test p=0.0094
same test with discordant 12 vs 8: p=0.5034
by hand: 2 * 313912 / 67108864 = 0.0094
recall@5 lower bound 0.73, answer-correct lower bound 0.66, unsupported upper bound 0.08, p 0.0094
GO to pilot
STOP: too many unsupported claims
STOP: retrieval too weak for generation to fix
STOP: access or privacy leak
NO-GO: not shown better than baseline

Reading it. At an observed 70 percent, 50 questions give an interval 0.246 wide, 100 give 0.177, 200 give 0.126 and 400 give 0.089, so 100 to 200 questions is the practical range for a four-week effort: 50 cannot distinguish 60 percent from 80 percent.

The paired outcome: the new system is right and the old one wrong on 20 questions, the reverse on 6, so 26 questions are discordant. If the systems were equally good, those 26 would split like 26 fair coin flips. A split of 20 to 6 is 77 percent for one side, and the chance of a split that lopsided or worse in either direction is 2 x 313,912 / 67,108,864 = 0.0094 (the program prints this hand computation next to the library value), so the assistant is better than the baseline on that set. With 12 and 8 discordant questions the split is 60 percent, which luck produces easily; the program prints p = 0.5034 and the difference would not be distinguishable from chance.

The decide function turns the pre-agreed rules into a decision: it stops for any leak, stops when the lower bound on retrieval recall@5 is under 0.60 (generation cannot add evidence that was not retrieved), stops when the upper bound on unsupported claims exceeds 0.10, and says no-go when correctness is not clearly above the floor or not shown better than the baseline. Its inputs come from the week-3 counts like this: 82 of 100 questions had the gold passage in the top 5, and the Wilson lower bound of that proportion is 0.73; 75 of 100 answers were judged correct (the same 75 as above), lower bound 0.66; 3 of 100 answers contained an unsupported claim, upper bound 0.08; and the p-value is the 0.0094 from the paired test. The first decide call uses exactly those values and prints GO to pilot. The next four calls use hand-set inputs (plausible bounds, not outputs of the example): the first fires the unsupported-claims stop, the second the weak-retrieval stop, the third the leak stop and the fourth the no-go, so with the first call every branch of the function fires once. The thresholds 0.60 and 0.10 are examples to be set with the stakeholders in week 1, not standards.

What would make me stop

Any access or privacy leak; retrieval recall below the floor after one round of fixes to chunking and hybrid search (the problem is then the content or the question mix, not the model); unsupported claims above the agreed rate; or no measurable gain over the baseline. Each stop is a legitimate result, and the memo should say what was learned (for example, "documents are scanned images, so extraction is the real project").

Pitfalls

• Choosing the metrics after seeing the results; demoing hand-picked questions; grading with the same model that generates; testing without the permission model; counting a good demo as evidence; leaving cost and latency to the end.

Direct answer

Put vector search on CPU nodes sized for the index's memory and availability, not for throughput, and put reranking and generation on GPUs sized from the load each query class creates. Classify queries cheaply at the front: the 80 percent short FAQ queries go through an answer cache and a light path (few candidates, little or no reranking, short generation), and the 20 percent long research queries get the full path. Because the long queries hold GPU capacity for seconds each while FAQ queries hold it for fractions of a second, give each class its own GPU pool (a "bulkhead") so a burst of research queries cannot starve FAQ users, and scale the pools on concurrency and queue depth with scheduled prewarming for known peaks. Every number below is an assumed input to show the method; replace them with measurements from your own service.

Terms

ANN (approximate nearest neighbour) search finds vectors close to the query vector without scanning all of them. A reranker reorders retrieved candidates with a slower model. Generation is the language model writing the answer. A slot is one concurrent sequence a GPU can serve (a GPU can process many requests at once). Prewarming is starting capacity before it is needed, because new GPU nodes need minutes to start and load a model. A bulkhead is a separate resource pool per traffic class so one class's overload cannot consume the other's capacity. Utilization is busy time divided by available time. p95 and p99 are the wait that 95 percent and 99 percent of requests stay under. A quantized model stores its weights with fewer bits (for example 8 instead of 16), which makes it smaller and faster at some cost in quality. Queue depth is the number of requests waiting for a slot. Failure zones are groups of machines that do not share power or network, so one failure does not take out both. Token throughput is the number of tokens a GPU generates per second across all its requests. Batching serves several requests in one pass of the model. Admission control rejects or defers requests when a queue is too long. A first-in-first-out queue serves requests in the order they arrive.

Sizing from the load, class by class

1. Write the load per class. For each class: arrival rate, CPU time for search, GPU time for rerank and generation, and the cache hit rate. Time on the GPU depends on the number of rerank pairs and the number of output tokens, so measure them on real traffic.
2. Convert to busy capacity: arrival rate times service time gives the average number of busy servers (Little's law: average number in service = arrival rate times time in service).
3. Divide by a target utilization (here 70 percent) to leave room for variability: queues grow quickly as utilization approaches 100 percent.
4. Add redundancy: at least two replicas of each tier in separate failure zones, whatever the throughput arithmetic says.

CPU versus GPU placement

• Vector search (CPU). ANN indexes such as HNSW live in memory and are lookup-bound; for these assumed numbers throughput needs only one node, so the node count is set by index size in memory, replicas and availability. Keep the index in RAM; use compressed vectors if memory dominates.
• Reranking (GPU or CPU). A cross-encoder reranker (a model reading query and passage together) costs a model pass per pair. For FAQ queries, skip it when the first-stage score is decisive or rerank only a few candidates; for research queries rerank many. A small reranker can be affordable on CPU; measure before assuming a GPU is needed.
• Generation (GPU). This is normally the dominant cost. Levers in order of impact: answer caching for repeated FAQ questions, a smaller or quantized model for the FAQ class (accept only if your evaluation set shows the quality is adequate), limits on output length and on retrieved context, and batching of concurrent requests on one GPU.

Why separate pools, and what flips it

At normal load a pool shared by both classes is a little better, because sharing smooths random fluctuations. When one class surges, the shared pool lets it take every slot. The simulation below shows both effects. The decision: use separate pools when you have a latency target for FAQ users that must hold during research bursts; use one pool with priority queues (FAQ jobs first, long jobs capped to a fraction of slots) when load is low and you want the sharing efficiency. Admission control for the long class (reject or defer with a clear message when its queue exceeds a deadline) is needed in both layouts, because no amount of isolation helps the class that is itself overloaded.

Autoscaling and prewarming

• Scale on the right signal: slot utilization, in-flight requests per replica or queue wait time, not CPU percentage, which says little about a GPU server that is saturated on memory or token throughput.
• Know the lag: starting a GPU node includes provisioning, fetching the serving software and loading model weights; assume minutes. During the lag, a reactive autoscaler cannot protect latency, so the policy has three parts: a minimum replica count that covers normal peak, scheduled prewarming before known daily or weekly peaks, and a small headroom buffer for unexpected rises, with the autoscaler handling the rest.
• Scale down slowly (cool-down periods), because a scale-down followed by a spike pays the start-up lag twice.
• Keep a warm pool of preloaded nodes if cold starts are the bottleneck, and weigh its cost against the cost of failing the SLO (service-level objective: the latency target).
• Degrade before failing: under pressure, skip reranking, shorten the context or the output, or serve a cached answer, in that order.

Worked example

Assumed inputs: 40 requests per second in total, 32 FAQ and 8 research; half of the FAQ answers come from the cache; vector search costs 0.02 CPU-seconds (FAQ) or 0.06 (research) on 16-core nodes; GPU slot time is 0.5 s (FAQ) or 6 s (research) per request; a GPU serves 16 slots; a new GPU takes 180 s to become ready; relative prices are 1 unit per hour for a CPU node and 10 for a GPU node (illustrative units, not any provider's prices). The simulation is a first-in-first-out queue with Poisson arrivals (random arrivals at a fixed average rate), 3,000 s per run and 5 seeds, and it reports the FAQ queue wait only. In plain words: each pool is a list of the times at which its slots become free; a request arriving at time $t$ takes the slot that frees first, starts at the later of $t$ and that time, waits for the difference, and keeps the slot busy for 0.5 s (FAQ) or 6 s (research). The inputs are chosen so that the arithmetic is easy to follow; real slots per GPU depend on model size, memory and sequence lengths, so treat 80 slots and 5 GPUs as an illustration of the method, not as a benchmark. The "burst" rows multiply the research arrival rate by 1.6 for the whole window, so the shared pool is overloaded throughout and its waits keep growing with the window length.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import heapq, math
import numpy as np

# ASSUMED inputs for illustration (not measurements). Raw load: 40 requests/s, 80% short FAQ and 20% long research.
RAW_FAQ, RAW_LONG, CACHE_HIT_FAQ = 32.0, 8.0, 0.5          # half of the FAQ answers come from an answer cache
CPU_FAQ, CPU_LONG, CORES = 0.02, 0.06, 16                    # CPU-seconds of vector search per request, cores per node
GPU_FAQ, GPU_LONG, SLOTS_PER_GPU = 0.5, 6.0, 16              # slot-seconds on the GPU path, concurrent sequences per GPU
TARGET_UTIL = 0.7

cpu_load = RAW_FAQ * CPU_FAQ + RAW_LONG * CPU_LONG
print(f"CPU search: {cpu_load:.2f} busy cores -> {math.ceil(cpu_load / (TARGET_UTIL * CORES))} node(s) for throughput")
lam_faq, lam_long = RAW_FAQ * (1 - CACHE_HIT_FAQ), RAW_LONG
lam = lam_faq + lam_long
load_faq, load_long = lam_faq * GPU_FAQ, lam_long * GPU_LONG
load = load_faq + load_long
slots = math.ceil(load / TARGET_UTIL)
print(f"GPU path: {lam:.0f} requests/s, busy slots FAQ={load_faq:.0f} long={load_long:.0f} total={load:.0f}; at {TARGET_UTIL:.0%} utilization {slots} slots = {math.ceil(slots / SLOTS_PER_GPU)} GPUs")

def simulate(seed, slots_by_pool, route, long_mult=1.0, horizon=3000.0):
    rng = np.random.default_rng(seed)
    rate_long = lam_long * long_mult
    rate = lam_faq + rate_long
    free = {p: [0.0] * n for p, n in slots_by_pool.items()}
    t, waits = 0.0, []
    while t < horizon:
        t += rng.exponential(1 / rate)
        faq = rng.random() < lam_faq / rate
        pool = route["faq" if faq else "long"]
        start = max(t, heapq.heappop(free[pool]))
        heapq.heappush(free[pool], start + (GPU_FAQ if faq else GPU_LONG))
        if faq:
            waits.append(start - t)
    return np.array(waits)

faq_slots = math.ceil(load_faq / TARGET_UTIL)
layouts = {
    f"one shared pool, {slots} slots": ({"all": slots}, {"faq": "all", "long": "all"}),
    f"split pools {faq_slots} + {slots - faq_slots} slots": ({"faq": faq_slots, "long": slots - faq_slots}, {"faq": "faq", "long": "long"}),
}
for mult, label in ((1.0, "normal load"), (1.6, "research burst x1.6")):
    for name, (pools, route) in layouts.items():
        p95, p99, over1 = [], [], []
        for seed in range(5):
            w = simulate(seed, pools, route, long_mult=mult)
            p95.append(np.percentile(w, 95)); p99.append(np.percentile(w, 99)); over1.append(np.mean(w > 1.0))
        print(f"{label:20s} {name:28s} FAQ wait, mean of 5 seeds: p95={np.mean(p95):6.2f} s p99={np.mean(p99):6.2f} s share>1s={np.mean(over1):.3f}")

# Step increase to twice the load; new GPUs need 180 s (assumed) before they take traffic.
need = math.ceil(2 * load / TARGET_UTIL)
extra_gpus = math.ceil((need - slots) / SLOTS_PER_GPU)
growth = 2 * load - slots                                    # slot-seconds of work arriving faster than capacity per second
avg_service = load / lam
print(f"after a 2x step: {need} slots needed (+{extra_gpus} GPUs); until they are ready, backlog grows {growth:.0f} slot-s per s")
print(f"over 180 s: {growth * 180:.0f} slot-seconds of backlog = {growth * 180 / avg_service:.0f} average requests queued")
CPU_NODE, GPU_NODE = 1.0, 10.0                               # relative price units per hour (assumed)
print(f"relative cost per hour: base {math.ceil(slots / SLOTS_PER_GPU) * GPU_NODE:.0f} units; holding {extra_gpus} prewarmed GPUs adds {extra_gpus * GPU_NODE:.0f}; one CPU search node is {CPU_NODE:.0f}")

Output:

CPU search: 1.12 busy cores -> 1 node(s) for throughput
GPU path: 24 requests/s, busy slots FAQ=8 long=48 total=56; at 70% utilization 80 slots = 5 GPUs
normal load          one shared pool, 80 slots    FAQ wait, mean of 5 seeds: p95=  0.00 s p99=  0.00 s share>1s=0.000
normal load          split pools 12 + 68 slots    FAQ wait, mean of 5 seeds: p95=  0.09 s p99=  0.19 s share>1s=0.000
research burst x1.6  one shared pool, 80 slots    FAQ wait, mean of 5 seeds: p95=170.71 s p99=177.15 s share>1s=0.990
research burst x1.6  split pools 12 + 68 slots    FAQ wait, mean of 5 seeds: p95=  0.09 s p99=  0.19 s share>1s=0.000
after a 2x step: 160 slots needed (+5 GPUs); until they are ready, backlog grows 32 slot-s per s
over 180 s: 5760 slot-seconds of backlog = 2469 average requests queued
relative cost per hour: base 50 units; holding 5 prewarmed GPUs adds 50; one CPU search node is 1

Reading it:

• CPU is cheap here: 1.12 busy cores need one node for throughput; add nodes for memory and failure zones, not load.
• The GPU path carries 24 requests per second, of which research queries are 20 percent of the raw traffic but 48 of the 56 busy slots (86 percent of the GPU time), and a single shared pool needs 80 slots, which is 5 GPUs at 70 percent utilization (the committed split plan below rounds each pool up to whole GPUs).
• Normal load: both layouts give FAQ users essentially no queueing (shared p95 0.00 s, split 0.09 s); the split costs a little because its pools cannot borrow from each other.
• Burst: with the shared pool, FAQ p95 wait is 170.71 s and 99.0 percent of FAQ requests wait more than a second; with the split pools it stays at 0.09 s. The long pool is itself overloaded in this scenario, which is why it also needs admission control and a deadline.
• Scaling lag: a step to twice the load needs 160 slots (5 more GPUs), and during the 180 s start-up the backlog grows at 32 slot-seconds per second, reaching 5,760 slot-seconds, the work of about 2,469 average requests. That is why a reactive policy alone cannot meet a latency target for a step increase. Holding 5 spare GPUs raises the hourly cost from 50 to 100 relative units while held, so prewarm on a schedule (or on leading indicators such as the time of day) instead of holding the spare capacity permanently.

The committed plan

The simulation compares 12 + 68 slots to isolate the effect of splitting. A real deployment buys whole GPUs and applies the rule of at least two replicas per tier, so the plan below recomputes the pools from the same busy-slot figures, for normal load and for a scheduled peak at twice the load.

import math
SLOTS_PER_GPU, TARGET_UTIL = 16, 0.7
load_faq, load_long = 8, 48                       # busy slots from the sizing run
def plan(scale):
    faq_slots = math.ceil(scale * load_faq / TARGET_UTIL)
    long_slots = math.ceil(scale * load_long / TARGET_UTIL)
    faq_gpus = max(2, math.ceil(faq_slots / SLOTS_PER_GPU))      # at least two replicas per tier
    long_gpus = max(2, math.ceil(long_slots / SLOTS_PER_GPU))
    return faq_slots, long_slots, faq_gpus, long_gpus
for name, scale in (("normal", 1), ("scheduled peak at 2x load", 2)):
    fs, ls, fg, lg = plan(scale)
    cpu = 2                                        # one node covers throughput; a second for a separate failure zone
    cost = (fg + lg) * 10 + cpu * 1
    print(f"{name:26s} FAQ pool {fs:3d} slots -> {fg} GPUs | research pool {ls:3d} slots -> {lg} GPUs | CPU search nodes {cpu} | relative cost {cost} per hour")
shared = math.ceil(math.ceil((load_faq + load_long) / TARGET_UTIL) / SLOTS_PER_GPU)
print(f"one shared pool at normal load: {shared} GPUs = {shared * 10} units of GPU cost; split pools: {sum(plan(1)[2:])} GPUs = {sum(plan(1)[2:]) * 10} units")
lead = 15 * 60
print(f"prewarm lead: start the extra GPUs {lead // 60} minutes ({lead} s) before the peak, against an assumed {180} s start-up")

Output:

normal                     FAQ pool  12 slots -> 2 GPUs | research pool  69 slots -> 5 GPUs | CPU search nodes 2 | relative cost 72 per hour
scheduled peak at 2x load  FAQ pool  23 slots -> 2 GPUs | research pool 138 slots -> 9 GPUs | CPU search nodes 2 | relative cost 112 per hour
one shared pool at normal load: 5 GPUs = 50 units of GPU cost; split pools: 7 GPUs = 70 units
prewarm lead: start the extra GPUs 15 minutes (900 s) before the peak, against an assumed 180 s start-up

Reading it: the bulkhead costs 2 more GPUs than one shared pool (7 against 5, or 70 against 50 units, 40 percent more at normal load), and that is the price of keeping FAQ latency flat during a research burst; the FAQ pool is two GPUs only because of the replica rule. Prewarming is a schedule: with a daily peak starting at 09:00 (assumed), raise the research pool from 5 to 9 GPUs by 08:45. Fifteen minutes is five times the assumed 180 s start-up, leaving margin for health checks and slow model loads, and the pool scales back after the peak with a cool-down.

Trade-offs and mistakes

• More caching saves the most money but risks stale or cross-user answers: key by permission scope and index version and give it a time-to-live.
• Small model for FAQ: cheaper and faster, but you must show on an evaluation set that quality holds, and keep a fallback to the large model for low-confidence cases.
• Misclassification: a research question routed to the FAQ path gets a thin answer, and the reverse wastes money. Log the route, review samples, and let the FAQ path escalate when retrieval scores are weak.
• Mistakes: sizing for average load instead of peak and variability; autoscaling on CPU; one shared queue with no admission control; counting only generation cost and ignoring the cache and the reranker; scaling down right after a spike; treating the assumed numbers here as facts about your system.

Direct answer

At a billion vectors the design is driven by three numbers: bytes per vector (it decides whether anything fits in RAM), shards touched per query (it decides the tail), and replicas per shard (it decides survival and hot-spot headroom). Priorities: bytes per vector and shards per query matter most; replicas come after. My design: compress vectors with product quantization (PQ: cut each 768-number vector into 64 pieces of 12 numbers and replace each piece by the 1-byte number of the nearest of 256 learned representative pieces, so a 3,072-byte vector becomes a 64-byte code that is searched approximately) so the compressed codes live in RAM, keep the neighbour graph and the full-precision vectors on SSD (solid-state disk), stored together so that every node the search visits also yields its exact vector for re-scoring (recomputing exact distances for the visited candidates and reordering them), partition by semantic cluster with a routing layer that sends each query to a few shards, and run at least two replicas of every shard with extra replicas on the hot ones.

All arithmetic below assumes 768-dimension float32 vectors, 64-byte PQ codes, 8-byte ids and 50 million vectors per shard; change those inputs and the plan changes with them. Why these values: 64 pieces of 1 byte is a typical accuracy-versus-size trade (fewer bytes means a coarser code and lower recall, so tune it against measured recall); an 8-byte id is a 64-bit integer that never runs out; 64 neighbours of 4 bytes each per vector is a typical graph degree, and a 4-byte number can address 4.29 billion vectors, enough for a billion; 50 million vectors per shard keeps one shard's codes at 3.6 GB of RAM and its graph and full vectors at about 166 GB of SSD, small enough for one commodity machine and quick to rebuild or move.

Partitioning and shard key

• Random (hash of id) placement is simple, balanced and good for writes, but any shard may hold any neighbour, so every query must go to every shard. Fan-out equals shard count.
• Semantic (cluster) placement assigns each vector to the shard of its nearest centroid (a centroid is the mean of a k-means cluster). A router compares the query to the shard centroids and contacts only the closest few shards. Fan-out is a design parameter. The measure of whether the router chooses well is routing recall: the fraction of the true top 10 neighbours that sit in the shards the router contacted. The costs are uneven shard sizes, hot spots when topics are popular, and a periodic re-clustering when the data drifts.
• A shard key that matches a business boundary (tenant, region) is the right first level when most queries carry that filter, because it prunes shards for free and gives an access-control boundary; semantic routing then applies inside a tenant.

Memory budget, tail and hot spots, computed

How the disk-resident index searches (the layout DiskANN popularised; the program only counts its sizes): every vector keeps a list of 64 neighbour ids, which forms a graph. A query starts at an entry vector, repeatedly reads the neighbour list of the most promising vector from SSD, scores those neighbours with the in-RAM PQ codes (per query, build a table of the distances from each query piece to its 256 representative pieces; a vector's distance is then the sum of 64 table look-ups), and stops after a bounded number of hops, so the number of SSD reads per query is capped. In that layout each node's SSD record holds its full-precision vector and its neighbour list side by side, so the read that fetches a neighbour list also brings the exact vector, and the exact vectors of all visited nodes re-rank the candidates at the end with no extra reads (the DiskANN paper calls this implicit re-ranking).

Stand-ins: part 3 uses a small synthetic Gaussian-cluster set with scikit-learn 1.5.2 k-means as the clustering step and exact scoring inside the contacted shards; it shows the effect of the shard key, not the numbers of a real corpus.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np
from sklearn.cluster import KMeans

# 1. Memory and node budget (assumptions are the inputs, change them to re-plan)
N, D = 1_000_000_000, 768
TB, GB = 1e12, 1e9
codes = N * 64                      # product-quantization codes, 64 bytes per vector, kept in RAM
ids = N * 8
graph = N * 64 * 4                  # on-disk graph, 64 neighbours of 4 bytes each
full = N * D * 4                    # full float32 vectors on SSD for re-scoring
print(f"PQ: {D} floats / 64 pieces = {D // 64} floats each; {D // 64} x 4 B = {D // 64 * 4} B becomes a 1 B code, so {D * 4} B -> 64 B ({D * 4 // 64}x smaller)")
print(f"RAM for codes+ids : {(codes+ids)/GB:.0f} GB   | SSD graph: {graph/GB:.0f} GB | SSD full vectors: {full/TB:.2f} TB")
print(f"SSD per shard of 50,000,000 vectors: graph {graph/20/GB:.1f} GB, full vectors {full/20/GB:.0f} GB")
print(f"all-in-RAM float32 would need {full/TB:.2f} TB per replica")
per_shard = 50_000_000
shards = N // per_shard
replicas = 3
print(f"shards at {per_shard/1e6:.0f}M vectors each: {shards}; RAM per shard for codes+ids: {(codes+ids)/shards/GB:.1f} GB; "
      f"nodes with {replicas} replicas: {shards*replicas}")

# 2. Fan-out and the tail: P(at least one of n shards answers slowly) when each is slow with probability p
p = 0.01
print("\nshards queried  P(query waits on a slow shard)")
for n in (1, 5, 20, 100):
    print(f"{n:14d}  {1-(1-p)**n:.3f}")
print(f"check for n = 20: P(none slow) = 0.99^20 = {(1-p)**20:.3f}, so P(at least one slow) = 1 - {(1-p)**20:.3f} = {1-(1-p)**20:.3f}")

# 3. Shard-key choice on a small synthetic set (stand-in for the real corpus)
rng = np.random.default_rng(0)
n, d, S, k = 30000, 32, 20, 10
centers = rng.normal(size=(S, d)) * 1.2
X = (centers[rng.integers(0, S, n)] + rng.normal(size=(n, d))).astype("float32")
Q = (centers[rng.integers(0, S, 200)] + rng.normal(size=(200, d))).astype("float32")
d2 = (Q**2).sum(1)[:, None] - 2 * Q @ X.T + (X**2).sum(1)[None, :]
truth = np.argsort(d2, axis=1)[:, :k]

hash_shard = rng.integers(0, S, n)                       # shard key = hash of id: random placement
km = KMeans(S, n_init=1, random_state=0).fit(X)
cl_shard = km.labels_                                    # shard key = nearest k-means cell
cents = km.cluster_centers_

def recall_with_fanout(shard_of, order_fn, fan):
    hit = 0
    for qi in range(len(Q)):
        chosen = order_fn(qi)[:fan]
        keep = np.isin(shard_of, chosen)
        ids_ = np.where(keep)[0]
        top = ids_[np.argsort(d2[qi, ids_])[:k]]
        hit += len(set(top) & set(truth[qi]))
    return hit / (k * len(Q))

rand_order = lambda qi: rng.permutation(S)               # no routing information: any shard may hold the answer
near_order = lambda qi: np.argsort(((cents - Q[qi]) ** 2).sum(1))
print("\nfan-out  recall@10 hash-sharded  recall@10 cluster-sharded + centroid routing")
for fan in (1, 2, 4, 20):
    print(f"{fan:7d}  {recall_with_fanout(hash_shard, rand_order, fan):17.3f}  {recall_with_fanout(cl_shard, near_order, fan):26.3f}")
print("shard sizes (rows): hash min", np.bincount(hash_shard).min(), "max", np.bincount(hash_shard).max(),
      "| cluster min", np.bincount(cl_shard).min(), "max", np.bincount(cl_shard).max())

# 4. Hot spots: skewed traffic over cluster-shards, then replicas allocated in proportion to load
w = 1.0 / np.arange(1, S + 1)                            # Zipf-like popularity of topics
w /= w.sum()
load = w * 1000                                          # queries per second per shard at 1000 total qps, fan-out 1
print(f"\nhottest shard load {load.max():.0f} qps vs mean {load.mean():.0f} qps (ratio {load.max()/load.mean():.1f}x)")
total_nodes = 60
rep = np.full(S, 2)                                      # floor of 2 replicas per shard so any node can fail
while rep.sum() < total_nodes:
    rep[np.argmax(load / rep)] += 1                      # spend each spare node on the currently hottest replica set
print(f"with {total_nodes} nodes: uniform 3 replicas -> max per-node load {load.max()/3:.0f} qps; "
      f"floor of 2 plus load-based extras -> {np.max(load/rep):.0f} qps (replicas of hottest shard: {rep[0]}, of coldest: {rep[-1]})")

Output:

PQ: 768 floats / 64 pieces = 12 floats each; 12 x 4 B = 48 B becomes a 1 B code, so 3072 B -> 64 B (48x smaller)
RAM for codes+ids : 72 GB   | SSD graph: 256 GB | SSD full vectors: 3.07 TB
SSD per shard of 50,000,000 vectors: graph 12.8 GB, full vectors 154 GB
all-in-RAM float32 would need 3.07 TB per replica
shards at 50M vectors each: 20; RAM per shard for codes+ids: 3.6 GB; nodes with 3 replicas: 60

shards queried  P(query waits on a slow shard)
             1  0.010
             5  0.049
            20  0.182
           100  0.634
check for n = 20: P(none slow) = 0.99^20 = 0.818, so P(at least one slow) = 1 - 0.818 = 0.182

fan-out  recall@10 hash-sharded  recall@10 cluster-sharded + centroid routing
      1              0.046                       0.981
      2              0.089                       1.000
      4              0.205                       1.000
     20              1.000                       1.000
shard sizes (rows): hash min 1428 max 1577 | cluster min 672 max 3034

hottest shard load 278 qps vs mean 50 qps (ratio 5.6x)
with 60 nodes: uniform 3 replicas -> max per-node load 93 qps; floor of 2 plus load-based extras -> 23 qps (replicas of hottest shard: 13, of coldest: 2)

Reading the numbers

1. Memory. Float32 vectors for a billion items are 3.07 TB per replica; that is not a RAM plan. PQ codes plus ids are 72 GB for the whole corpus, 3.6 GB per 50-million-vector shard. The graph (256 GB for the whole corpus, 12.8 GB per shard) and the full vectors (3.07 TB for the whole corpus, 154 GB per shard) go on SSD, which is the disk-resident layout described above: the in-RAM codes steer the search, a bounded number of SSD reads fetch each visited node's neighbour list together with its full vector, and the visited candidates are re-scored with those full vectors. Counted cost per query is then SSD reads, so measure read count per query as the budget instead of wall-clock time.
2. Tail. If each contacted shard is slow with probability 0.01 (an assumed figure), the chance that a query waits on at least one slow shard is 0.010 with one shard, 0.182 with 20 and 0.634 with 100. The formula is $1-(1-p)^n$, which assumes shards are slow independently of one another (shards that share a rack, network switch or noisy neighbour are slow together, so measure the real figure): each shard is fast with probability $1-p = 0.99$, all $n$ are fast with probability $0.99^n$ (0.99^20 = 0.818), and the complement is the printed figure. So fan-out multiplies the tail. Cutting fan-out through routing is the first lever; hedged requests (send to a second replica after a short delay and use the first answer) and partial results with a deadline are the second.
3. Shard key. In the synthetic set, contacting 1 of 20 shards gives recall@10 of 0.046 with hash placement and 0.981 with cluster placement plus centroid routing; with 4 shards the figures are 0.205 and 1.000. Shard sizes differ: 672 to 3034 rows under cluster placement against 1428 to 1577 under hash placement. The synthetic topics are all the same size, so here that spread comes from how k-means happened to split them, not from the data; real topics differ widely in size and popularity, so under cluster placement shard sizes must be measured and balanced (split large cells, merge small ones). Real data are less cleanly clustered than this, so the real routing recall must be measured at the fan-out you can afford.
4. Hot spots. With Zipf-like topic popularity (an assumed skew, 1/rank), the hottest shard takes 278 qps against a 50 qps mean, 5.6 times. Three replicas everywhere leaves 93 qps on each hot replica; keeping a floor of 2 replicas per shard and spending the other 20 nodes on the shards with the highest load per replica gives 23 qps worst case (13 replicas on the hottest shard, 2 on the coldest), with the same 60 nodes.

Node failure, writes, consistency, monitoring

• Failures: at least 2 replicas per shard in different failure domains (groups of machines that can fail together, such as one rack, power feed or availability zone), health checks, retry on another replica within the latency budget, and a router that stops sending to a shard whose replicas are all down and returns a partial answer marked as partial.
• Writes: upsert and delete go to the owning shard through a durable log; replicas apply the log in order. Deletes are tombstones filtered at query time until compaction. Consistency choice: eventual with a freshness bound is enough for search; deletions that carry legal or permission meaning need a faster path to every replica.
• Embedding pipeline and query volume: documents flow through an embedding job to a queue to the owning shard; model upgrades require re-embedding the full corpus into a new index version and a blue-green swap, so size the embedding compute for that rebuild, not just steady-state writes. Query volume sets replicas; corpus size sets shards.
• Monitor recall: sample live queries, compute exact top-k on a held-out slice offline, and track recall@k (the share of the exact top k that the live system returns) and routing recall per shard over time; alert on a drop, which also detects centroid drift. Also track per-shard load, p99 per shard, partial-answer rate and replica lag.
• Library choice: pick on four tested properties (disk-resident index support, filtered search, incremental updates, operational tooling) rather than on a name, and keep ids and embeddings in your own store so you can move.

Common mistakes

• Planning memory for float32 vectors and discovering terabytes.
• Sharding by hash and then fanning out to every shard on every query.
• Judging the tail from the median shard instead of the slowest of the contacted shards.
• No floor on replicas, so a single node loss on a cold shard removes data from results.