Retrieval-Augmented Generation (RAG) Questions

Direct answer

Split the request path into an orchestrator (the thin front service that owns the user request) and three backend services, each scaled and deployed on its own: retrieval (query to candidate chunks), reranking (reorder the candidates with a slower, more accurate model), and generation (build the prompt and call the language model). Each boundary carries a versioned request and response schema, a deadline, and the caller's identity (who is asking and what they may read). Authorization is enforced where the data is read (inside the retrieval query), not only at the front door. Caches sit at each stage with keys that include everything that changes the answer, and batching is applied where hardware benefits from it (reranker and embedding calls), not where it only adds waiting time.

Terms

A reranker (often a cross-encoder: a model that reads the query and one passage together and outputs a relevance score) is more accurate than vector similarity but costs one model pass per query-passage pair. Micro-batching means holding requests for a few milliseconds so several can be scored in one model call. Tenant means a customer organisation whose data must stay separate from others. Top-k (top_k) is how many results a caller asks for. Hybrid search combines keyword matching with vector similarity. An access label is a tag stored with each chunk saying who may read it (for example public or internal). Idempotent means repeating a call has the same effect as making it once, so a retry is safe. A semantic cache reuses a stored answer for a question whose meaning is close to an earlier one.

The services and their contracts

1. Orchestrator / API gateway. Authenticates the caller, derives an internal identity (tenant, user, roles, group memberships), assigns a request id and a deadline (deadline propagation: a total time budget, for example 800 milliseconds, that each stage subtracts its own time from and passes on as "time remaining", so a late stage knows to skip optional work), calls the services in order, assembles the response, and applies output policies. It is the only component the client talks to.
2. Retrieval service.
   • Input: {query, tenant, identity claims, top_k, filters, index_version?}. Output: a list of {chunk_id, doc_id, score, text or pointer, access_label, index_version}, plus the version of the index and embedding model that produced it (index_version names the index build, so the same query can be compared across builds).
   • It embeds the query (an embedding model call, cacheable), runs the approximate nearest-neighbour search, optionally merges keyword search (hybrid), and applies the access filter inside the index query.
3. Reranking service.
   • Input: the query and the candidate list (about 20 to 100 chunks); output: the same chunk ids with new scores and the reranker version. It is stateless and GPU- or CPU-bound, so it scales independently of retrieval.
   • It can be skipped: if the deadline is nearly spent or the service is down, the orchestrator continues with the retrieval order and flags the response as degraded.
4. Generation service.
   • Input: the query, the chosen chunks (already authorized) with their ids, the prompt template version, and generation parameters; output: the answer text with citations as chunk ids, the model identifier, and token counts.
   • It holds the prompt template and the context-assembly rules (ordering, truncation to the token budget, de-duplication) and the output checks (citations refer only to supplied ids). Streaming responses use a streamed channel back through the orchestrator.

Cross-cutting rules for every boundary: a schema with explicit versions and a validator that rejects unknown or wrongly typed input; a propagated request id for tracing; the deadline carried as "time remaining"; timeouts and bounded retries only for idempotent calls (all of these calls are read-only, so retrying is safe, but retries multiply load, so cap them); and the component versions echoed in every response so the logs can reconstruct what ran.

Where authorization goes

Check at three places, for different reasons. The gateway authenticates and builds the identity (who is asking). The retrieval service enforces the data filter, because it is the component that reads the index and must never rank chunks the caller cannot see. The orchestrator re-checks document-level permission when it turns citations into links or fetches full documents, since a permission can change between indexing and answering. The reranker and generator receive only what retrieval already authorized, and service-to-service credentials (a token or certificate each service presents to the next, separate from the end user's login) ensure only the orchestrator can call them. Putting the filter after the top-k cut is a defect: restricted chunks use up result slots, so legitimate users get fewer results (the worked example shows this), and a prompt instruction is not an access control.

Batching strategy

• Retrieval is usually a per-request call; batching the query embedding across concurrent requests helps only when the embedding model runs on an accelerator, and then with a window of a few milliseconds.
• Reranking benefits most: with 40 candidates per request, 8 concurrent requests make 320 query-passage pairs. Scoring per request takes 8 model calls, one per request, because 40 pairs fit in one batch of at most 64 pairs; pooling the 320 pairs across requests takes 320 / 64 = 5 calls. The price is the waiting window, so pick it from the deadline budget (a few milliseconds, not tens), and cap the batch by memory.
• Generation: inference servers commonly batch many requests at the token level on their own, so the service should send requests promptly instead of building its own queue.
• Separate queues for interactive and bulk (evaluation, backfill) traffic, so a large job does not starve user requests.

Caching, by stage

Permission scope belongs in every key that holds content derived from the data. Short time-to-live plus event-driven invalidation on document change or delete bounds staleness.

A traced request with real values

One request goes through the orchestrator and the three services. Everything runs on stand-ins (TF-IDF as the retriever, word overlap as the reranker, string joining as the generator, the token counts and remaining time written by hand), so the point is the shape of each payload and what crosses each boundary, not the scores. access_label is the sensitivity tag stored with each chunk at ingestion; index_version and embed_version say which index and embedding model produced the result; top_k is the number of chunks the caller wants back; deadline_ms is the time budget for the whole request.

import json
from sklearn.feature_extraction.text import TfidfVectorizer

# Stand-ins: TF-IDF retriever, word-overlap reranker, string-joining generator.
CHUNKS = [  # (chunk_id, tenant, access_label, text)
    ("a1", "acme", "internal", "Acme refund policy allows returns within thirty days."),
    ("a2", "acme", "internal", "Acme refund requests are approved by the finance team."),
    ("a3", "acme", "public", "Acme shipping takes five business days."),
    ("b1", "bolt", "internal", "Bolt refund policy allows returns within ninety days."),
]
vec = TfidfVectorizer().fit([c[3] for c in CHUNKS])
MAT = vec.transform([c[3] for c in CHUNKS])
show = lambda title, obj: print(f"{title}\n  {json.dumps(obj)}")

# Orchestrator -> retrieval service
req = {"request_id": "r-1001", "deadline_ms": 800, "query": "acme refund policy", "tenant": "acme",
       "identity": {"user": "u-42", "roles": ["staff"], "levels": ["public", "internal"]},
       "top_k": 3, "filters": {}, "index_version": "idx-7"}
show("1. retrieval request", req)
sims = (MAT @ vec.transform([req["query"]]).T).toarray().ravel()
allowed = [i for i, c in enumerate(CHUNKS) if c[1] == req["tenant"] and c[2] in req["identity"]["levels"]]
ranked = sorted(allowed, key=lambda i: (-sims[i], i))[: req["top_k"]]
ret = {"request_id": req["request_id"], "index_version": "idx-7", "embed_version": "embedder-v1",
       "chunks": [{"chunk_id": CHUNKS[i][0], "doc_id": "doc-" + CHUNKS[i][0], "score": round(float(sims[i]), 2),
                   "access_label": CHUNKS[i][2], "text": CHUNKS[i][3]} for i in ranked if sims[i] > 0]}
show("2. retrieval response", {**ret, "chunks": [{k: v for k, v in c.items() if k != "text"} for c in ret["chunks"]]})

# Orchestrator -> reranking service: ids and text in, new scores out.
qwords = set(req["query"].split())
rr = [{"chunk_id": c["chunk_id"], "score": round(len(qwords & set(c["text"].lower().replace(".", "").split())) / len(qwords), 2)}
      for c in ret["chunks"]]
rr.sort(key=lambda x: (-x["score"], x["chunk_id"]))
show("3. rerank response", {"request_id": req["request_id"], "reranker_version": "rerank-v2", "scores": rr})

# Orchestrator -> generation service: only authorized chunks, chosen by the orchestrator.
best = next(c for c in ret["chunks"] if c["chunk_id"] == rr[0]["chunk_id"])
gen = {"request_id": req["request_id"], "model": "stub-generator", "prompt_version": "p-3",
       "answer": best["text"] + " [" + best["chunk_id"] + "]", "citations": [best["chunk_id"]],
       "tokens": {"in": 41, "out": 12}, "deadline_left_ms": 610}
show("4. generation response", gen)

Output:

1. retrieval request
  {"request_id": "r-1001", "deadline_ms": 800, "query": "acme refund policy", "tenant": "acme", "identity": {"user": "u-42", "roles": ["staff"], "levels": ["public", "internal"]}, "top_k": 3, "filters": {}, "index_version": "idx-7"}
2. retrieval response
  {"request_id": "r-1001", "index_version": "idx-7", "embed_version": "embedder-v1", "chunks": [{"chunk_id": "a1", "doc_id": "doc-a1", "score": 0.55, "access_label": "internal"}, {"chunk_id": "a2", "doc_id": "doc-a2", "score": 0.24, "access_label": "internal"}, {"chunk_id": "a3", "doc_id": "doc-a3", "score": 0.15, "access_label": "public"}]}
3. rerank response
  {"request_id": "r-1001", "reranker_version": "rerank-v2", "scores": [{"chunk_id": "a1", "score": 1.0}, {"chunk_id": "a2", "score": 0.67}, {"chunk_id": "a3", "score": 0.33}]}
4. generation response
  {"request_id": "r-1001", "model": "stub-generator", "prompt_version": "p-3", "answer": "Acme refund policy allows returns within thirty days. [a1]", "citations": ["a1"], "tokens": {"in": 41, "out": 12}, "deadline_left_ms": 610}

Reading it: the retrieval request carries the identity and tenant, so the access filter runs inside the index query and chunk b1, which belongs to another tenant, never appears. The retrieval response returns ids, scores and labels (the chunk text is hidden in this printout to keep it short; the real payload carries text or a pointer). The reranker returns only new scores for the same ids plus its version, and the generation response carries the answer, citations that name only supplied chunk ids, and the token counts and remaining time the orchestrator needs for the response.

Worked example

The program has two tenants with similar documents, a contract validator, an authorization-placement check, a cache-key check, and the batching arithmetic. The retriever is TF-IDF and the "generator" just returns the top chunk text; they stand in for the real components. The validator is a minimal illustration: it covers non-object input, the wrong types, an empty or oversized query, a tenant string outside a strict pattern (the regular expression [a-z0-9_-]{1,32} accepts only 1 to 32 characters, each a lower-case letter, digit, underscore or hyphen, which rules out path-like strings such as ../acme), and a boolean passed as an integer. A production validator must also handle unknown fields, Unicode normalization, size limits on whole payloads and schema versioning.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import math, re
from sklearn.feature_extraction.text import TfidfVectorizer

# Stand-ins: TF-IDF retriever, word-overlap "reranker", string-joining "generator".
CHUNKS = [  # (chunk_id, tenant, text)
    ("a1", "acme", "Acme refund policy allows returns within thirty days."),
    ("a2", "acme", "Acme refund requests are approved by the finance team."),
    ("a3", "acme", "Acme shipping takes five business days."),
    ("b1", "bolt", "Bolt refund policy allows returns within ninety days."),
    ("b2", "bolt", "Bolt refund requests are approved automatically."),
    ("b3", "bolt", "Bolt warranty covers two years."),
]
vec = TfidfVectorizer().fit([c[2] for c in CHUNKS])
MAT = vec.transform([c[2] for c in CHUNKS])

def validate(req):
    """Contract check for the retrieval service. Minimal: it covers the shapes listed in the tests below."""
    if not isinstance(req, dict):
        return None, "request must be an object"
    q, tenant, k = req.get("query"), req.get("tenant"), req.get("top_k", 3)
    if not isinstance(q, str) or not 0 < len(q.strip()) <= 500:
        return None, "query must be a non-empty string of at most 500 characters"
    if not isinstance(tenant, str) or re.fullmatch(r"[a-z0-9_-]{1,32}", tenant) is None:
        return None, "tenant must match [a-z0-9_-]{1,32}"
    if isinstance(k, bool) or not isinstance(k, int) or not 1 <= k <= 5:
        return None, "top_k must be an integer from 1 to 5"
    return {"query": q.strip().lower(), "tenant": tenant, "top_k": k}, None

def retrieve(req, prefilter=True, candidates=4):
    sims = (MAT @ vec.transform([req["query"]]).T).toarray().ravel()
    order = sorted(range(len(CHUNKS)), key=lambda i: (-sims[i], i))
    if prefilter:      # authorization inside the index query, before the cut-off
        order = [i for i in order if CHUNKS[i][1] == req["tenant"]]
        return [CHUNKS[i][0] for i in order[:candidates] if sims[i] > 0]
    top = order[:candidates]   # post-filter: cut first, check afterwards
    return [CHUNKS[i][0] for i in top if CHUNKS[i][1] == req["tenant"] and sims[i] > 0]

print("-- contract validation")
for bad in (None, 7, ["x"], {"query": "refund", "tenant": "acme", "top_k": "3"}, {"query": "", "tenant": "acme"},
            {"query": "x" * 501, "tenant": "acme"}, {"query": "refund", "tenant": "../acme"}, {"query": "refund", "tenant": "acme", "top_k": True},
            {"query": "refund policy", "tenant": "acme", "top_k": 2}):
    ok, err = validate(bad)
    print(f"{str(bad)[:58]:60s} -> {'OK ' + str(ok['top_k']) if ok else err}")

print("-- authorization placement (tenant bolt, query about refund)")
r = validate({"query": "acme refund policy", "tenant": "bolt"})[0]
print("pre-filter :", retrieve(r, True))
print("post-filter:", retrieve(r, False))

print("-- cache key")
cache_bad, cache_good = {}, {}
def answer(req, cache, key):
    if key in cache:
        return cache[key] + " (cache hit)"
    ids = retrieve(req)
    text = " | ".join(c[2] for c in CHUNKS if c[0] in ids[:1])
    cache[key] = text
    return text
qa = validate({"query": "acme refund policy", "tenant": "acme"})[0]
qb = validate({"query": "acme refund policy", "tenant": "bolt"})[0]
print("key=query only, acme :", answer(qa, cache_bad, qa["query"]))
print("key=query only, bolt :", answer(qb, cache_bad, qb["query"]))
print("key=(tenant,query), acme:", answer(qa, cache_good, (qa["tenant"], qa["query"], "idx-7")))
print("key=(tenant,query), bolt:", answer(qb, cache_good, (qb["tenant"], qb["query"], "idx-7")))

print("-- batching the reranker (model calls)")
requests, cands, max_batch = 8, 40, 64
per_request = requests * math.ceil(cands / max_batch)
pooled = math.ceil(requests * cands / max_batch)
print(f"pairs scored={requests * cands}; per-request batches={per_request} calls; pooled micro-batches={pooled} calls")

Output:

-- contract validation
None                                                         -> request must be an object
7                                                            -> request must be an object
['x']                                                        -> request must be an object
{'query': 'refund', 'tenant': 'acme', 'top_k': '3'}          -> top_k must be an integer from 1 to 5
{'query': '', 'tenant': 'acme'}                              -> query must be a non-empty string of at most 500 characters
{'query': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx   -> query must be a non-empty string of at most 500 characters
{'query': 'refund', 'tenant': '../acme'}                     -> tenant must match [a-z0-9_-]{1,32}
{'query': 'refund', 'tenant': 'acme', 'top_k': True}         -> top_k must be an integer from 1 to 5
{'query': 'refund policy', 'tenant': 'acme', 'top_k': 2}     -> OK 2
-- authorization placement (tenant bolt, query about refund)
pre-filter : ['b1', 'b2']
post-filter: ['b1']
-- cache key
key=query only, acme : Acme refund policy allows returns within thirty days.
key=query only, bolt : Acme refund policy allows returns within thirty days. (cache hit)
key=(tenant,query), acme: Acme refund policy allows returns within thirty days.
key=(tenant,query), bolt: Bolt refund policy allows returns within ninety days.
-- batching the reranker (model calls)
pairs scored=320; per-request batches=8 calls; pooled micro-batches=5 calls

Reading it: the validator returns an error for None, a number, a list, a string top_k, an empty query, a 501-character query, a path-like tenant and a boolean top_k (a Python True is an int, so the check excludes it explicitly), and accepts the one well-formed request. With the filter inside the index query, tenant bolt gets two of its own chunks (b1, b2); with the filter after the cut-off of four candidates, other tenants' chunks consume slots and bolt gets only b1. In the cache demo, a key made of the query alone served Acme's refund policy to Bolt (a cross-tenant leak); including the tenant and index version in the key gave each tenant its own answer. The batching line shows 320 pairs scored in 8 calls per-request or 5 calls pooled at a limit of 64 pairs per call.

Trade-offs and when not to split

• Each network hop adds latency and a failure mode. Three extra hops can be justified only when the parts scale differently (the reranker needs accelerators, retrieval needs memory, generation needs the most capacity) or are owned by different teams. A small system should start as modules inside one service with the same interfaces, and split later along those seams.
• Payload size: sending chunk text between services costs bandwidth; pass ids plus text only where the next stage needs it.
• Version skew: if the retriever and reranker are deployed independently, record the versions on every response and test the combinations.
• Failure handling: define per-stage fallbacks (skip rerank, smaller context, "I could not find an answer" instead of a guess) and test them.

Common mistakes

• Treating authorization as a front-door concern only.
• A cache key without tenant, permission scope or index version.
• Retrying on timeouts without a budget, which turns a slow dependency into an outage.
• Micro-batching windows longer than the latency saved.
• Letting the generation service accept chunk text from the client, which bypasses retrieval authorization.

Direct answer

A language model must have its evidence in the prompt before it can ground the words it writes, so the safe default is: retrieve and rerank first, then stream the generation, with the whole stream pinned to one version of the corpus. "Retrieving as more context arrives" is then handled as controlled extra rounds at sentence boundaries, not as a free-running race between retrieval and generation. Consistency comes from three mechanisms: a corpus snapshot id fixed at the start of the request, citations attached to each sentence and checked against that snapshot, and an event protocol that lets the client replace or retract text that fails the check. Latency comes from overlapping the work that has no dependency (embedding, keyword and vector search in parallel, prompt prefix caching, warm connections) and from stopping the rest at a deadline.

Terms

Streaming sends tokens (word pieces) to the client as the model produces them, so the user sees the start of the answer after the time to first token instead of waiting for the whole text. Reranking reorders retrieved candidates with a slower, more accurate model. Grounding means each claim is supported by a retrieved passage. A snapshot is a fixed version of the document index. An entailment model judges whether one text (the cited chunk) supports another (the generated sentence). A prompt prefix cache lets the model server reuse its work on the start of a prompt that is identical across requests. A speculative start begins generation from provisional results before the final ones are ready, and cancels it if they turn out to differ. Back-pressure means a slow consumer signals the producer to slow down or stop instead of letting data pile up without limit.

Pipeline, in order

1. Request start: authenticate, record the corpus snapshot version (an integer or timestamp the index reports), set a total deadline, and open the stream channel.
2. Retrieve: embed the question and run vector and keyword search in parallel, merge the two lists, apply the caller's access filter, and take about 20 to 50 candidates. Cache this result under (normalized query, snapshot version, permission scope) so a repeat question skips this step.
3. Rerank: score the candidates and keep a handful. Among the retrieval stages this is the slowest in the worked example (80 ms against 15 for embedding and 25 for search), while the model's own prompt processing is longer still (300 ms there) and can only be shortened with a shorter prompt. Reranking is therefore the step to trim first under a latency budget, by reranking fewer candidates or skipping it when the first-stage scores are decisive.
4. Assemble context and start streaming: build the prompt with numbered chunks and an instruction to cite them, then forward tokens as they arrive. Keep the static prefix of the prompt (instructions, tool descriptions) identical across requests so the model server's prompt cache can reuse it, where the server supports that.
5. Check as you stream: buffer tokens into sentences. Verify each sentence's citations against the pinned snapshot with a cheap check (cited chunk exists and was supplied; text overlap, or a small entailment model) and release it to the client only if it passes, or release it immediately and retract on failure. Holding each sentence adds the check time to every sentence; releasing immediately risks showing something that is later withdrawn. Choose by stakes: hold for compliance or medical answers, release and retract for casual assistants.
6. Finish: send the final citation list, the snapshot version, and a notice if cited documents changed after the snapshot.

Retrieving while generating

There are three patterns, in increasing order of risk:

• Retrieve first (default). Time to first token is the sum of embedding, search, rerank and the model's prompt processing.
• Speculative start. Begin generation from the first-stage results while reranking runs, and cancel and restart if the reranker's top results differ materially. It lowers the time to first token at the cost of wasted model work on a miss, so use it only when first-stage and reranked lists usually agree; measure that agreement in shadow traffic.
• Mid-stream retrieval. At a sentence boundary the generator may request more evidence (a follow-up query derived from what it just wrote, or because a claim lacked support). The stream pauses for one retrieval round, appends the new chunks to the context, and continues. Cap rounds per answer, since each adds a pause and tokens to the prompt, and do not let new chunks silently change statements already shown.

Partial answer updates

Define a small event protocol between server and client, for example token, sentence_done(citations), retract(sentence_id, reason), replace(sentence_id, text), notice(text) and done(snapshot_version). The client marks unverified text visually and finalizes it on sentence_done. Avoid rewriting text the user has read unless it is wrong, and when you must, say so in the interface.

Caching

• Retrieval results: key = normalized query, snapshot version, permission scope, pipeline version.
• Embeddings of frequent queries: key = text and embedding model version.
• Prompt prefix cache at the model server: stable instructions first, variable evidence last.
• Whole answers only for questions where staleness is acceptable, keyed additionally by prompt and model version, and invalidated when any cited chunk changes.

Corpus updates in flight

Pin the stream to the snapshot taken at the start: all retrieval rounds in that stream read the same version, so the evidence cannot change under the sentences already written. New or changed documents appear on the next request. At the end, compare the cited chunks' current versions with the pinned ones and, if any changed, send a notice. Deletions and access revocations are the exception: re-check authorization at each sentence boundary and stop the stream (or retract) if the caller lost access or a cited document was withdrawn for legal reasons, because staleness is not acceptable there.

Worked example

Part A is arithmetic on assumed stage durations in milliseconds (15 for embedding, 25 for search, 80 for reranking, 300 from prompt to first token, 40 per generated token, 120 tokens); they are inputs for illustration, not measurements of any system. Part B runs a scripted four-sentence stream. The generator is a scripted stand-in, and the sentence check is a word-overlap stand-in for a real entailment or citation checker (a sentence passes when at least 80 percent of its content words appear in its cited chunk). Chunk c3 is edited after sentence 2, once with the stream pinned to the starting snapshot and once without. Sentence 4 is deliberately unsupported by the chunk it cites.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import re

# Part A: time to first token under three orderings. All durations are ASSUMED values in milliseconds.
EMBED, ANN, RERANK, PROMPT_TO_FIRST_TOKEN, PER_TOKEN, TOKENS = 15, 25, 80, 300, 40, 120
sequential = EMBED + ANN + RERANK + PROMPT_TO_FIRST_TOKEN
speculative = EMBED + ANN + PROMPT_TO_FIRST_TOKEN          # start generating from the unreranked top results
total_seq = sequential + (TOKENS - 1) * PER_TOKEN
midstream_pause = EMBED + ANN + RERANK                      # one extra retrieval round at a sentence boundary
total_mid = total_seq + midstream_pause
print(f"first token: retrieve-rerank-then-generate={sequential} ms, speculative start={speculative} ms")
print(f"full answer: one retrieval={total_seq} ms, one extra mid-stream retrieval round={total_mid} ms")

# Part B: a stream pinned to the corpus snapshot it started on. Stand-ins: scripted generator, word-overlap verifier.
class Store:
    def __init__(self):
        self.version = 1
        self.history = {1: {"c1": "The standard plan costs 20 dollars per month.",
                            "c2": "The standard plan includes five seats.",
                            "c3": "Annual billing gives two months free."}}
    def update(self, chunk_id, text):
        snap = dict(self.history[self.version])
        snap[chunk_id] = text
        self.version += 1
        self.history[self.version] = snap
    def read(self, chunk_id, version):
        return self.history[version][chunk_id]

STOP = {"the", "a", "an", "it", "is", "per", "of", "to"}
def words(t): return {w for w in re.findall(r"[a-z0-9]+", t.lower()) if w not in STOP}
def supported(sentence, chunk_text, threshold=0.8):
    s = words(sentence)
    return len(s & words(chunk_text)) / len(s) >= threshold

SCRIPT = [("The standard plan costs 20 dollars per month.", "c1"),
          ("It includes five seats.", "c2"),
          ("Annual billing gives two months free.", "c3"),
          ("Shipping is free worldwide.", "c2")]

def stream(store, pin, update_after=2):
    pinned = store.version if pin else None
    events = []
    for n, (sentence, cid) in enumerate(SCRIPT, 1):
        if n == update_after + 1:
            store.update("c3", "Annual billing gives one month free.")   # corpus changes mid-stream
        ver = pinned if pin else store.version
        ok = supported(sentence, store.read(cid, ver))
        events.append(f"  sentence {n} [{cid}@v{ver}] {'emit   ' if ok else 'RETRACT'} {sentence}")
    if pin and store.version != pinned:
        changed = sorted({c for _, c in SCRIPT if store.read(c, pinned) != store.read(c, store.version)})
        events.append(f"  notice: corpus moved v{pinned} -> v{store.version} during the answer; cited chunks changed: {', '.join(changed)}")
    return events

for pin in (True, False):
    print(f"pinned snapshot = {pin}")
    print("\n".join(stream(Store(), pin)))

Output:

first token: retrieve-rerank-then-generate=420 ms, speculative start=340 ms
full answer: one retrieval=5180 ms, one extra mid-stream retrieval round=5300 ms
pinned snapshot = True
  sentence 1 [c1@v1] emit    The standard plan costs 20 dollars per month.
  sentence 2 [c2@v1] emit    It includes five seats.
  sentence 3 [c3@v1] emit    Annual billing gives two months free.
  sentence 4 [c2@v1] RETRACT Shipping is free worldwide.
  notice: corpus moved v1 -> v2 during the answer; cited chunks changed: c3
pinned snapshot = False
  sentence 1 [c1@v1] emit    The standard plan costs 20 dollars per month.
  sentence 2 [c2@v1] emit    It includes five seats.
  sentence 3 [c3@v2] RETRACT Annual billing gives two months free.
  sentence 4 [c2@v2] RETRACT Shipping is free worldwide.

Reading it: skipping the wait for the reranker brings the first token from 420 ms to 340 ms (both assumed), and one extra mid-stream retrieval round adds 120 ms to a 5,180 ms answer. In the pinned run, sentences 1 to 3 pass against version 1, sentence 4 fails the support check and is retracted, and the closing notice names c3 as changed so the client can offer a refresh. In the unpinned run, the edit makes sentence 3, already generated from the old text, fail against version 2, and sentence 4 is judged against version 2 as well: the answer's meaning shifts while the user is reading it. The pinned version is predictable and the unpinned one is fresher but incoherent within one answer.

Which rule triggers each retraction, and what the client receives

The next program prints the support score behind each decision and then the event sequence of the pinned run, using the same word-overlap stand-in. A content word is any word not in the small stop list (the, a, per, and so on). The 0.8 threshold is an assumed value; in practice you pick it by scoring a labelled set of supported and unsupported sentences and choosing the value that separates them, since a strict threshold retracts honest paraphrases and a loose one lets mismatches through.

import re
STOP = {"the", "a", "an", "it", "is", "per", "of", "to"}
def words(t): return {w for w in re.findall(r"[a-z0-9]+", t.lower()) if w not in STOP}
cases = [("Annual billing gives two months free.", "c3 at version 1", "Annual billing gives two months free."),
         ("Annual billing gives two months free.", "c3 at version 2", "Annual billing gives one month free."),
         ("Shipping is free worldwide.", "c2 at version 1", "The standard plan includes five seats.")]
for sentence, label, chunk in cases:
    s, c = words(sentence), words(chunk)
    print(f"{sentence!r} against {label}: content words {len(s)}, found {len(s & c)}, share {len(s & c) / len(s):.2f}, "
          f"{'pass' if len(s & c) / len(s) >= 0.8 else 'fail'} at the 0.8 threshold")
chunks_v1 = {"c1": "The standard plan costs 20 dollars per month.", "c2": "The standard plan includes five seats.",
             "c3": "Annual billing gives two months free."}
script = [("The standard plan costs 20 dollars per month.", "c1"), ("It includes five seats.", "c2"),
          ("Annual billing gives two months free.", "c3"), ("Shipping is free worldwide.", "c2")]
print("events the client receives in the pinned run:")
for n, (sentence, cid) in enumerate(script, 1):
    print(f"  token x{len(sentence.split())} for sentence {n}")
    s, c = words(sentence), words(chunks_v1[cid])
    if len(s & c) / len(s) >= 0.8:
        print(f"  sentence_done({n}, citations=[{cid}])")
    else:
        print(f"  retract({n}, reason='not supported by {cid}')")
print("  notice('cited chunk c3 changed while the reply was streaming')")
print("  done(snapshot_version=1)")

Output:

'Annual billing gives two months free.' against c3 at version 1: content words 6, found 6, share 1.00, pass at the 0.8 threshold
'Annual billing gives two months free.' against c3 at version 2: content words 6, found 4, share 0.67, fail at the 0.8 threshold
'Shipping is free worldwide.' against c2 at version 1: content words 3, found 0, share 0.00, fail at the 0.8 threshold
events the client receives in the pinned run:
  token x8 for sentence 1
  sentence_done(1, citations=[c1])
  token x4 for sentence 2
  sentence_done(2, citations=[c2])
  token x6 for sentence 3
  sentence_done(3, citations=[c3])
  token x4 for sentence 4
  retract(4, reason='not supported by c2')
  notice('cited chunk c3 changed while the reply was streaming')
  done(snapshot_version=1)

Reading it: sentence 4 is retracted because none of its three content words appear in the chunk it cites (share 0.00), whatever the snapshot. Sentence 3 passes against version 1 (all 6 words found) and fails against version 2, where only 4 of 6 are found (0.67), which is why it is retracted only in the unpinned run. In the event list, sentence_done finalizes a verified sentence, retract withdraws a sentence the client already displayed, and the notice and done events close the stream with the snapshot version.

Trade-offs and failure modes

• Time to first token versus verification. Holding sentences until verified delays text by the checker time; unverified streaming is faster but exposes retractions.
• Speculation wastes work when the lists disagree, and mid-stream retrieval makes latency unpredictable; both need a deadline and a cap.
• Cancellation: when the client disconnects, cancel retrieval and generation promptly, otherwise abandoned requests consume capacity.
• Back-pressure: a slow client must not stall the pipeline; buffer a bounded amount and drop the stream if the buffer fills.
• Observability: log per-stage timings, snapshot version, retracted sentence count and cache hits per request, and alarm on retraction rate.
• Common mistakes: letting each retrieval round read the live index; streaming without a way to retract; running retrieval concurrently with generation so the prompt lacks the evidence the first sentences need; caching answers without the snapshot version in the key.

Direct answer

Four index families cover most vector search at scale. In practice flat (small corpora, and the exact baseline everything else is measured against) and HNSW (when the full vectors fit in memory) are the usual first choices; IVF cuts the work per query once a full scan is too slow (it still stores every full vector, so it does not shrink memory), and IVF+PQ is what a growing memory bill pushes you toward. Recall@10 below means the fraction of the true 10 nearest neighbours, found by exact search, that the index returns.

• Flat: store every vector and compare the query with all of them. Exact, no build step, but cost per query grows linearly with the corpus.
• IVF (inverted file): cluster the vectors with k-means (repeatedly assign each vector to its nearest centre, then move each centre to the mean of its vectors) into nlist cells, each represented by its centre, called a centroid, and at query time scan only the nprobe cells whose centroids are closest to the query. Fewer vectors scored, at the risk of missing a neighbour that sits in an unprobed cell.
• HNSW (hierarchical navigable small world): a layered neighbour graph searched greedily. Very good recall for low latency, but the vectors stay in memory in full precision and the graph adds links.
• IVF+PQ: IVF with product quantisation (PQ), which splits each vector into $m$ pieces and stores, for each piece, only the id of its nearest centroid in a small codebook (the list of centroids for that piece, learned by k-means; with 256 centroids an id fits in one byte). A vector shrinks from $4d$ bytes to about $m$ bytes, at the cost of approximate distances, usually repaired by re-scoring a short list with the original vectors.

The decision is driven by corpus size against the memory budget, the recall target, the latency target and how often the data changes.

Product quantisation on one small vector

Take an 8-number vector, cut it into 2 pieces of 4 numbers, and give each piece a codebook of 4 centroids, so each piece is replaced by a 2-bit id. The program finds the nearest centroid per piece, shows what the stored code decodes back to, and then does the lookup-table distance used at query time (asymmetric distance: the query stays exact and only the stored vectors are compressed). The last lines are the byte arithmetic that turns a 768-number vector into 64 one-byte codes, using $768/64=12$ numbers per piece.

import numpy as np

# A hand-sized product quantiser: an 8-number vector, 2 pieces of 4 numbers, 4 centroids per piece (a 2-bit code each).
book0 = np.array([[0, 0, 0, 0], [1, 2, 0, 3], [3, 1, 1, 1], [4, 4, 4, 4]], float)   # codebook for piece 0
book1 = np.array([[0, 0, 0, 0], [4, 1, 2, 0], [2, 2, 2, 2], [1, 3, 0, 1]], float)   # codebook for piece 1
books = [book0, book1]
v = np.array([1.0, 2.0, 0.5, 3.0, 4.0, 1.0, 2.5, 0.0])
pieces = [v[:4], v[4:]]
code = [int(((b - p) ** 2).sum(axis=1).argmin()) for b, p in zip(books, pieces)]
decoded = np.concatenate([b[c] for b, c in zip(books, code)])
print("pieces:", [p.tolist() for p in pieces])
print("code (nearest centroid id per piece):", code)
print("decoded vector:", decoded.tolist())
print("reconstruction error (squared distance, original vs decoded):", float(((v - decoded) ** 2).sum()))

# Asymmetric distance: the query stays exact; one small table per piece, then a lookup per stored code.
q = np.array([1.0, 2.0, 1.0, 3.0, 3.0, 1.0, 2.0, 1.0])
table = np.stack([((b - q[4 * j:4 * j + 4]) ** 2).sum(axis=1) for j, b in enumerate(books)])   # shape (2, 4)
print("lookup table rows (squared distance from query piece to each centroid):")
print(table.tolist())
print("approx squared distance = table[0][code0] + table[1][code1] =", float(table[0][code[0]] + table[1][code[1]]))
print("same value computed against the decoded vector:", float(((q - decoded) ** 2).sum()), "| exact against the original:", float(((q - v) ** 2).sum()))

# Why the memory shrinks: bytes per vector at 768 dimensions, and the 10M-vector budget in the table above.
d, N, GiB = 768, 10_000_000, 1024 ** 3
print("float32 vector bytes:", d * 4, "| 64 one-byte codes:", 64, "| ratio:", d * 4 // 64)
print("10M x (64 code bytes + 8 id bytes) =", N * 72, "bytes =", round(N * 72 / GiB, 3), "GiB")
print("65,536 centroids x 768 x 4 bytes =", round(65536 * d * 4 / GiB, 3), "GiB | average cell size =", round(N / 65536, 1), "vectors")
print("64 codebooks x 256 centres x 12 dims x 4 bytes =", round(64 * 256 * 12 * 4 / GiB, 4), "GiB")
print("100M float32 vectors of 768 dims =", round(100_000_000 * d * 4 / GiB, 1), "GiB | 300,000 vectors scored per flat query =", 300_000 * d, "multiply-adds")

Output:

pieces: [[1.0, 2.0, 0.5, 3.0], [4.0, 1.0, 2.5, 0.0]]
code (nearest centroid id per piece): [1, 1]
decoded vector: [1.0, 2.0, 0.0, 3.0, 4.0, 1.0, 2.0, 0.0]
reconstruction error (squared distance, original vs decoded): 0.5
lookup table rows (squared distance from query piece to each centroid):
[[15.0, 1.0, 9.0, 23.0], [15.0, 2.0, 3.0, 12.0]]
approx squared distance = table[0][code0] + table[1][code1] = 3.0
same value computed against the decoded vector: 3.0 | exact against the original: 2.5
float32 vector bytes: 3072 | 64 one-byte codes: 64 | ratio: 48
10M x (64 code bytes + 8 id bytes) = 720000000 bytes = 0.671 GiB
65,536 centroids x 768 x 4 bytes = 0.188 GiB | average cell size = 152.6 vectors
64 codebooks x 256 centres x 12 dims x 4 bytes = 0.0007 GiB
100M float32 vectors of 768 dims = 286.1 GiB | 300,000 vectors scored per flat query = 230400000 multiply-adds

Reading it:

• The vector [1, 2, 0.5, 3, 4, 1, 2.5, 0] is stored as the pair of ids [1, 1]. Decoding gives [1, 2, 0, 3, 4, 1, 2, 0]; the reconstruction error (how far the decoded vector is from the original, as a squared distance) is 0.5 (each of the third and seventh numbers moved by 0.5, and 0.25 + 0.25 = 0.5).
• The lookup table has one row per piece and one entry per centroid: entry [j][c] is the squared distance from the query's piece $j$ to centroid $c$. A stored vector's distance is then just two table lookups added together, with no arithmetic on the original numbers. In the larger program below, the tab = np.stack(...) line builds exactly this table (shape $(m, ks)$, where ks is the codebook size) and tab[np.arange(m), codes[ids]].sum(axis=1) does the lookups for all candidates at once.
• The approximate distance (3.0) matches the distance to the decoded vector exactly and differs from the true distance (2.5) by the compression error. That gap is why a short list is usually re-scored with the original vectors.
• At 768 dimensions, 64 codes take 64 bytes against 3072 bytes for the float32 vector, 48 times fewer. The memory section below is this multiplication at 10 million vectors.

The computed comparison

The program below builds a small IVF and a small PQ from scratch with scikit-learn 1.5.2 k-means (a stand-in for production libraries), on 6000 points in 32 dimensions drawn around 40 centres, and reports recall@10 against exact search together with the number of vectors scored per query, which is the cost measure (counted, not timed). This stand-in applies PQ to the raw vectors; FAISS's IndexIVFPQ by default encodes each vector's residual (the vector minus its cell's centroid) instead, which this program does not do. The PQ here uses 8 pieces of 4 dimensions with 64 centroids each, to suit the small data (6000 points give about 94 points per centroid at 64 centroids, but only about 23 at 256); production codebooks commonly have 256 centroids per piece so that a code is one byte. The IVF here uses nlist = 64 cells, so an average cell holds about 94 vectors. The last block is memory arithmetic for 10 million 768-dimensional float32 vectors with an assumed 8-byte id per vector and an assumed nlist of 65,536 cells (about 153 vectors per cell on average; the right value depends on the data and is tuned). One GiB is $1024^3$ bytes.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import numpy as np
from sklearn.cluster import KMeans

rng = np.random.default_rng(0)
n, d, nq = 6000, 32, 100
centers = rng.normal(size=(40, d)) * 1.5
X = (centers[rng.integers(0, 40, n)] + rng.normal(size=(n, d))).astype(np.float32)
Q = (centers[rng.integers(0, 40, nq)] + rng.normal(size=(nq, d))).astype(np.float32)
d2 = lambda A, b: ((A - b) ** 2).sum(axis=1)
truth = [set(np.argsort(d2(X, q))[:10]) for q in Q]
recall = lambda found: np.mean([len(set(f) & t) / 10 for f, t in zip(found, truth)])

# ---- IVF: coarse k-means into nlist cells; search only the nprobe closest cells ----
nlist = 64
km = KMeans(nlist, n_init=1, random_state=0).fit(X)
cells = [np.where(km.labels_ == c)[0] for c in range(nlist)]
cent = km.cluster_centers_

# ---- PQ: split d=32 into m=8 sub-vectors of 4 dims; each sub-vector is replaced by the id of its nearest of 64 centroids ----
m, ks = 8, 64
sub = d // m
books = [KMeans(ks, n_init=1, random_state=0).fit(X[:, j * sub:(j + 1) * sub]) for j in range(m)]
codes = np.stack([b.labels_ for b in books], axis=1)                  # (n, m) small integers
decoded = np.concatenate([books[j].cluster_centers_[codes[:, j]] for j in range(m)], axis=1)
print("PQ reconstruction error (mean squared, per vector):", round(float(((X - decoded) ** 2).sum(axis=1).mean()), 3),
      "| total variance of the data (sum over dims):", round(float(X.var(axis=0).sum()), 3))

def ivf_search(q, nprobe, use_pq=False, rerank=0):
    probe = np.argsort(d2(cent, q))[:nprobe]
    ids = np.concatenate([cells[c] for c in probe])
    if not use_pq:
        dist = d2(X[ids], q)
    else:                                                              # asymmetric distance: exact query vs coded vectors
        tab = np.stack([((books[j].cluster_centers_ - q[j * sub:(j + 1) * sub]) ** 2).sum(axis=1) for j in range(m)])  # (m, ks)
        dist = tab[np.arange(m), codes[ids]].sum(axis=1)
    order = ids[np.argsort(dist, kind="stable")]
    if use_pq and rerank:
        top = order[:rerank]; order = top[np.argsort(d2(X[top], q), kind="stable")]
    return order[:10], len(ids)

print("flat (exact)        : recall@10 = 1.000, vectors scored per query =", n)
for nprobe in (1, 4, 16):
    res = [ivf_search(q, nprobe) for q in Q]
    print(f"IVF-flat nprobe={nprobe:2d}: recall@10 = {recall([r[0] for r in res]):.3f}, vectors scored per query = {np.mean([r[1] for r in res]):.0f}")
for rr in (0, 50):
    res = [ivf_search(q, 16, use_pq=True, rerank=rr) for q in Q]
    print(f"IVF-PQ nprobe=16 rerank={rr:2d}: recall@10 = {recall([r[0] for r in res]):.3f}, codes scored per query = {np.mean([r[1] for r in res]):.0f}")

# ---- memory arithmetic at production scale (10M vectors, 768 dims) ----
N, D, GiB = 10_000_000, 768, 1024 ** 3
flat = N * D * 4
print(f"flat float32 : {flat / GiB:6.1f} GiB")
print(f"IVF-flat     : {(flat + N * 8 + 65536 * D * 4) / GiB:6.1f} GiB  (+ 8-byte id per vector + 65,536 centroids)")
for mm in (64, 96):
    print(f"IVF-PQ m={mm:3d}: {(N * (mm + 8) + 65536 * D * 4 + mm * 256 * (D // mm) * 4) / GiB:6.1f} GiB  (m one-byte codes + 8-byte id; centroids; codebooks of 256 centres)")
print(f"HNSW M=16    : {(flat + N * 2 * 16 * 4) / GiB:6.1f} GiB  (vectors + about 2M four-byte links per point at the bottom layer; upper layers add little)")

Output:

PQ reconstruction error (mean squared, per vector): 14.372 | total variance of the data (sum over dims): 99.252
flat (exact)        : recall@10 = 1.000, vectors scored per query = 6000
IVF-flat nprobe= 1: recall@10 = 0.836, vectors scored per query = 116
IVF-flat nprobe= 4: recall@10 = 1.000, vectors scored per query = 420
IVF-flat nprobe=16: recall@10 = 1.000, vectors scored per query = 1633
IVF-PQ nprobe=16 rerank= 0: recall@10 = 0.410, codes scored per query = 1633
IVF-PQ nprobe=16 rerank=50: recall@10 = 0.900, codes scored per query = 1633
flat float32 :   28.6 GiB
IVF-flat     :   28.9 GiB  (+ 8-byte id per vector + 65,536 centroids)
IVF-PQ m= 64:    0.9 GiB  (m one-byte codes + 8-byte id; centroids; codebooks of 256 centres)
IVF-PQ m= 96:    1.2 GiB  (m one-byte codes + 8-byte id; centroids; codebooks of 256 centres)
HNSW M=16    :   29.8 GiB  (vectors + about 2M four-byte links per point at the bottom layer; upper layers add little)

Reading the output:

• Flat scores all 6000 vectors per query and is exact.
• IVF-flat at nprobe=1 scores 116 vectors per query with recall@10 of 0.836; at 4 probes it scores 420 vectors with recall 1.000 on this data. The probe count is the recall dial, and the cost of the dial is the number of cells scanned.
• IVF-PQ scores the same 1633 candidates at 16 probes but with approximate distances: recall@10 is 0.410 without a re-score, and 0.900 once the best 50 are re-scored with the original vectors. The PQ reconstruction error is 14.372 against a total data variance (the summed spread of the data around its mean, over all dimensions) of 99.252, so about 14.5% of the spread is lost in compression. On this noisy toy the quantisation is coarse; real embedding data and more centroids behave differently, so treat the direction (compression lowers raw recall, re-scoring repairs it) rather than the values.
• Memory at 10 million vectors of 768 dimensions: flat 28.6 GiB, IVF-flat 28.9 GiB, HNSW with $M=16$ about 29.8 GiB, and IVF-PQ with 64 or 96 one-byte codes 0.9 or 1.2 GiB. The PQ index is more than 20 times smaller; that is its reason to exist. By hand for 64 codes: 10 million times (64 code bytes + 8 id bytes) is 720,000,000 bytes, 0.67 GiB; the 65,536 centroids at 768 floats of 4 bytes each add 0.19 GiB and the codebooks about 0.001 GiB, giving the printed 0.9 GiB. Keeping the original vectors for the re-score step (on disk or in another tier) is extra.

Which to use

For a read-heavy index that rarely changes, build cost is paid once, so choose by memory budget: HNSW if the full vectors fit in memory and low latency is the priority (accepting the long build and the extra link memory), IVF-PQ with a re-score if they do not.

For an index that changes often: HNSW accepts inserts incrementally, while deleting is commonly handled by marking points as removed and rebuilding periodically, because removing a node leaves its neighbours' links broken. IVF accepts inserts by assigning each new vector to its nearest cell and encoding it with the existing codebooks; if the data distribution shifts, centroid quality degrades and recall falls until you retrain and re-assign. Flat has no such issue. Plan a rebuild schedule and monitor recall against an exact-search sample.

Pitfalls

• Reporting recall without the latency or memory beside it, or comparing indexes at different recall levels.
• Training IVF or PQ on a sample that does not represent the data you will insert.
• Forgetting that a metadata filter applied after the search can leave fewer than $k$ results, in any of these families.
• Choosing by habit: measure recall@k against exact search on real queries before committing.

Direct answer

Enforce access control inside retrieval, before any text reaches the language model: authenticate the user once, resolve their groups and clearances into a filter, apply that filter as part of the vector search itself (vector search finds the stored vectors nearest to the query's vector; applying the filter inside that search, before results are cut to the top k, is a pre-filter), and log what was retrieved and shown. The model must never be the thing that decides what a user may see, because anything placed in its prompt can be quoted in its answer. Latency stays low because the identity lookup is cached for a short time and the filter is a metadata predicate evaluated inside the index, not a second pass over results.

The request path, in order

1. Authenticate. Verify the caller's token at the edge (who is this, signed and unexpired). Do not accept a user id or a list of groups from the browser body.
2. Authorize, per request. Resolve the user into the set of groups and clearance level the document ACLs (access-control lists: per-document records of who may read it) are written in. Cache this resolution for a short, bounded time (for example 60 seconds, an illustrative value) so group changes take effect quickly; the cache time is the revocation delay and should be stated.
3. Build the filter and pass it to the search. Every chunk is stored with its ACL as metadata copied from the source document at ingestion. The query carries a predicate such as "chunk ACL intersects user groups AND classification <= user clearance". The search returns only chunks that pass it.
4. Assemble context only from filtered results. Reranking, query rewriting, caching and any follow-up fetch use the same filter. A shared answer cache must be keyed by the access scope too, otherwise one user's answer is served to another.
5. Check the output. Every citation in the answer must be one of the chunks given to the model on this request. An answer citing something else is withheld or repaired. This catches the model naming or quoting material that came from somewhere other than the permitted context, for example from a stale conversation memory.
6. Audit. Write an append-only record per request: request id, user, the groups they were resolved to, the chunk ids given to the model, the chunk ids cited, and any blocked citation. Store ids, not text, so the audit log is not a second copy of the sensitive corpus. Write it asynchronously off the critical path (a queue), with a guarantee that a failed write alarms rather than silently drops.

Why pre-filtering rather than filtering the results afterwards

The program below uses random unit vectors as stand-ins for embeddings (a real system would use an embedding model) and brute-force similarity search. The corpus has 2,000 chunks, each readable by one or two of six groups; each of 300 simulated users belongs to two groups. The point is the selectivity arithmetic, which does not depend on the embedding model. Selectivity here means the share of the corpus a user's filter lets through: a user who may read 47 percent of the chunks has a selectivity of 0.47, and a user who may read 2 percent has a narrow filter.

How to read the code: V holds the 2,000 stand-in vectors and acl[i] is the set of groups allowed to read chunk i. top(qv, allowed_idx, k) scores only the listed chunks against the query vector and returns the best k. In each of the 300 trials a random user with two groups is drawn, ok lists the chunks that user may read, and truth is the best 10 among those, the exact permitted answer. plain is the best 10 over everything, with no access control. Then three strategies are compared against truth: filter plain afterwards (post_ret counts how many results survive, post_recall how many of the true top 10 are among them), fetch 50 then filter (over_ret, over_recall), and search only ok (pre_ret). The lists are averaged at the end to print one line each.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import json
import numpy as np

rng = np.random.default_rng(0)
N, D, K = 2000, 32, 10
V = rng.normal(size=(N, D)); V /= np.linalg.norm(V, axis=1, keepdims=True)   # stand-in embeddings (random unit vectors)
groups = ["hr", "legal", "fin", "eng", "exec", "sales"]
# each chunk is readable by 1 or 2 groups (its access-control list)
acl = [set(rng.choice(groups, size=rng.integers(1, 3), replace=False)) for _ in range(N)]

def top(qv, allowed_idx, k):
    idx = np.array(allowed_idx)
    sc = V[idx] @ qv
    return idx[np.argsort(-sc)[:k]].tolist()

n_trials = 300
share_ok, unauth_share, post_ret, post_recall, over_ret, over_recall, pre_ret = [], [], [], [], [], [], []
for _ in range(n_trials):
    user = set(rng.choice(groups, size=2, replace=False))
    ok = [i for i in range(N) if acl[i] & user]
    share_ok.append(len(ok) / N)
    qv = rng.normal(size=D); qv /= np.linalg.norm(qv)
    truth = top(qv, ok, K)                              # exact answer restricted to what the user may see
    plain = top(qv, range(N), K)                         # no access control at all
    unauth_share.append(np.mean([not (acl[i] & user) for i in plain]))
    post = [i for i in plain if acl[i] & user]           # filter AFTER retrieving k
    post_ret.append(len(post)); post_recall.append(len(set(post) & set(truth)) / K)
    over = [i for i in top(qv, range(N), 5 * K) if acl[i] & user][:K]   # fetch 5k, then filter
    over_ret.append(len(over)); over_recall.append(len(set(over) & set(truth)) / K)
    pre_ret.append(len(top(qv, ok, K)))                  # filter INSIDE the search

print(f"mean share of chunks a user may read: {np.mean(share_ok):.2f}")
print(f"no filter: unauthorized share of top-{K}: {np.mean(unauth_share):.2f}")
print(f"post-filter k={K}:   results {np.mean(post_ret):.2f}, recall vs exact allowed top-{K} {np.mean(post_recall):.2f}")
print(f"candidates needed on average to find {K} allowed chunks: {K / np.mean(share_ok):.1f}")
print(f"overfetch 5x:       results {np.mean(over_ret):.2f}, recall vs exact allowed top-{K} {np.mean(over_recall):.2f}")
print(f"pre-filter:         results {np.mean(pre_ret):.2f}, recall 1.00 by construction")

# Per-request enforcement, citation check and audit record (illustration with fixed values)
user_groups = {"legal", "fin"}
allowed_ids = [i for i in range(N) if acl[i] & user_groups]
shown = top(V[7], allowed_ids, 3)
forbidden = next(i for i in range(N) if i not in set(allowed_ids))
cited_by_model = shown[:2] + [forbidden]                 # model cites a chunk it was never given
violations = [c for c in cited_by_model if c not in set(shown)]
audit = dict(request="r-001", user="u-42", groups=sorted(user_groups), chunks_given=shown, cited=cited_by_model, blocked_citations=violations)
print(json.dumps(audit))

Output:

mean share of chunks a user may read: 0.47
no filter: unauthorized share of top-10: 0.55
post-filter k=10:   results 4.48, recall vs exact allowed top-10 0.45
candidates needed on average to find 10 allowed chunks: 21.5
overfetch 5x:       results 10.00, recall vs exact allowed top-10 1.00
pre-filter:         results 10.00, recall 1.00 by construction
{"request": "r-001", "user": "u-42", "groups": ["fin", "legal"], "chunks_given": [7, 257, 329], "cited": [7, 257, 0], "blocked_citations": [0]}

Reading it. With no access control, 0.55 of the top ten are chunks the user may not read, and those would go straight into the prompt. Filtering after retrieving ten leaves on average 4.48 results and only 0.45 of the exact permitted top ten, because most of the ten slots were wasted on forbidden chunks. Overfetching (retrieve 50, then filter) recovers the full list here, because this average user may read about 0.47 of the corpus, so finding ten permitted chunks takes about 10 / 0.47 = 21 candidates (the printed 21.5 divides by the unrounded share), and 50 candidates is comfortably more than that. The overfetch factor is a guess about selectivity: a user who may read a small share of the corpus needs about K divided by that share candidates (here K = 10), so the fetch size has to grow as the share shrinks, and the cost and the recall both degrade as selectivity falls. A pre-filter does not depend on that guess. The last line is the per-request audit record, and it shows the output check firing: the model cited chunk 0, which was never given to it, and the record lists it under blocked_citations.

Latency

• The authorization lookup is one cached call; the filter is a predicate inside the index, whose cost depends on the engine and on how selective the predicate is. Measure p95 (the latency that 95 percent of requests beat) with your real ACL distribution, because narrow filters are the slow case for approximate indexes. An approximate index finds near neighbours without scanning every vector, typically by walking from one nearby vector to the next; with a filter, it discards each visited vector the user may not read, so when the filter lets through only a few percent of the corpus most visited vectors are discarded and the search has to visit many more of them (or return fewer than k results) to find enough permitted neighbours. How an engine handles this differs, so test it.
• If one tenant or group is tiny, give it its own partition (a physically separate slice of the index holding only that group's chunks) or index; the filter then costs nothing and isolation is stronger.
• Audit logging and the output check are the only additions after generation; keep them cheap (set membership on ids) and the log write asynchronous.

Trade-offs and failure modes

• Stale permissions. Revoking access in the source system must reach the index. Either copy ACL changes through the same incremental update path (with a short lag target and an alarm) or re-check authorization against the source of truth for the top results at query time, which costs a call per request.
• Leakage during generation. The guarantee comes from what is in the prompt: if only permitted chunks are there, the model has nothing restricted to quote. The citation check above is a backstop that catches references to chunks that were not supplied; it does not prove that every sentence is grounded in permitted text, so keep restricted material out of model weights (no fine-tuning on it) and out of any memory shared between users.
• Derived data. Summaries, caches, embeddings of conversation history and fine-tuned weights are copies of documents. Each must inherit the ACL of its inputs or not exist.
• ACL explosion. Per-user ACLs on millions of chunks are expensive; prefer groups and roles, and a classification level, as the stored attribute.
• Common mistakes: trusting groups sent by the client; filtering after retrieval and calling it done; one shared cache across users; logging full chunk text into the audit trail.

Direct answer

Retrieval-augmented generation (RAG) is a pattern where, before the language model answers, the system looks up relevant passages in an external knowledge source and puts them into the prompt. The model then writes its answer from that supplied evidence instead of from memory alone. This reduces hallucination (a fluent statement that nothing supports) because the answer is tied to text that can be shown, cited and checked, and because the system can refuse to answer when nothing relevant is found.

A plain-language picture: a standalone LLM takes a closed-book exam from whatever it absorbed during training. RAG turns it into an open-book exam where the book is your own, current, searchable document set.

The five core components

1. Ingestion and chunking. Source documents are cleaned and split into chunks (passages of a few hundred tokens, where a token is the sub-word unit a model reads).
2. Index. Each chunk is stored so it can be searched: as term statistics for keyword search (BM25 is a keyword scoring formula that rewards rare matching words), as an embedding (a list of numbers that represents meaning) in a vector index, or both.
3. Retriever. At question time it turns the question into the same representation and returns the top-k most similar chunks (k is how many to return, a setting such as 5). An optional reranker (a slower model that re-scores those k candidates) can sit after it.
4. Context assembly. The chosen chunks are placed in the prompt with an instruction such as "answer only from the sources below, cite the source id, say you do not know if the sources do not cover it". Chunk order, de-duplication and a token budget are decided here.
5. Generator. The LLM writes the answer conditioned on the question plus the sources, and the system returns the answer together with its citations.

Why grounding reduces hallucination

• Evidence is in the context window (the text the model reads for one request: instructions, question and retrieved chunks, up to a fixed maximum length). The model copies and paraphrases supplied facts, which is more reliable than recalling a rare fact from its parameters (the numbers learned in training).
• Verifiability. Each claim can point to a chunk id, so a human or an automatic check can compare claim and source.
• Abstention path. If retrieval returns nothing above a relevance threshold (a minimum similarity score, chosen by checking on labelled questions which cut-off separates useful passages from noise), the system can answer "no supporting passage" instead of guessing.
• Knowledge is not frozen. Facts that changed after training, or that were never public (internal policies), are available at inference time.

RAG lowers the rate; it does not remove it. Remaining failure modes: the right chunk was never retrieved (recall failure: recall at k asks whether a passage containing the answer is among the k returned; averaged over many questions it is the share of questions that retrieval got right), irrelevant chunks distract the model, two chunks conflict, or the model ignores the context and answers from memory. Measuring those separately is what RAG evaluation is for.

Worked example

The program below uses BM25 (the rank_bm25 library, version 0.2.2) as the retriever. The "standalone LLM" and the "generator" are deterministic stand-ins, not real models: the first is a frozen lookup of an older policy, the second copies the best retrieved sentence and cites its id. A real system would put an LLM in both places. The latency lines use assumed rates, so only the arithmetic is computed.

import os
os.environ["OMP_NUM_THREADS"] = "1"
import re
from rank_bm25 import BM25Okapi

STOP = {"the", "a", "an", "of", "is", "what", "who", "to", "in", "can", "be", "are", "or", "for", "and", "until", "after", "within", "from", "per", "not", "by"}

def tok(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

# A small "knowledge base". Passage p6 is the current policy; the closed-book
# stand-in below only knows the older 30-day rule.
passages = {
    "p1": "Standard shipping takes five business days within the country.",
    "p2": "Gift cards never expire and cannot be exchanged for cash.",
    "p3": "Support is available by chat from nine to five on weekdays.",
    "p4": "Orders can be cancelled until they leave the warehouse.",
    "p5": "Damaged items are replaced free of charge after a photo is sent.",
    "p6": "Refund window: customers can request a refund within 45 days of delivery.",
    "p7": "Loyalty points are credited after the return window closes.",
    "p8": "Express shipping is charged per kilogram and is not refundable.",
}

# Stand-in for a standalone LLM: a frozen lookup of what it "learned" before the policy changed.
frozen_knowledge = {"refund": "Refunds are accepted within 30 days of delivery."}

def closed_book(question):
    for key, answer in frozen_knowledge.items():
        if key in tok(question):
            return answer
    return "I am not sure."

def build_index(store):
    ids = list(store)
    return ids, BM25Okapi([tok(store[i]) for i in ids])

def retrieve(question, store, index, k=2):
    ids, bm25 = index
    scores = bm25.get_scores(tok(question))
    order = sorted(range(len(ids)), key=lambda j: (-scores[j], ids[j]))[:k]
    return [(ids[j], float(scores[j])) for j in order]

def rag_answer(question, store, index, min_score=1.0):
    # Stand-in for the generator: copy the retrieved sentence that shares most words with the question.
    hits = retrieve(question, store, index)
    best_id, best_score = hits[0]
    if best_score < min_score:
        return "No supporting passage found, so I will not answer.", hits
    return f"{store[best_id]} [source: {best_id}]", hits

store = dict(passages)
index = build_index(store)

q = "What is the refund window?"
print("closed-book :", closed_book(q))
ans, hits = rag_answer(q, store, index)
print("RAG         :", ans)
print("top hits    :", [(i, round(s, 3)) for i, s in hits])

q2 = "Who is the CEO of the company?"
ans2, hits2 = rag_answer(q2, store, index)
print("off-topic   :", ans2, [(i, round(s, 3)) for i, s in hits2])

# Freshness: the policy changes again; only the index is updated, no retraining.
store["p6"] = "Refund window: customers can request a refund within 60 days of delivery."
index = build_index(store)
print("after update:", rag_answer(q, store, index)[0])
print("closed-book after update:", closed_book(q))

# Latency budget: ASSUMED figures, only the arithmetic is computed.
retrieval_ms, chunks, tokens_per_chunk, prefill_ms_per_token = 30, 4, 250, 0.4
extra_tokens = chunks * tokens_per_chunk
added_ms = retrieval_ms + extra_tokens * prefill_ms_per_token
print(f"extra prompt tokens = {extra_tokens}; added latency = {added_ms:.0f} ms (assumed rates)")

Output:

closed-book : Refunds are accepted within 30 days of delivery.
RAG         : Refund window: customers can request a refund within 45 days of delivery. [source: p6]
top hits    : [('p6', 3.006), ('p7', 0.99)]
off-topic   : No supporting passage found, so I will not answer. [('p1', 0.0), ('p2', 0.0)]
after update: Refund window: customers can request a refund within 60 days of delivery. [source: p6]
closed-book after update: Refunds are accepted within 30 days of delivery.
extra prompt tokens = 1000; added latency = 430 ms (assumed rates)

Reading it: the closed-book stand-in still says 30 days because that is what it "learned". The RAG path finds passage p6 (BM25 score 3.006, well above the runner-up p7 at 0.99) and answers 45 days with a citation. The off-topic question has no matching terms, both scores are 0.0, and the pipeline abstains rather than invents an answer. After the policy passage is edited, the next answer says 60 days with no retraining, while the closed-book stand-in is unchanged.

Operational challenges: latency and freshness

• Latency. RAG adds a retrieval call and a longer prompt. With the assumed figures above (30 ms retrieval, four chunks of 250 tokens, 0.4 ms of prompt processing per token) the added time is 30 + 1000 x 0.4 = 430 ms. Tuning levers: fewer or shorter chunks, a cheaper first-stage retriever, caching frequent queries, streaming the answer.
• Freshness. Updating knowledge means re-ingesting changed documents, not retraining. The new risks are stale chunks (an old version still in the index), partial re-indexing, and embedding-model upgrades that force re-embedding the whole corpus. Track a document version or timestamp per chunk and delete superseded chunks.
• Cost. Each query pays for the extra prompt tokens; the index costs storage and rebuild time.

Common mistakes

• Saying RAG "eliminates" hallucination. It reduces one cause and adds retrieval-quality as a new failure point.
• Evaluating only the final answer. If the answer is wrong you cannot tell whether retrieval or generation failed unless you also measure the retriever's recall at k (the share of questions for which a passage containing the answer is among the k returned).
• Treating the vector index as the only option: keyword search is often better for exact identifiers.
• No abstention rule, so the model answers confidently from irrelevant chunks.