Retrieval-Augmented Generation (RAG) Questions

Requirements & high-level: build a low-latency, high-precision RAG pipeline that retrieves relevant KB passages and conditions a sequence model to answer customer queries. Key goals: high recall of relevant context, concise context under token budget, fresh index updates, predictable latency (<200–500ms retrieval target) and measurable QA quality.

Architecture & components:

• Embedding model: use a dual approach — a fast bi-encoder for ANN retrieval (e.g., Sentence-Transformers like all-mpnet-base-v2 or distilled OpenAI/Anthropic embeddings) for 768–1536d vectors; reserve a higher-capacity cross-encoder (MonoBERT/Instruction-tuned transformer) for re-ranking.
• Vector store: FAISS with HNSW or IVF-HNSW for scale. Compress with OPQ + PQ when dataset grows to millions to reduce memory.
• Similarity strategy: initial ANN using cosine or inner-product on L2-normalized vectors; combine with lexical BM25 hybrid scoring (weighted sum) to handle rare tokens and names.
• Re-ranking: top-N (e.g., 100→10) re-rank using a cross-encoder trained/fine-tuned on in-domain query–passage relevance; use features: cross-score, BM25, recency, source-trust.
• Document processing & chunking: chunk KB into overlapping passages (e.g., 200–400 tokens, 50–100 token overlap), store metadata (source, timestamp, version).
• Context assembly and token limits: select top-K re-ranked passages, then apply a greedy selector: prioritize by combined score and uniqueness (semantic similarity threshold to avoid duplication) until prompt token budget is reached (e.g., 2048 tokens minus model prompt footprint). If long answers needed, include "pointer" to source sections and retrieval augmentation.
• Freshness/updating: support hybrid updates:
  • Nearline incremental embedding for small updates (embed new/updated docs and upsert to FAISS).
  • Periodic full re-index (daily/weekly) to re-run OPQ/PQ and rebuild indexes.
  • Version tags and time-decay scoring so recent docs get bonus.
  • Use a change-data-capture pipeline (Kafka) triggering embedding workers for low-latency ingestion.
• Latency considerations:
  • Keep ANN index in RAM on retrieval nodes; shard indices per vertical for locality.
  • Use HNSW for single-query low latency; tune efSearch for recall/latency tradeoff.
  • Cache top-K retrievals for frequent queries and results of re-ranker for hot items.
  • Batch embedding requests where possible; use GPU for cross-encoder scoring and limit cross-encoder calls to top candidates.
  • Monitor p95/p99 latency; autoscale retrieval/re-ranking workers.
• Evaluation in production:
  • Retrieval metrics: Recall@k, MRR, nDCG against labeled relevance sets and synthetic queries.
  • End-to-end: answer correctness via automated metrics (EM, F1, ROUGE where applicable) on held-out QA pairs; measure hallucination rate using fact-checking heuristics (e.g., answer-claim vs. supporting passages).
  • Human eval: periodic annotation for answer accuracy, helpfulness, and attribution (does the model cite right source).
  • Online: A/B tests measuring task success (issue resolution rate, deflection to self-service), user satisfaction (CSAT), and metrics for harmful hallucinations.
  • Monitoring & feedback: log retrievals + generated answer + clicked sources; use click-through and manual flags to fine-tune reranker and generation prompts. Retrain reranker on fresh labels and run safety checks.

Trade-offs: heavier re-rankers and larger cross-encoders improve precision but add latency and cost—mitigate by limiting re-ranker depth and caching. Hybrid lexical+dense retrieval improves robustness to queries with entity/morphology changes.

Requirements:

• Functional: answer user queries using private documents with low latency.
• Security/Privacy: strong confidentiality, per-request authorization, no data leakage, auditable retrievals.
• Non-functional: scalable indexing, low retrieval latency, compliance (e.g., GDPR).

High-level architecture:

• Ingestion service → PII scrub & transform → Chunker + encoder → Encrypted vector store (VS) + metadata DB → Retrieval service (authz) → RAG generator (model) → Response sanitizer → Audit & logging.
• Components run in VPC with mTLS between services, KMS-managed keys, and HSM for key material if required.

Indexing private data:

• Preprocessing: canonicalize, extract metadata (owner_id, doc_id, sensitivity_level), tokenize and chunk with overlap, compute embeddings in a secure enclave or isolated inference service.
• Encrypt: store ciphertext for chunks (AES-GCM) and store encrypted embeddings if threat model demands. Keep searchable plaintext embeddings only inside secure VS process memory.
• Metadata: keep minimal plaintext metadata for fast ACL checks; sensitive metadata encrypted and decrypted only after authz.

Per-request access control:

• Use short-lived access tokens (OAuth2 / JWT signed by auth service) containing user_id, scopes, and purpose; tokens bound to request context and client certificate.
• Retrieval service enforces ACL checks before any similarity search: filter candidate chunks by owner_id / group / sensitivity tags using metadata-level filters.
• Attribute-Based Access Control (ABAC): combine user attributes, document sensitivity, purpose-of-use, and consent flags. Policy engine (OPA) evaluates per-request.

Preventing leakage during generation:

• Retrieval-only sandbox: only pass vetted chunk snippets to generator, not raw documents. Redact or replace exact PII if not required.
• Context window controls: limit number/size of retrieved chunks; add prompt-level safety instructions (do not reveal verbatim private data unless explicit consent).
• Differential privacy / token filtering: apply response filtering to detect verbatim reproduction of long spans from retrieved chunks (n-gram overlap checks). If overlap > threshold, either paraphrase automatically or refuse.
• Model isolation: run LLM in a trusted execution environment; no outbound network from model container except to audit sink.

Runtime checks and audit:

• Retrieval audit log: record request_id, user_id, timestamp, query embedding, candidate chunk ids and similarity scores, ACL decisions (policy evaluation result). Store logs immutably (WORM) and encrypted.
• Generation audit: record prompts given to model, final model output, and any redaction/paraphrase actions. Hash stored for non-repudiation.
• Real-time monitors: anomaly detection on retrieval patterns (e.g., many high-similarity hits across users), rate limits, and automated throttling.
• Compliance tooling: support subject access requests and data deletion by correlating doc_id to index entries and logs; enable re-encryption/rotation.

Encryption & keys:

• Use envelope encryption: data encrypted with per-tenant DEKs; DEKs encrypted by KMS/CMK. KMS enforces IAM policies and supports key rotation.
• Keys scoped by tenant or sensitivity level; access to DEKs requires successful authz and purpose attestation.
• Embeddings: if stored encrypted, keep a separate secure service that decrypts embeddings into memory for similarity search; no plaintext embeddings at rest.

Trade-offs & failures:

• Performance vs. security: encrypted-embeddings search adds latency; mitigate with secure enclaves or homomorphic approximate search (expensive).
• False negatives from strict ACL filters: provide escalation workflows for legitimate cross-user access with audit trail and time-bound access tokens.
• Leakage detection may cause false positives; tune overlap thresholds and fallback human review.

Example flow:

1. Client requests answer with JWT.
2. Retrieval service validates token, calls OPA policy -> filters index to authorized chunks.
3. Similarity search returns top-K chunk ids; service redacts PII, logs retrieval, and sends sanitized context to generator.
4. Generator produces output; overlap detector compares output to chunks; if verbatim leakage detected, trigger paraphrase or block and log incident.
5. Final response returned; full audit trail persisted.

This design balances usability and privacy via strict authz at retrieval, encryption-at-rest, runtime sanitization, and comprehensive auditing.

High-level approach

1. Create embeddings with Azure OpenAI for each document (or chunk). 2) Index documents into Azure Cognitive Search with a vector field plus usual textual fields and metadata. 3) Implement hybrid search: combine keyword (BM25) and vector similarity. 4) Rerank top candidates with a stronger model (cross-encoder or OpenAI prompt-based reranker). 5) Keep index fresh via event-driven incremental updates and periodic reingestion.

Detailed steps

1. Embedding creation

• Split long documents into semantic chunks (200–1000 tokens) with overlap.
• Call Azure OpenAI embeddings API per chunk and store embedding (float array) with doc_id, chunk_id, source metadata.
  Example (Python, simplified):

from azure.ai.openai import OpenAIClient
client = OpenAIClient(endpoint, credential)
emb = client.get_embeddings(model="text-embedding-3-small", input=["text chunk"])[0].embedding

2. Index design & indexing vectors

• Create a Cognitive Search index with fields: content (searchable), title, metadata, and a vector field (collection of floats) with appropriate dimensions.
• Use the REST/SDK to upload documents with embedding in the vector field. Enable retrievable/scoring profiles.

3. Hybrid search (keyword + vector)

• First, run a keyword search (full-text) to capture exact matches and filters; also run a vector search (KNN) to capture semantic matches.
• Combine scores: normalized BM25 score + alpha * vector cosine score. Use Cognitive Search scoring profiles or merge results in application layer.
• Example flow: use search.parameters to request vector search topK=50, then combine with lexical search topK, dedupe by doc_id and compute combined score.

4. Reranking techniques

• Take top N (e.g., 20) candidates, build a prompt for Azure OpenAI or use a lightweight cross-encoder (fine-tuned transformer) to compute relevance scores given query + candidate content.
• Optionally use supervised learning-to-rank model trained on click/feedback for production.
• Use reranker to order final results and to extract answer snippets.

5. Keeping the index up-to-date

• Use event-driven pipelines: when source (blob/db/cms) changes, emit events (Event Grid). An Azure Function consumes events, computes new/updated embeddings, and calls Cognitive Search indexer (or SDK) to merge/update documents.
• For deletes: mark soft-delete metadata and purge via indexer.
• For bulk changes: schedule nightly reingestion or rebuild when many updates occur.
• Maintain provenance and embedding versioning (model version, timestamp) to know when to recompute embeddings after model changes.

Operational considerations

• Monitor latency and cost: embeddings are expensive—batch requests and cache frequent queries.
• Dimension and index trade-offs: larger models→better quality but higher storage/latency.
• Relevance tuning: adjust alpha between lexical and vector, tune chunk size, and evaluate with offline metrics (MRR, NDCG) and A/B tests.
• Security & compliance: encrypt embeddings at rest, control PII before sending to OpenAI.