Retrieval-Augmented Generation (RAG) Questions

Requirements & constraints:

• Only users with sufficient clearance can retrieve sensitive docs.
• Minimal added latency to RAG pipeline; preserve cache hit rate.
• Strong audit trail of who queried what and which docs were returned.
• Scalable for many users and large vector index.

High-level architecture:
User -> AuthN (OIDC) -> API Gateway -> RAG Service (Retriever + Reader) -> Vector DB / Metadata DB -> Cache (tiered) -> Audit Log (WORM/SIEM)

Design components and flow:

1. Authentication (AuthN)

• Use OIDC / JWT issued by IdP containing user_id, roles, clearance_level, and session_id. Tokens are short-lived and signed.

2. Authorization (AuthZ)

• Use Attribute-Based Access Control (ABAC): each document indexed carries metadata: sensitivity_level, set of permitted attributes, ACL bitmask.
• At ingest time store per-doc ACL bitmasks (or tags) in vector DB metadata and in a separate fast metadata store.
• Retriever performs ANN search with a metadata filter: vector DB supports filtering by ACL (e.g., metadata query or boolean mask) so candidate results are already clearance-filtered.
• To minimize latency, precompute ACL bitmasks and push filtering into the ANN engine rather than post-filtering full results.
• Cache-aware keys: caches are sharded by clearance namespace. Cache key = hash(query_vector, semantic_query_text) + clearance_bucket. This preserves cache usefulness while preventing leakage across clearance levels.
• Cache two tiers:
  • Embedding / candidate set cache (non-sensitive): reusable across clearances.
  • Document-content / final-context cache: keyed by clearance_bucket so a high-clearance user won’t use low-clearance cached content and vice versa.

3. Minimize auth latency

• Cache AuthZ decisions per (user_id, session, resource pattern) in an in-memory TTL store (e.g., Redis) to avoid repeated policy evals.
• Use lightweight ACL checks (bitwise AND) and Bloom filters for quick exclusion before invoking expensive ANN queries.
• Push filtering into vector DB to avoid retrieving extra docs that would be filtered afterwards.

4. Audit logging

• Emit structured, immutable logs per request: request_id, user_id, token_claims, query_hash, clearance_bucket, returned_doc_ids with hashed content IDs, timestamp, decision reasons.
• Store logs in WORM storage and forward to SIEM/EDR. Include content-hash not raw content for privacy; retain full content only under controlled workflows.
• Optionally record retrieval scores and re-ranker traces for forensic review.

Trade-offs & notes:

• Pushing ACLs into ANN filters slightly increases index metadata size but massively reduces post-filtering cost.
• Sharding cache by clearance reduces cross-user cache hits; balance by consolidating non-sensitive caches (embeddings) while isolating sensitive content caches.
• Ensure token freshness and revoke on compromise; short JWT lifetimes + revocation lists in Redis.
• Monitor latency impact; measure cache hit ratio per clearance bucket and optimize bucket granularity.

This design enforces clearance at retrieval (lowest-latency enforcement), maintains cache effectiveness by separating non-sensitive shared caches from clearance-scoped caches, and provides strong, query-level audit trails.

Hybrid retrieval combines sparse lexical matching (like BM25) with dense embedding-based retrieval to leverage both exact term overlap and semantic similarity. It returns candidates from both retrievers and fuses their scores to produce a ranked list.

Example architecture:

• Indexing:
  • Build an inverted index for BM25 over tokenized documents.
  • Encode documents with a sentence-transformer and store vector embeddings in a vector DB (FAISS).
• Query flow:
  • Receive query → compute BM25 top-K1 candidates.
  • Encode query → retrieve top-K2 dense candidates by cosine similarity.
  • Union candidates, deduplicate, fuse scores.
  • Optional: re-rank top-N with a cross-encoder (cross-attention model) for final ranking.

Why hybrid helps:

• BM25 captures exact phrase/keyword matches and strong lexical signals (dates, names, technical terms).
• Dense retrievers capture paraphrases and semantic intent where wording differs.
• Combining reduces blind spots of each approach, improving recall and robustness across query types.

Simple scoring fusion strategy:

1. Normalize scores:
   • BM25_norm = (BM25 - minBM25) / (maxBM25 - minBM25)
   • Dense_norm = (cosine + 1) / 2 (maps [-1,1] → [0,1])
2. Weighted sum:
   • Hybrid_score = alpha * Dense_norm + (1 - alpha) * BM25_norm
   • Typical alpha: 0.4–0.6; tune on validation set.
3. Sort by Hybrid_score; optionally re-rank top-M with a cross-encoder.

Practical notes:

• Ensure score normalization on per-query basis to avoid scale mismatch.
• Use deduplication and diversity heuristics; tune alpha for query types or learn fusion weights via logistic regression on labeled pairs.

Positives

• Use high-quality human-labeled (or click-filtered) query-document relevance pairs as positives (q, d+) with binary or graded labels (relevant / highly-relevant). For graded labels include label-aware weighting.

Negatives

• Mix of: (1) Random negatives (documents from corpus not labeled relevant) to provide coarse signal; (2) In-batch negatives (other positives in same batch) for efficiency; (3) Retrieval-stage hard negatives (see below) to teach subtle rank distinctions.

Loss function

• Use a listwise softmax cross-entropy over a candidate set per query:
  P(d_i|q) = exp(sim(q,d_i)/τ) / Σ_j exp(sim(q,d_j)/τ); loss = −log P(d+|q)
  This encourages relative ordering and is stable; temperature τ tuned (0.05–0.2). Optionally combine with margin ranking loss for robustness:
  L = CE_listwise + λ * Σ max(0, margin − sim(q,d+) + sim(q,d−)).

Hard-negative mining from retrieval

1. Initial pass: use a bi-encoder (or BM25/ANN) to produce top-K candidates per q. Treat top-K non-positives as hard negatives.
2. Iterative mining: periodically re-generate hard negatives using current reranker or a late-stage bi-encoder to find negatives the model confuses.
3. Diversity & difficulty filtering: keep negatives that are (a) high-scoring by retriever or reranker, (b) semantically similar (embedding distance), and (c) not near-duplicates to avoid label noise.
4. Adversarial/mined-by-gradient: optionally generate or score candidates by gradient-based perturbations for toughest negatives.
   Practical tips

• Maintain mix ratio (e.g., 1 positive : 3 hard negatives : 2 random).
• Deduplicate and sanity-check clicks to reduce label noise.
• Evaluate with NDCG@k and recall@k; monitor calibration to avoid overfitting to top-K artifacts.

Start by clarifying goals: use the KB’s symbolic relations to (1) improve retrieval precision for RAG and (2) enforce or verify logical constraints in generated answers.

High-level approach:

• Hybrid index: keep a dense vector index (embeddings of docs/passages) and a symbolic KG (triples, entity IDs, relation types). Maintain an entity-to-passage mapping.
• Query pipeline:
  1. Entity & intent extraction: run NER + relation-intent classifier on the user query to identify entities and probable relation types (e.g., "Who manages Alice?" → entity: Alice, relation: is-manager-of).
  2. KG-guided retrieval: use extracted entities/relations to (a) fetch relevant neighbor entities/triples from KG (1–2 hops), (b) expand the query with relation-aware phrases and entity aliases, and (c) build a hybrid candidate set: union of KG-derived passages and top-k vector-similar passages to the expanded query.
  3. Reranking with symbolic signals: score candidates by a weighted sum — semantic similarity + KG match score (exact relation presence, path length, provenance confidence). Use learned ranker (e.g., cross-encoder) that takes passage text and binary features: relation_exists, path_length, relation_confidence.
  4. Constrained generation / verification: condition the LLM on selected passages plus explicit KG facts. Post-generate, validate claims against the KG; if contradiction detected, either correct or flag uncertainty.

Implementation notes:

• Entity linking: map mentions → canonical KG IDs; necessary for reliable joins.
• Represent KG signals: precompute sparse relation embeddings or binary feature vectors per passage.
• Latency trade-offs: cache frequent entity neighborhoods; do shallow KG lookups (1–2 hops) inline; heavier graph traversal offline to augment embeddings.
• Evaluation: measure precision@k, factuality (entity/relation F1), and hallucination rate. A/B test weighted scoring and reranker variants.

Why it works:

• KG supplies high-precision relational facts that disambiguate and constrain retrieval.
• Vectors supply contextual relevance and synonyms. Combining both reduces hallucination and improves answer correctness while retaining recall.

Start by distinguishing the uncertainty types you care about:

• Aleatoric (intrinsic ambiguity in user input) — signals: low-confidence tokens, question length, vague phrasing.
• Epistemic (model ignorance) — signals: model calibration, out-of-distribution (OOD) inputs, sparse retrieval.

Calibration & basic tooling

• Calibrate raw model confidences on a labeled validation set (temperature scaling, isotonic/Platt) so probability estimates map to real correctness likelihoods.
• Produce per-answer confidence metrics: softmax/logit gap, answer entropy, top-token probabilities, and normalized perplexity.
• Use ensemble or MC-dropout to estimate epistemic uncertainty: variance across samples or ensemble members → uncertainty score.

Retrieval / pipeline signals to combine

• Retrieval score(s): top-k similarities, score gap between top hits, number of hits above threshold.
• Source trustworthiness: provenance, freshness, domain match.
• Answer grounding: percentage of generated claims that are supported by retrieved passages (explicit citations, faithfulness score).
• Conflicting evidence: contradictory retrieved passages or high source disagreement.

How to decide abstain vs clarify vs answer

• Define risk thresholds using cost-sensitive criteria (false-accept worse than false-abstain?). For example:
  • If calibrated answer confidence < C_low OR ensemble variance high → abstain or ask for clarification.
  • If retrieval trust < R_low OR top-k retrieval scores low or contradictory → ask for clarifying question (e.g., "Do you mean X or Y?") rather than hallucinate.
  • If confidence medium but retrieval supports partial facts → answer with hedging and cite sources.
• Use a learned abstention classifier (features: calibrated prob, entropy, ensemble variance, retrieval scores, provenance flags, token-level hallucination detector). Train it on historical examples labeled "safe to answer" vs "should abstain/clarify".

Practical patterns

• Favor clarification when signals indicate ambiguity (short question, multiple possible intents, high token uncertainty) but retrieval scores are low-to-medium.
• Favor abstention when model shows high epistemic uncertainty (OOD input, ensemble disagreement) and the cost of a wrong answer is high.
• Always return transparent signal: include confidence, citations, and reason for abstention/clarification.

Monitoring & iteration

• Continuously recalibrate with post-deployment feedback, track calibration drift, and tune thresholds by outcome metrics (precision of accepted answers, abstain rate, user satisfaction).

Example rule set (simple):

• If calibrated_confidence < 0.6 OR ensemble_variance > v1 → abstain.
• Else if retrieval_top_score < 0.4 OR provenance_low OR conflicting_evidence → ask clarifying question.
• Else → answer with citations and confidence.