Requirements & constraints:
- Prevent LLM hallucinations that lead to incorrect customer actions (high safety).
- Support human-in-the-loop for risky answers.
- Low latency (<500ms preferred for simple Qs), but allow higher for verified responses.
- Auditability, traceability, and metrics for hallucination rate.
- Scalable to many customers with different knowledge bases.
High-level architecture:
Customer UI ↔ Orchestrator → {Retriever (vector DB + metadata), Context Assembler, Prompt Engine} → LLM → Output Validator → Attribution & UI Renderer → Monitoring & Audit Log. Human Review Queue and Feedback Loop attach to Validator and Monitoring.
Core components & responsibilities:
- Retriever (RAG): hybrid retrieval — vector search for semantic matches + keyword/metadata filters to ensure source relevance and freshness. Include per-customer knowledge indices and a fallback to canonical product docs.
- Context Assembler & Defensive Prompting: build concise context with source snippets, provenance metadata, and explicit instructions (e.g., “If no confident source, respond: ‘I don’t know — escalate’”). Use temperature control, token limits, and chain-of-thought suppression for production.
- Grounding & Attribution: responses must include cited snippets with source links and confidence score. Normalize citations to sections/lines for customer docs.
- Output Filters & Validators: rule-based validators (regex, policy checks), semantic validators (answer consistency with retrieved snippets using an entailment model), and action-safety checks that block any response that would trigger irreversible operations without explicit human confirmation.
- User Verification Flows: for medium/high-risk actions require explicit user confirmation, 2-step verification, or escalation to human agent. Present provenance and “why this is suggested” UI.
- Human-in-the-loop: uncertain/conflicting retrieval results go to an agent queue with prefilled context to minimize review time.
- Monitoring & Metrics: log retrieval hits, provenance coverage, confidence, validator pass/fail, and downstream actions. Compute hallucination rate via sampled human review and automated entailment mismatches. Alerting for rising trends and per-customer SLAs.
- Feedback loop: store confirmed corrections to retriever index and prompt templates; fine-tune or use retrieval augmentation periodically.
Latency vs UX trade-offs:
- Fresh retrieval + entailment checks add 100–400ms. For low-risk FAQs, prioritize speed: cache common Q+A and allow concise answers with source links.
- For high-risk operations, accept higher latency for extra validation and human review — surface “Checking sources — this may take ~30–90s” and provide interim safe suggestions.
- Offer progressive disclosure: a fast, conservative answer (short, with source links and “may be incomplete”) then background enrichment that updates UI when full validation completes.
Trade-offs & justifications:
- Strong grounding and validators reduce hallucinations but increase complexity and latency. Use risk-based routing: low-risk queries prioritize latency; high-risk prioritize safety.
- Entailment checks may produce false negatives; combine with sampled human review to tune thresholds.
- Storage & indexing cost rises with per-customer indices; mitigate via sharding and TTLs for stale data.
Operational considerations:
- Define SLAs for acceptable hallucination rates and per-customer risk profiles.
- Regular audits, red-team testing, and synthetic hallucination injection for monitoring sensitivity.
- Clear product UX messaging: transparent citations, “confidence” indicators, and easy escalation paths.
This design balances safety, traceability, and practical latency by applying layered defenses: RAG grounding, prompt constraints, automated validators, and human verification where risk demands it.
