Approach: combine strict system instructions, retrieval-augmented grounding, iterative clarification (reciprocal asking), and post-generation verification to limit hallucinations. Below are concrete templates, implementation patterns, and evaluation metrics.
System message (high priority):
You are Copilot for Microsoft 365. Only provide information that can be directly supported by the user’s documents, company knowledge base, or reputable web sources retrieved for this session. If the answer cannot be fully grounded, clearly state which parts are uncertain and cite sources with anchors (document id + excerpt + sentence index).
Prompt template for user task (RAG + grounding):
User request: <USER_TEXT>
Context: include user’s mailbox/documents: [doc_id:score:top_k_sentences]
Instruction: Using only the provided context and cited web sources, produce: (1) concise answer (≤5 bullets), (2) supporting evidence lines (doc_id: sentence#), (3) a short uncertainty statement if any claim is not fully supported. Do NOT invent dates, names, or figures not in context.
Reciprocal asking (clarification loop):
Before answering, ask up to two targeted questions if critical grounding info is missing. Example:
I need to confirm which document to prioritize: (A) latest project spec dated <date> or (B) leadership email thread? Reply with A or B or paste the relevant paragraph.
Grounding / RAG pipeline:
- Retriever: semantic + keyword hybrid search over indexed M365 artifacts with passage-level scoring.
- Reranker: cross-encoder to select top-k passages.
- Evidence extraction: return exact sentence spans and offsets.
- Answering: instruct LM to only use those spans; include quoted evidence inline and citation anchors.
- Verification: call a lightweight QA model that asks factual questions about each claim and checks if answers are present in retrieved contexts.
Post-generation verification (self-consistency):
- Generate N candidates, extract factual claims, and accept only claims that appear in ≥ceil(N/2) candidates AND are supported by at least one retrieved span; otherwise mark as “uncertain”.
Example reciprocal-asking template to reduce hallucination:
You asked for “project timeline.” I have two documents that mention timelines with different dates. Do you want (1) dates from the Project Spec doc or (2) dates from the PM email thread? Reply 1 or 2.
Evaluation metrics:
- Groundedness rate: fraction of model claims with at least one direct supporting span (target >95%).
- Hallucination rate: fraction of factual claims contradicted by retrieved corpus or absent from it.
- Precision@claims: correct supported claims / total claims.
- Factual QA accuracy: auto-generated QA pairs from answer checked against corpus (F1 score).
- Citation usefulness: human-rated 1–5 whether provided citations resolve the claim.
- Human evaluation: blind annotators label answers as “grounded,” “partially grounded,” or “hallucinated.”
- A/B online metric: reduction in downstream correction/edit rate by users and reduction in help-desk tickets caused by wrong Copilot output.
Implementation notes:
- Enforce strict token-level filters to prevent confident-sounding ungrounded statements.
- Log provenance for every claim (doc_id, span, retriever score).
- Use thresholds on retriever/reranker scores to trigger clarification rather than guessing.
This combination—explicit system constraints, RAG with sentence-level evidence, reciprocal asking to fill gaps, and automated + human evaluation—reduces hallucination while keeping Copilot useful in Microsoft 365 workflows.
