Direct answer
The smallest believable scope is one search use case, on one slice of the buyer's own content, compared against their current search on a set of real queries, judged by a few real users. I would propose about 2,000 to 5,000 documents from a single source, 50 real queries taken from their search logs, and a blind side-by-side comparison by 5 to 8 users. That is enough to be credible because it uses their data, their questions and a baseline they already know. Everything else (more sources, personalisation, scale) is a later phase.
Two-week plan
| Days | Milestone | Risk to watch |
|---|
| 1 to 2 | Agree the use case, the 50 queries and the pass criteria; get an export of the content slice | Data not available |
| 3 to 5 | Load and index the content (index = build the lookup structure the search engine queries); run the baseline (their current search) on the 50 queries | Content quality, access delays |
| 6 to 8 | Tune the AI search; spot-check the failures | Chasing edge cases past the window |
| 9 to 10 | Run the comparison and write up | Disagreement over judging |
| End of week 2 | Readout | Over-claiming |
Deciding what data is enough
- Content: one source with real variety (long and short documents, some messy ones). If the content is mostly duplicates, the test proves nothing.
- Queries: sampled from their real search logs, including bad ones and ones with no result. Queries I invent flatter the product.
- Sensitive data: if the buyer has limited production access or strict security rules, use anonymised or masked data. If the content slice comes as a CSV (comma-separated values) export, for example a customer-records table behind the search, I would ask them to drop identifying columns or hash them (replace each value with a fixed scrambled code; use a keyed or salted hash, because a plain hash of an email address can be reversed by hashing guessed addresses and comparing), and to replace names with consistent tokens so joins still work (rows about the same person still line up when tables are combined on that column). Before and after for one column:
ana.silva@acme.com becomes user_0417 everywhere it appears. I ask them to confirm in writing what was removed. Where even that is refused, I use synthetic data (invented records) modelled on their schema (the same fields and types as theirs), and say plainly that the result shows function, not their data quality.
Deciding what user validation is enough
A relevance judgement (a person deciding whether a result answers the query) is cheap to collect: for each of the 50 queries, show the user the two result lists in random order, unlabelled, and ask whether each list has a correct result in its top three. A worked example (illustrative numbers), using that one measure throughout: the baseline puts a correct result in the top three for 31 of 50 queries and the AI search for 41. That is 62% against 82%. Counted query by query, the AI search wins on 14 queries (correct where the baseline is not), loses on 4 (the baseline is correct and it is not) and ties on 32 (both correct or both wrong), so 31 + 14 - 4 = 41. Only the 18 non-tied queries carry information. If the AI search wins 14 of those 18, a fair coin would produce a split at least that lopsided, in either direction, only about 3 times in 100 (exact two-sided sign test, computed: 0.031; counting one direction only it is 0.015). Why 50 queries cannot support a small gap: a four-point difference is just 2 queries out of 50, for example 7 wins against 5 losses, which a coin flip produces about three times in four, so it is noise; a twenty-point difference (10 queries) is hard to dismiss. I say that in the readout. I also report the failure cases, because a buyer who finds them first stops trusting the rest.
The components I would include and exclude
Include: indexing (preparing the content so it can be searched quickly), query handling, ranking (ordering results by relevance) and one results screen they can click. Exclude: permissions trimming (hiding results a given user is not allowed to see, across every connected system), personalisation (results tailored to each user's history), analytics dashboards and tuning for every document type.
Tighter constraints
- Strict security, a 30-day window, and an integration that might slip: I plan two fallbacks up front. Fallback 1: the buyer exports a static snapshot and I load it into my sandbox. Fallback 2: synthetic data on their schema. Both keep the clock moving.
- Minimum viable integration in 48 hours: one read-only endpoint (a web address that can only return data, never change it), one service credential (a login for a program rather than a person) with read-only rights, a sample of 100 records, a field mapping (a one-page table saying which of their fields feeds which of ours) written on one page, and a quick validation (row counts match, ten records spot-checked by their data owner).
- Early-career engineer, small account: the same two-week table, with a mentor review on day 2 and day 8.
Trade-offs and pitfalls
Smaller scope lowers the risk of failing but also what the result proves: the buyer must accept that a pass on 3,000 documents is not proof about 30 million. The wrong turn is a pretty demo on cherry-picked queries. I flip toward a bigger scope only if the buyer's decision depends on scale, in which case I say so and price it separately.
