Consultative Discovery and Requirements Gathering Questions

Direct answer

A stakeholder's demand (a stakeholder is anyone with a stake in the outcome, such as the business owner or the risk team) for "99.9% precision" is a symptom. First I would find out what the number protects: wasted investigator time, blocked genuine customers, or a regulatory risk. Then I show the trade-off between precision and the share of fraud caught, with the client's own volumes, and propose a target that matches the real cost of each error, usually with a human review stage for borderline cases.

Step 1: find what is behind the number

• "What happens when we flag a legitimate transaction? Who pays?"
• "What happens when we miss a fraud? What does it cost?"
• "How many alerts can your investigators review per day?"
• "Where did 99.9% come from: a regulator, a vendor, a past incident?"
• "Precision of what: all transactions flagged, or only those auto-blocked?"

Precision is the share of flagged transactions that are genuinely fraud. Recall is the share of all fraud that gets flagged. The false positive rate is the share of legitimate transactions wrongly flagged. They are different, and stakeholders often mix them.

Step 2: feasibility with illustrative volumes

Take 1,000,000 transactions in a month, of which 0.2% are fraud: 2,000 frauds and 998,000 legitimate. Fraud is rare, so errors are expensive.

• 99.9% precision means at most about 1 false alarm per 1,000 flags. The model gives every transaction a fraud score, and the threshold is the score above which a transaction is flagged. Raising the threshold shrinks the flagged set and raises precision, but more real fraud scores below the line and goes unflagged. So a threshold strict enough for 99.9% might flag only 1,000 transactions (illustrative), 999 of them real, which catches 999 of 2,000, about 50% of fraud.
• Accuracy hides this. A model that flags nothing is 99.8% accurate here (998,000 of 1,000,000 correct) and catches 0 of the 2,000 frauds, which is why imbalance (one class far rarer than the other) makes accuracy misleading.
• For comparison, suppose the head of risk had asked instead for "15% fraud reduction with under 1% false positives". That gives: 15% of 2,000 is 300 frauds caught. A false positive rate of 1% means 9,980 legitimate transactions flagged. Precision is 300 out of 10,280 flags, about 2.9%. At 0.1% false positives, there are 998 false alarms and precision is 300 out of 1,298, about 23.1%.

So "under 1%" sounds small but may flood the team (10,280 flags a month is more than the 9,000 a month that 300 reviews a day allows), and in this illustration 99.9% precision leaves about half the fraud uncaught (the real share depends on how well the model separates fraud from legitimate transactions, which I would measure on the client's own data).

Step 3: what I propose instead

A tiered design: auto-block only above a very high score (precision near the stated figure on that slice), send the mid-score band to human review within the team's daily capacity, and allow the rest. Report precision and recall at each tier. Example target (illustrative figures): "At a review capacity of 300 alerts per day, catch at least 60% of fraud by value, with auto-block precision of at least 99%". I would confirm the 60% against a measured baseline before committing to it.

Making a target testable

State the baseline period (for example the last 12 months), how fraud labels were created (chargebacks, where a cardholder disputes a charge and the bank takes the money back, arrive weeks late, so recent months look cleaner than they are), and sampling bias (the labelled data is a skewed sample: labels exist only for transactions that were investigated or not blocked). Use a held-out time period, meaning train on older months and test on later months the model never saw, not a random split that shuffles all rows together and lets the model peek at future patterns.

Other requirements to raise during scoping

Imbalance and drift risk apply to any fraud model. Interpretability, fairness and audit needs apply when the model also drives lending decisions, is used by field agents, or runs as an internal tool.

• Trustworthy lending model: ask about interpretability (can a person see why the model decided what it did), fairness checks across groups (comparing error rates for different groups of customers), and audit and record-keeping needs.
• Interpretability for field agents: what reason would an agent need to see to act on a flag?
• Auditable internal tool: log every score, model version and decision, so a case can be reviewed later.
• Imbalance and drift risk: rare fraud makes accuracy misleading, and fraudsters change behaviour, which makes the model's scores less reliable over time (drift), so agree monitoring and retraining triggers.

Trade-offs and pitfalls

• Promising a number before measuring a baseline.
• Optimising precision alone, which silently drops recall.
• Ignoring the cost of review time and customer friction.
• Evaluating on random splits when fraud patterns shift over time.

Direct answer

Spend the hour on four groups of questions: the business decision and the cost of being wrong, how the outcome is defined and when you learn it, how the score will be used in daily operations, and the legal and data constraints. A credit-risk model estimates how likely an applicant is to fail to repay (to default). The answers that most change what you build are: which decision the score drives, what a bad loan costs compared with a lost good customer, when outcomes become known, whether every decline must be explained, and how fast the score must return.

Suggested hour: 60 minutes

Business 15 + risk 15 + operations 15 + constraints 15 = 60. Legal and infrastructure people are often not in the room, so ask the constraints questions of whoever is present and name an owner for each unanswered one.

Questions, and what the answer changes

Worked example: why "accuracy" is the wrong ask

Suppose 5 of every 100 applicants default (illustrative). A model that approves everyone is wrong on 5 and right on 95, so it is "95% accurate" and useless. This is why the cost question and the cutoff question come first.

Worked example: choosing the cutoff by cost (illustrative)

Suppose a good loan earns 100 in margin and a default loses 1,000, so a default costs 10 times the margin. Expected profit per approved applicant with default probability p is (1 - p) x 100 - p x 1,000 = 100 - 1,100p, which is break-even at p = 100 / 1,100, about 9.1%.

Approving bands 1 to 3 earns 49,200. Approving band 4 too drops it to 36,200, so the cutoff sits between bands 3 and 4. If a default cost only twice the margin, break-even would be 100 / 300, about 33%, and every band would be approved, which is why the cost question changes the build.

If time runs short

Ask three questions first: which decision the score drives, what a bad loan costs against a lost good customer, and whether every decline must be explained and which inputs are forbidden. They set the model type, the cutoff and the allowed inputs. The rest can go to written follow-ups.

The requirement I would write down (illustrative numbers)

"Score each new application within 1 second at decision time, using only fields available then. At the same approval rate as today's policy, the bad-loan rate among approved applicants on held-back applications old enough for the outcome window to have matured (for a 12-month window, applications at least 12 months old; for newer ones, the early-warning proxy label) (past applications set aside, not used for training, so the model can be tested on cases it has not seen) must be lower than today's policy, and every decline must carry reason codes."

Pitfalls

• Discussing algorithms before the decision and error costs are agreed.
• Skipping the operations team, then discovering the score arrives after the decision is made.
• Treating one hour as enough: record open questions with owners and dates.

Direct answer

I would stay curious, not dismissive. The stakeholder (a person with a stake in the outcome, here the one asking for the technique) has proposed a solution, not a problem. "Use deep learning" (neural networks with many layers that learn patterns from large amounts of example data) is a proposed solution, and I need the problem behind it first. In the conversation I would say: "Happy to use it if the problem needs it. Help me understand what we want to improve, so I can pick the simplest approach that meets the bar." Then I would learn the need, the current performance, the data and the constraints, and agree to build a simple baseline before committing to the heavy method.

Responding in the room

• Acknowledge the intent: "Sounds like you want something more accurate than today."
• Separate solution from need: "What decision or outcome should improve, and for whom?"
• Offer a cheap test, not a refusal: "Let's run a simple baseline for a week, then see what a bigger model would add."

What I need to learn before agreeing

1. The outcome and decision. What changes when the prediction is right, and what does a wrong one cost? A missed fraud case and a wrongly flagged email cost very different amounts.
2. Today's performance. What does the current process or rule achieve? There is no baseline yet, so the first deliverable is one.
3. The data. How many examples, whether they are labelled (marked with the right answer), how clean they are, and what type they are: images, text or audio versus rows in a table.
4. Constraints. Response time, explainability (can we say why it decided?), privacy or regulatory limits (legal rules on how decisions may be made), and who maintains the model.
5. Why deep learning. Sometimes there is a real reason (a vendor, a leadership goal, images or text as input). Asking is not an attack.

What a simple baseline is

The simplest reasonable method: a rule, a count of the most common outcome, or a plain model such as logistic regression (a simple statistical model that weighs each input and outputs a probability). Baseline and benchmark are different: a baseline is what the simplest method achieves on our own data, while a benchmark is an outside reference such as a published result or a competitor's figure. It sets the bar a heavier model must beat, and it often shows how much signal the data holds at all.

Worked example (illustrative)

A stakeholder wants deep learning to flag urgent support tickets. Support can review about 100 tickets a day, so the metric is how many urgent tickets land in the top 100 flagged. I build a keyword-rule baseline and a simple text model in days. Before looking at results, I agree the decision rule with the stakeholder: "We adopt the heavier model only if it catches clearly more urgent tickets at the same review volume, by a margin worth its extra cost and upkeep." Illustrative numbers from a test week, counting urgent tickets among the top 100 flagged:

The agreed margin is "at least 10 more than the best simple method". Logistic regression adds 7 over the baseline, and deep learning adds 71 - 62 = 9 over logistic regression, so deep learning misses the bar and we ship the simple model. Had deep learning scored 75, it would add 13 and the extra cost would be justified. If the baseline is already close, we ship it and save the effort. If the gap is large, deep learning is justified with evidence.

Trade-offs and pitfalls

• On small, structured tables, simpler models are often competitive, but I would test it, not assert it.
• Refusing outright damages trust. Agreeing without a baseline commits cost with no evidence.
• Agree the threshold before seeing results, so nobody picks the bar after the fact.