Consultative Discovery and Requirements Gathering Questions

Direct answer

Spend the hour on four groups of questions: the business decision and the cost of being wrong, how the outcome is defined and when you learn it, how the score will be used in daily operations, and the legal and data constraints. A credit-risk model estimates how likely an applicant is to fail to repay (to default). The answers that most change what you build are: which decision the score drives, what a bad loan costs compared with a lost good customer, when outcomes become known, whether every decline must be explained, and how fast the score must return.

Suggested hour: 60 minutes

Business 15 + risk 15 + operations 15 + constraints 15 = 60. Legal and infrastructure people are often not in the room, so ask the constraints questions of whoever is present and name an owner for each unanswered one.

Questions, and what the answer changes

Worked example: why "accuracy" is the wrong ask

Suppose 5 of every 100 applicants default (illustrative). A model that approves everyone is wrong on 5 and right on 95, so it is "95% accurate" and useless. This is why the cost question and the cutoff question come first.

Worked example: choosing the cutoff by cost (illustrative)

Suppose a good loan earns 100 in margin and a default loses 1,000, so a default costs 10 times the margin. Expected profit per approved applicant with default probability p is (1 - p) x 100 - p x 1,000 = 100 - 1,100p, which is break-even at p = 100 / 1,100, about 9.1%.

Approving bands 1 to 3 earns 49,200. Approving band 4 too drops it to 36,200, so the cutoff sits between bands 3 and 4. If a default cost only twice the margin, break-even would be 100 / 300, about 33%, and every band would be approved, which is why the cost question changes the build.

If time runs short

Ask three questions first: which decision the score drives, what a bad loan costs against a lost good customer, and whether every decline must be explained and which inputs are forbidden. They set the model type, the cutoff and the allowed inputs. The rest can go to written follow-ups.

The requirement I would write down (illustrative numbers)

"Score each new application within 1 second at decision time, using only fields available then. At the same approval rate as today's policy, the bad-loan rate among approved applicants on held-back applications old enough for the outcome window to have matured (for a 12-month window, applications at least 12 months old; for newer ones, the early-warning proxy label) (past applications set aside, not used for training, so the model can be tested on cases it has not seen) must be lower than today's policy, and every decline must carry reason codes."

Pitfalls

• Discussing algorithms before the decision and error costs are agreed.
• Skipping the operations team, then discovering the score arrives after the decision is made.
• Treating one hour as enough: record open questions with owners and dates.

Direct answer

Pick one real story where the request was one line and you made it concrete. Say what you asked, which assumptions you wrote down, and how someone confirmed them. The strongest stories include a moment where an assumption was wrong, caught early.

What to put in each STAR part

STAR means Situation, Task, Action, Result.

• Situation: the vague request, who made it, why it mattered. One or two sentences.
• Task: what you were responsible for delivering and by when.
• Action: three parts. The questions you asked (and why). The assumptions you made explicit, written down with a name against each. How you confirmed: a read-back, a mock-up, or a walk-through with the people affected.
• Result: what shipped, what was avoided, and one thing you would change.

Worked example (a story skeleton; replace the details with your own)

• Situation: a support lead said, "We need a better way to track escalations." No format, no date.
• Task: deliver something usable within two weeks.
• Action, questions: I asked to see the last three escalations that went wrong. They were not lost in tracking: they stalled at the handoff between first-line and second-line support. I asked who needed to see what, and how they would know a handoff had worked.
• Action, assumptions: written on one page, each with an owner: (1) the existing ticket tool is where both lines work; (2) volume is small enough for a daily list; (3) no new personal data is stored.
• Action, confirming: I built a mock spreadsheet, a one-page scope with what was in and out, and walked the support lead and the second-line lead through a real escalation. The second-line lead corrected assumption 1: they worked in a separate tool.
• Result: I added a required handoff field and a daily "unclaimed after 48 hours" list in the tool second line used, and skipped the dashboard first imagined. Support lead's feedback: it was what they meant. Measure I would track: the count of escalations unclaimed after 48 hours, before and after (I would only quote real numbers I had).
• What I would do differently: show the mock-up in the first meeting instead of the second.

Trade-offs and pitfalls

• Generic virtue ("I always ask lots of questions") with no specific question. Name one.
• Hiding the wrong assumption. A caught mistake is evidence of the method.
• Quoting invented percentages. If you have no hard number, describe the outcome plainly.
• Making the stakeholder (anyone with a stake in the outcome, here the support lead and the second-line lead) look foolish. Credit them for the correction.

Direct answer

I would not pick yet. Three vague KPIs (key performance indicators, the numbers a team watches to judge success) such as "engagement", "growth" and "customer happiness" are labels, not metrics. I would ask what decision the metric must drive, turn each label into a measurable definition, and then choose the one that best reflects the launch goal. The other two are dropped or demoted if they do not fit. Separately I pick guardrail metrics (numbers that must not get worse while we chase the primary one). I justify the pick against explicit criteria, not taste.

Questions I ask, in order

1. What will you do differently if this number goes up, or down? (Ship wider, roll back, fund phase two. If nobody can say, the metric is decoration.)
2. What behaviour is the launch supposed to change, for which users? (Names the action the metric must count.)
3. By when must we know? (A 90-day outcome cannot judge a 2-week launch window.)
4. For each KPI, how is it measured today, from what source, and who trusts it? (Separates a product problem from a measurement problem: if the event is not logged, or two dashboards disagree, a "movement" may just be instrumentation, meaning the tracking code that records user actions as events. A change to that code can make a number jump with no change in user behaviour.)
5. What must not get worse? (Produces the guardrails.)

If the stakeholder says "just be data-driven" and has no metrics at all, I accept proxies (stand-in measures close to the real goal): a count straight from a database table, a support-ticket tag, or a manual review of a small sample of sessions. I label these directional, not conclusive.

Choosing and justifying

Worked example (illustrative numbers)

A launch adds a setup checklist to a team-collaboration app. KPIs offered: "engagement", "growth", "happiness".

• Engagement becomes: share of new accounts that create a first project within 7 days. This is the activation rate, the share of new users who reach a first meaningful action. Baseline 30%, target 36% (both illustrative).
• Growth (signups) is driven by marketing, not by this launch, so it fails the "tied to the goal" test.
• Happiness (survey score) is slow and sparse in a 2-week window, so it fails "sensitive in the window".

Primary: activation rate. Guardrails: support tickets per 100 new accounts, and 7-day return rate (share of new accounts that come back in the week after signup), support tickets per 100 new accounts is readable throughout the launch window, while the 7-day return rate, like the 7-day activation rate, is readable for early signups inside the window and for the last signups only about day 21. 30-day retention (share of new accounts still active a month later) is a longer-horizon guardrail: it cannot be observed inside a 2-week window, so I schedule its read for after the window closes and say so up front. Likewise a 7-day activation rate counts an account only once it is 7 days old, so for a 2-week launch the last signups are readable about day 21, and I tell the stakeholder when each number becomes available.

Trade-offs and pitfalls

• A "comprehensive model" that blends several KPIs into one weighted score (for example 40% engagement plus 30% growth plus 30% happiness added into one number) hides trade-offs and cannot be explained to the stakeholder. I prefer one primary metric with guardrail constraints.
• A primary metric with no guardrail can be gamed (activation rises because the checklist nags users).
• Changing definitions after seeing results. I write the definition and target before launch.
• What would change my pick: if the launch is explicitly a retention play, retention becomes primary.

Direct answer

I would stay curious, not dismissive. The stakeholder (a person with a stake in the outcome, here the one asking for the technique) has proposed a solution, not a problem. "Use deep learning" (neural networks with many layers that learn patterns from large amounts of example data) is a proposed solution, and I need the problem behind it first. In the conversation I would say: "Happy to use it if the problem needs it. Help me understand what we want to improve, so I can pick the simplest approach that meets the bar." Then I would learn the need, the current performance, the data and the constraints, and agree to build a simple baseline before committing to the heavy method.

Responding in the room

• Acknowledge the intent: "Sounds like you want something more accurate than today."
• Separate solution from need: "What decision or outcome should improve, and for whom?"
• Offer a cheap test, not a refusal: "Let's run a simple baseline for a week, then see what a bigger model would add."

What I need to learn before agreeing

1. The outcome and decision. What changes when the prediction is right, and what does a wrong one cost? A missed fraud case and a wrongly flagged email cost very different amounts.
2. Today's performance. What does the current process or rule achieve? There is no baseline yet, so the first deliverable is one.
3. The data. How many examples, whether they are labelled (marked with the right answer), how clean they are, and what type they are: images, text or audio versus rows in a table.
4. Constraints. Response time, explainability (can we say why it decided?), privacy or regulatory limits (legal rules on how decisions may be made), and who maintains the model.
5. Why deep learning. Sometimes there is a real reason (a vendor, a leadership goal, images or text as input). Asking is not an attack.

What a simple baseline is

The simplest reasonable method: a rule, a count of the most common outcome, or a plain model such as logistic regression (a simple statistical model that weighs each input and outputs a probability). Baseline and benchmark are different: a baseline is what the simplest method achieves on our own data, while a benchmark is an outside reference such as a published result or a competitor's figure. It sets the bar a heavier model must beat, and it often shows how much signal the data holds at all.

Worked example (illustrative)

A stakeholder wants deep learning to flag urgent support tickets. Support can review about 100 tickets a day, so the metric is how many urgent tickets land in the top 100 flagged. I build a keyword-rule baseline and a simple text model in days. Before looking at results, I agree the decision rule with the stakeholder: "We adopt the heavier model only if it catches clearly more urgent tickets at the same review volume, by a margin worth its extra cost and upkeep." Illustrative numbers from a test week, counting urgent tickets among the top 100 flagged:

The agreed margin is "at least 10 more than the best simple method". Logistic regression adds 7 over the baseline, and deep learning adds 71 - 62 = 9 over logistic regression, so deep learning misses the bar and we ship the simple model. Had deep learning scored 75, it would add 13 and the extra cost would be justified. If the baseline is already close, we ship it and save the effort. If the gap is large, deep learning is justified with evidence.

Trade-offs and pitfalls

• On small, structured tables, simpler models are often competitive, but I would test it, not assert it.
• Refusing outright damages trust. Agreeing without a baseline commits cost with no evidence.
• Agree the threshold before seeing results, so nobody picks the bar after the fact.

Direct answer

A stakeholder's demand (a stakeholder is anyone with a stake in the outcome, such as the business owner or the risk team) for "99.9% precision" is a symptom. First I would find out what the number protects: wasted investigator time, blocked genuine customers, or a regulatory risk. Then I show the trade-off between precision and the share of fraud caught, with the client's own volumes, and propose a target that matches the real cost of each error, usually with a human review stage for borderline cases.

Step 1: find what is behind the number

• "What happens when we flag a legitimate transaction? Who pays?"
• "What happens when we miss a fraud? What does it cost?"
• "How many alerts can your investigators review per day?"
• "Where did 99.9% come from: a regulator, a vendor, a past incident?"
• "Precision of what: all transactions flagged, or only those auto-blocked?"

Precision is the share of flagged transactions that are genuinely fraud. Recall is the share of all fraud that gets flagged. The false positive rate is the share of legitimate transactions wrongly flagged. They are different, and stakeholders often mix them.

Step 2: feasibility with illustrative volumes

Take 1,000,000 transactions in a month, of which 0.2% are fraud: 2,000 frauds and 998,000 legitimate. Fraud is rare, so errors are expensive.

• 99.9% precision means at most about 1 false alarm per 1,000 flags. The model gives every transaction a fraud score, and the threshold is the score above which a transaction is flagged. Raising the threshold shrinks the flagged set and raises precision, but more real fraud scores below the line and goes unflagged. So a threshold strict enough for 99.9% might flag only 1,000 transactions (illustrative), 999 of them real, which catches 999 of 2,000, about 50% of fraud.
• Accuracy hides this. A model that flags nothing is 99.8% accurate here (998,000 of 1,000,000 correct) and catches 0 of the 2,000 frauds, which is why imbalance (one class far rarer than the other) makes accuracy misleading.
• For comparison, suppose the head of risk had asked instead for "15% fraud reduction with under 1% false positives". That gives: 15% of 2,000 is 300 frauds caught. A false positive rate of 1% means 9,980 legitimate transactions flagged. Precision is 300 out of 10,280 flags, about 2.9%. At 0.1% false positives, there are 998 false alarms and precision is 300 out of 1,298, about 23.1%.

So "under 1%" sounds small but may flood the team (10,280 flags a month is more than the 9,000 a month that 300 reviews a day allows), and in this illustration 99.9% precision leaves about half the fraud uncaught (the real share depends on how well the model separates fraud from legitimate transactions, which I would measure on the client's own data).

Step 3: what I propose instead

A tiered design: auto-block only above a very high score (precision near the stated figure on that slice), send the mid-score band to human review within the team's daily capacity, and allow the rest. Report precision and recall at each tier. Example target (illustrative figures): "At a review capacity of 300 alerts per day, catch at least 60% of fraud by value, with auto-block precision of at least 99%". I would confirm the 60% against a measured baseline before committing to it.

Making a target testable

State the baseline period (for example the last 12 months), how fraud labels were created (chargebacks, where a cardholder disputes a charge and the bank takes the money back, arrive weeks late, so recent months look cleaner than they are), and sampling bias (the labelled data is a skewed sample: labels exist only for transactions that were investigated or not blocked). Use a held-out time period, meaning train on older months and test on later months the model never saw, not a random split that shuffles all rows together and lets the model peek at future patterns.

Other requirements to raise during scoping

Imbalance and drift risk apply to any fraud model. Interpretability, fairness and audit needs apply when the model also drives lending decisions, is used by field agents, or runs as an internal tool.

• Trustworthy lending model: ask about interpretability (can a person see why the model decided what it did), fairness checks across groups (comparing error rates for different groups of customers), and audit and record-keeping needs.
• Interpretability for field agents: what reason would an agent need to see to act on a flag?
• Auditable internal tool: log every score, model version and decision, so a case can be reviewed later.
• Imbalance and drift risk: rare fraud makes accuracy misleading, and fraudsters change behaviour, which makes the model's scores less reliable over time (drift), so agree monitoring and retraining triggers.

Trade-offs and pitfalls

• Promising a number before measuring a baseline.
• Optimising precision alone, which silently drops recall.
• Ignoring the cost of review time and customer friction.
• Evaluating on random splits when fraud patterns shift over time.