Amazon Data Scientist (Staff Level) Interview Preparation Guide

Direct answer
The same underlying story can honestly serve different companies' principle vocabularies, because a real story usually demonstrates more than one trait at once. The skill is choosing which true facet to lead with, and phrasing the takeaway in that company's specific language, without changing what actually happened.

Structured elaboration

1. Identify the story's multiple honest facets first. Most real stories touch two to four traits at once; a single incident might show both ownership and appropriate urgency, for instance.
2. For each target company, identify which facet of the story maps most naturally to that company's specific vocabulary and emphasis.
3. Write a one-sentence takeaway per company that leads with that facet, without inventing detail that wasn't true.
4. Be ready to explain, if asked directly, why you emphasized it that way for that audience. A candid answer to that follow-up is itself a good sign of self-awareness, not a weakness to hide.

Worked example
Consider a story about restoring a degraded service faster than the standard process would have, by trusting a well-reasoned read of the situation rather than escalating and waiting. For a company whose published language centers ownership and thoroughness, the one-sentence takeaway leads with taking full ownership of a problem outside the formal escalation path and following through on the root cause afterward. For a company whose language centers speed and bias toward appropriate action, the same story's takeaway instead leads with making a fast, well-reasoned call under uncertainty rather than waiting for permission. Both are true descriptions of the same incident; only the foregrounded facet changes.

Trade-offs and pitfalls
This only works when a story genuinely supports multiple facets; forcing a single-facet story to serve an unrelated principle produces something that falls apart under a follow-up question. This is a different concern from reusing the exact same story too many times within a single interview loop at one company, where interviewers compare notes afterward; tailoring across different employers, which is what this skill addresses, is not the same risk as repeating a story too often within one loop. Overclaiming detail that wasn't true in order to fit an audience is dishonest, and it tends to surface under a probing follow-up question.

Direct answer

Pick measures by working backwards from the money: start with the business outcome the sponsor cares about, then choose a few measures (four to six is a workable ceiling) that each trace to cash flow, each with one named owner, a data source and a review rhythm. For an automation or analytics project, use a mix of impact measures (what changed in the business) and adoption measures (whether people actually use the thing), because impact without adoption is a forecast and adoption without impact is a vanity number.

Selection criteria: what makes a measure worth keeping

1. Traces to cash flow: you can finish the sentence "if this moves by X, cash changes by Y dollars, through this mechanism". If not, it is a vanity metric (a number that looks good but changes no decision or cash, such as "dashboards built").
2. Has an owner who can change it: the person is accountable and controls the behaviour or process behind the number.
3. Measured the same way before and after: a baseline exists, or can be created before go-live.
4. Available at a useful frequency, with known quality: weekly usage logs are fine for adoption; a quarterly finance close is fine for cost. Data that arrives six months late cannot steer anything.
5. Hard to game: if people can hit the number without helping the business (closing tickets early, say), pair it with a counter-measure, a second measure that exposes the cheat, such as reopen rate (the share of closed tickets that come back open): closing tickets early lifts the first number and the reopen rate catches it.

Then cut the list: if two measures move together, keep the one nearer the cash.

Worked example: an invoice-automation project

The sponsor's goal is lower cost of accounts-payable processing. One measure traced to cash: minutes per invoice, with 40,000 invoices a year, falling from 12 to 5 minutes, at a loaded cost (salary plus overheads) of USD 38 per hour.

INVOICES_PER_YEAR = 40_000
MINUTES_BEFORE, MINUTES_AFTER = 12, 5
LOADED_COST_PER_HOUR = 38
minutes_saved = INVOICES_PER_YEAR * (MINUTES_BEFORE - MINUTES_AFTER)
hours_saved = minutes_saved / 60
print(f"Minutes saved per year: {minutes_saved:,}")
print(f"Hours saved per year: {hours_saved:,.0f}")
print(f"Value if every hour becomes real cash: ${hours_saved * LOADED_COST_PER_HOUR:,.0f}")
for realised in (0.25, 0.5):
    print(f"Value if {realised:.0%} of the hours are redeployed or avoided as hiring: ${hours_saved * LOADED_COST_PER_HOUR * realised:,.0f}")

Output:

Minutes saved per year: 280,000
Hours saved per year: 4,667
Value if every hour becomes real cash: $177,333
Value if 25% of the hours are redeployed or avoided as hiring: $44,333
Value if 50% of the hours are redeployed or avoided as hiring: $88,667

The result shows why "hours saved" is not the same as dollars: USD 177,333 is the ceiling if every freed hour becomes cash. If only a quarter of the freed time is redeployed or avoids hiring, the cash value is USD 44,333. So the set of KPIs (key performance indicators, the handful of numbers you manage by) needs one measure for the process change (minutes per invoice) and one for the financial conversion (the budgeted cost line, or hours redeployed to named work).

A starter set and who owns each

For an analytics platform, replace "invoices processed" with "decisions or reports that now use the platform" and add a measure of how often a decision changed because of it. For a migration, six reasonable measures in the first 12 months, and the stakeholder each one is for:

Trade-offs and pitfalls

• Too many measures: more than about six and nobody owns any of them.
• Activity measures dressed as outcomes: "dashboards built" says nothing about value.
• An owner who cannot act: a measure owned by the data team but driven by the sales team will not move.
• No counter-measure: a pure speed metric invites quality loss.

Direct answer

When a WHERE clause mixes AND and OR conditions, correctness depends on how the boolean expression is grouped, not on the order you physically write the conditions in. SQL evaluates the whole boolean expression by its precedence rules (AND binds tighter than OR), and a NULL-aware condition (IS NULL / IS NOT NULL) has to be used deliberately, since NULL never satisfies = or <>. Get the grouping or the NULL handling wrong and the query silently returns the wrong rows with no error.

Structured elaboration

Three pieces make a mixed WHERE clause correct:

1. Equality flag: is_paid = TRUE is a plain boolean check. It works as expected on TRUE and FALSE rows (a NULL is_paid row is excluded too, since NULL = TRUE evaluates to unknown, not true).
2. Date range: prefer a half-open interval, order_date >= start AND order_date < end, over BETWEEN. BETWEEN is inclusive on both ends, which can double-count a row that lands exactly on the boundary between two adjacent ranges.
3. NULL-aware condition: any column that can be NULL (a cancellation timestamp, a discount code) needs IS NULL / IS NOT NULL, not = NULL or <> value. column <> value evaluates to NULL (not TRUE) for a NULL row, so that row is silently dropped even when the intent was to include it.

Where correctness actually gets lost is precedence: AND binds tighter than OR. WHERE a AND b AND c OR d parses as (a AND b AND c) OR d, not a AND b AND (c OR d). Whenever AND and OR sit in the same WHERE clause, wrap the OR branch in explicit parentheses. That grouping is the "structure" that matters. Physical execution order does not: the optimizer is free to reorder predicates as long as the logical result is preserved.

Worked example

Sample data (orders), tested in DuckDB 1.5:

Goal: paid orders placed in Q1 2024 that are either not cancelled, or cancelled but refunded.

Correct (OR branch parenthesized):

SELECT order_id
FROM orders
WHERE is_paid = TRUE
  AND order_date >= DATE '2024-01-01'
  AND order_date <  DATE '2024-04-01'
  AND (cancelled_at IS NULL OR refunded = TRUE)
ORDER BY order_id;

Result: order_id = 1, 2 (2 rows). Order 3 is correctly excluded (cancelled and not refunded), order 4 is excluded (too early), order 5 is excluded (unpaid), order 6 is excluded (too late).

Buggy (same conditions, parentheses removed around the OR):

SELECT order_id
FROM orders
WHERE is_paid = TRUE
  AND order_date >= DATE '2024-01-01'
  AND order_date <  DATE '2024-04-01'
  AND cancelled_at IS NULL OR refunded = TRUE
ORDER BY order_id;

Result: order_id = 1, 2, 5 (3 rows). Order 5 leaks in even though it is unpaid and from March, because the clause is now (is_paid AND date_range AND cancelled_at IS NULL) OR refunded = TRUE, and refunded = TRUE alone satisfies the second half of the OR.

Trade-offs & pitfalls

• Always parenthesize an OR branch that sits inside a longer AND chain, even when the grouping "looks" unambiguous. The next person editing the query may not preserve your intended precedence.
• Prefer half-open date ranges (>= start AND < end) over BETWEEN for timestamp columns, to avoid double-counting or off-by-one-day gaps caused by boundary values.
• column <> value silently excludes NULL rows. If NULLs should be included, add OR column IS NULL explicitly, do not assume <> covers it.
• Some engines will not warn on a missing parenthesis. The query just runs and returns a wrong but plausible-looking row count, which is why this bug class tends to survive casual review.

Direct answer

Do not build "the model, then the system, then the rollout" as layers. Cut the product into vertical slices (each a thin path from data to a decision a person or system acts on), and order them so each slice either delivers value or produces something the next slice needs. For fraud, the scarce thing is labeled outcomes, so the first slice is a shadow-mode rules baseline (rules that run on live transactions and log what they would flag, without blocking anything) feeding a human review queue. It is useful on day one and starts generating labels.

Structured elaboration

Why labels drive the order. Fraud labels are confirmed late (a chargeback, the reversal a customer's bank makes after the customer disputes a card payment, can arrive weeks after the transaction; card networks typically allow disputes up to about 120 days after the payment) and fraud is rare, so a model trained too early learns from little and stale evidence. Also, transactions you block never show whether they were fraud, which biases later training data.

Slices, in order

1. Rules plus logging, in shadow. A handful of explicit rules (for example many attempts per card per hour) send transactions to an analyst queue without blocking. Value: analysts catch fraud now. Learning: base rate (how often fraud occurs, here 0.2%), review capacity, first labels.
2. Labeled dataset and offline evaluation. Time-based train and test split (train on the past, test on the future). Judge at fixed review capacity (recall among the top K flagged per day, where K is the number of cases analysts can review, for example 500), because with rare fraud accuracy is meaningless: a model that flags nothing is 99.8% accurate when only 0.2% of transactions are fraud, yet it catches no fraud at all.
3. First model in shadow. A simple tabular model (such as gradient-boosted trees, a common model type that combines many small decision trees) scores every transaction, logged but not acted on. Learning: lift over the rules (how much more fraud the model catches than the rules at the same review load), measured on later-arriving labels.
4. Model-ordered review queue. Analysts see highest scores first. Value: same staff catch more fraud.
5. Automated action on a narrow slice. Only the highest-confidence band, preferring a softer action such as step-up verification (asking for extra proof of identity) over outright decline, with a small random holdout (a group deliberately left alone and never blocked) so you can still measure.
6. Feedback loop and monitoring. Retraining cadence, drift alerts (the data changing so the model slowly gets worse), fraudsters adapting, the holdout kept running.

Worked example

Illustrative numbers: 100,000 transactions a day, 0.2% fraud, so 200 fraudulent. Analysts can review 500 a day. Precision is the share of flagged transactions that are truly fraud; recall is the share of all fraud that gets flagged.

• Rules fill the queue with 60 true frauds: precision 60/500 = 12%, recall 60/200 = 30%.
• A model filling the same 500 slots with 110 true frauds: precision 110/500 = 22%, recall 110/200 = 55%.
  That comparison, same staff and a bigger catch, is the evidence for slice 4.

Variation: a two-month personalized-ranking plan in four two-week sprints. Personalized ranking means ordering what each user sees (for example products in a feed) by predicted interest. The same logic applies: ship a baseline first, then earn each model step. Terms in the table: an impression is one time an item was shown to a user; implicit-label bias means clicks only exist for items the old ranking chose to show, so the data reflects the old ranking; a model registry is a versioned store of trained models that lets you roll back; an A/B test shows the new ranking to a random half of users and the baseline to the other half, then compares.

Trade-offs and pitfalls

• Model-first sequencing is the common trap: it spends months before any value or any label.
• Horizontal layers (all data work, then all modeling) delay learning; each slice must reach a decision.
• False positives cost real customers, so the automation slice needs a guardrail on declined good transactions.
• What would flip the order: if years of labeled history already exist, start at slice 2 or 3.
• Regulations or explainability needs may require a simpler model or a human in the loop longer than technical readiness alone suggests.

Direct answer (what I would say to the CEO)

"Growing faster means paying to acquire customers before they have paid us back. Spend more and the customer base grows, but profit this quarter gets worse, because the money goes out now and comes back over many months. The question is not 'growth or profit' but 'how long until each extra customer pays back what it cost, and can we afford to wait?' If payback is short, customers stay, and we have the cash, I would spend. If not, I would not."

Terms in plain language

• CAC (customer acquisition cost): marketing and sales spend divided by the new customers it brought in.
• Contribution margin: what is left of a customer's payment after the direct costs of serving them (hosting, support, payment fees).
• Churn: the share of customers who leave each month.
• Marginal cost: the cost of the next customer you add, as opposed to the average over all customers so far. Here the next 1,000 customers cost $50 each while the average is $40.
• Incremental: customers who joined only because of the extra spend, not people who would have signed up anyway. Only incremental customers justify the spend.
• Payback: months until a customer's contribution adds up to what we paid to win them.

A numerical example (illustrative)

Each customer pays $10 a month and keeps 70% after direct costs, so contributes $7 a month. Five percent of customers leave each month. Today we spend $30,000 a month to win 1,000 customers ($30 each). To win 2,000 we spend $80,000, because the extra 1,000 cost $50 each (the cheap audiences are already used up). So the next customers are far more expensive than the average.

What the extra 1,000 customers return, month by month, allowing for 5% leaving each month:

• First 3 months: $7 x (1 + 0.95 + 0.9025) = about $20 per customer, so roughly $20,000 back against $50,000 spent. This quarter's profit is about $30,000 worse.
• Payback: about 8.6 months, so the extra spend is earned back in the 9th month.
• After 12 months: $64,350 back, so the spend is $14,350 ahead.
• Over a customer's whole life: $7 / 0.05 = $140 each, which is 2.8 times the $50 paid.

When I would still do it

1. Payback fits inside cash. We can fund the $30,000 hit without shortening the time we can keep operating (runway).
2. Retention holds for new audiences. The most important sensitivity is churn: at 8% instead of 5%, payback stretches to about 10.2 months and lifetime return falls to $87.50 per customer (1.75 times the cost). Customers from a pricier audience often leave faster, so I check month-2 retention (the share of the new group still paying in their second month) before scaling further.
3. There is a reason to hurry: a rival is grabbing the same customers, switching is costly so early customers are sticky, or each user makes the product better for others.
4. We can stop quickly: spending is monthly and can be cut if retention disappoints.

I would say no if payback is longer than we can fund, if retention for new customers is unproven, or if the extra spend only buys customers we would have won anyway.

Pitfalls

• Using the average cost ($40) instead of the cost of the next customer ($50), which hides the diminishing returns.
• Treating lifetime value as certain when it rests on one churn number.
• Calling any spend 'investment' without a payback date.

Direct answer. Good version-control hygiene means every commit and PR tells a clear, minimal, reviewable story: small commits with messages that explain WHY, PRs scoped to one reviewable change, and a branching approach the whole team actually follows consistently.

Commit granularity and messages

• Each commit should represent ONE logical change that could, in principle, be reverted independently without breaking something unrelated -- a commit that mixes a bug fix with an unrelated formatting sweep makes both harder to review and harder to revert cleanly later.
• Good message: Fix race condition in order total calculation (see INC-204)\n\nTwo concurrent requests could both read the stale total before either\nwrite committed. Wrap the read-modify-write in a transaction. -- explains WHY (the bug, the mechanism) not just WHAT (which is visible in the diff already).
• Bad message: fix bug -- tells a future reader (including git blame six months from now) nothing they couldn't already see from the diff itself, and gives zero context for WHY this change was needed.

PR size and branch/PR conventions

• Smaller PRs get reviewed faster and more thoroughly -- a reviewer can hold a 100-line diff in their head; a 2,000-line diff gets a rubber-stamp approval because nobody can meaningfully review it in one sitting.
• A consistent branch-naming convention (fix/order-total-race, feature/bulk-export) makes it easy to scan open branches and understand what's in flight without opening each one.
• Rebasing versus merging is a team-level convention choice (rebase keeps history linear and easier to bisect; merge preserves the exact chronological record) -- the specific choice matters less than the TEAM actually agreeing on and following one consistently, so history reads predictably regardless of who wrote it.

Handling large binary or generated files

• Committing large binaries (design assets, model weights, video fixtures) directly into a normal git history bloats every future clone and slows down operations like git log and git blame for the whole team, forever, even after the file is deleted, since git history keeps every version.
• Use Git LFS (or an equivalent large-file extension) for binaries that genuinely need to be version-controlled alongside code, so the main repository only stores a lightweight pointer.
• For anything that can be REGENERATED from source (build output, compiled assets, lockfile-derived artifacts), keep it out of version control entirely via .gitignore rather than committing it and then fighting merge conflicts on a file nobody hand-edits.
• If large files were already committed by mistake, history-rewriting tools (git filter-repo) can remove them retroactively, but that rewrites shared history and needs the same coordination caution as any other history rewrite on a branch others have pulled.

Why this matters for maintainability specifically

Git history is a maintainability tool in its own right: git blame and git log are often the FASTEST way to understand why a confusing piece of code exists, but only if commit messages actually explain the why -- a history of 'fix bug', 'wip', 'more fixes' gives future maintainers nothing to work with when they're trying to understand a decision made months or years ago.

Trade-offs and pitfalls

• Enforcing small PRs can pressure people to under-scope a change that's genuinely indivisible (e.g., a schema migration that must ship atomically with the code that depends on it) -- the goal is REVIEWABLE size, not an arbitrary line-count limit that ignores what a change actually requires.
• Rewriting history (interactive rebase, squashing) before merging to clean up a messy WIP trail is generally good practice, but doing it on a SHARED branch others have already pulled causes real pain -- keep history rewrites scoped to your own not-yet-shared branch.

Direct answer: At hundreds of millions of rows, a windowed DELETE run in small batches and a build-a-deduplicated-copy-then-swap approach are the two realistic options, and they trade off differently: batched deletes touch the live table in place with many short transactions, keeping the table available throughout but taking longer overall and needing careful batching to avoid long-held locks; copy-and-swap builds a clean table from scratch and swaps it in with one fast rename, which is much simpler to reason about and to back out of, at the cost of needing roughly double the disk space and a brief write-blocking window at swap time.

Structured elaboration

Batched DELETE, in detail.

-- Step 0: snapshot before touching anything
CREATE TABLE duplicates_backup AS SELECT * FROM duplicates;

-- Repeat until 0 rows affected: delete a bounded batch of losers per run
WITH to_delete AS (
  SELECT id
  FROM (
    SELECT id,
      ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at ASC, id ASC) AS rn
    FROM duplicates
  ) d
  WHERE d.rn > 1
  LIMIT 1000
)
DELETE FROM duplicates WHERE id IN (SELECT id FROM to_delete);

Run this in a loop from outside the database (a script that keeps calling it until the affected-row count is 0), each iteration as its own short transaction. This keeps any single lock hold time bounded and predictable, which matters on a table other processes are actively reading from or writing to.

Copy-and-swap, in detail.

CREATE TABLE duplicates_new AS
SELECT * FROM (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at ASC, id ASC) AS rn
  FROM duplicates
) d
WHERE rn = 1;

-- rebuild indexes/constraints on duplicates_new to match the original, then:
ALTER TABLE duplicates RENAME TO duplicates_old;
ALTER TABLE duplicates_new RENAME TO duplicates;
-- keep duplicates_old around until confident, then drop it

Keeping the N most recent rows instead of exactly 1. Both approaches generalize directly: change the filter from rn = 1 (or rn > 1 for the delete side) to rn <= 2 / rn > 2 to keep the two most recent rows per key instead of just the latest one. Nothing else about the mechanism changes; it's the same ROW_NUMBER partition, just a different cutoff.

SQL-in-warehouse versus an external job (e.g. Spark) at this scale. For a table already living in a columnar warehouse (Snowflake, BigQuery, Redshift), the copy-and-swap pattern above, run as warehouse SQL, is usually simpler and cheaper: the warehouse's own engine already parallelizes the scan and sort, and you avoid the cost of exporting hundreds of millions of rows to an external cluster and reimporting the result. A separate Spark job earns its complexity when the source table lives in a system without a native, cheap CREATE-TABLE-AS-SELECT-and-swap primitive, or when the dedup logic needs to join against data that only exists outside the warehouse (e.g. an external reference dataset too large to load in cheaply).

Backup, transaction, and rollback considerations for each

• Backup. Snapshot before any destructive operation regardless of approach: a full table copy for copy-and-swap doubles as its own backup (the old table, kept around); for batched delete, take an explicit backup table or rely on point-in-time recovery / WAL retention, since there's no natural 'old table' left behind.
• Transactions. Batched delete needs many small transactions by design, to bound lock duration; copy-and-swap needs exactly one short transaction at the rename step, with the expensive build work happening outside any long-held lock.
• Rollback. Copy-and-swap rollback is a second rename back to the original table, essentially instantaneous. Batched-delete rollback means restoring from the backup table (or WAL-based point-in-time recovery), which is slower and coarser-grained, since individual batches aren't usually tracked well enough to undo just one.

Trade-offs & pitfalls

• Copy-and-swap needs roughly double the storage for the duration of the operation (old + new table coexist); on a genuinely huge table this can be the deciding constraint even when it's operationally simpler.
• A unique constraint (enforced via ON CONFLICT DO NOTHING on PostgreSQL/SQLite, or the dialect's equivalent conflict handling) prevents new duplicates from being written after cleanup, but only after you've already removed the existing ones through one of the two methods above; adding the constraint first, before cleanup, just fails outright on the existing duplicate rows.
• Disable or carefully sequence foreign-key-dependent triggers/cascades before batched deletes on a table other tables reference; deleting a 'loser' row that a child table still points to either fails the batch or silently cascades data loss, depending on the constraint's ON DELETE behavior.

Approach: aggregate counts by experiment_id/team_id and variant, compute conversion rates per variant, then calculate absolute difference and z-score for difference in proportions. I require at least 1,000 users in each variant (adjustable if you meant total users).

WITH agg AS (
  SELECT
    experiment_id,
    team_id,
    SUM(users) FILTER (WHERE variant = 'control') AS control_users,
    SUM(conversions) FILTER (WHERE variant = 'control') AS control_conversions,
    SUM(users) FILTER (WHERE variant = 'treatment') AS treatment_users,
    SUM(conversions) FILTER (WHERE variant = 'treatment') AS treatment_conversions
  FROM ab_results
  GROUP BY experiment_id, team_id
)
SELECT
  experiment_id,
  team_id,
  -- conversion rates
  control_conversions::float / NULLIF(control_users,0) AS control_rate,
  treatment_conversions::float / NULLIF(treatment_users,0) AS treatment_rate,
  -- absolute difference
  ABS(
    (treatment_conversions::float / NULLIF(treatment_users,0))
    - (control_conversions::float / NULLIF(control_users,0))
  ) AS absolute_delta,
  -- z-score for difference in proportions (uncorrelated samples)
  (
    (treatment_conversions::float / NULLIF(treatment_users,0))
    - (control_conversions::float / NULLIF(control_users,0))
  ) / NULLIF(
    SQRT(
      (treatment_conversions::float / NULLIF(treatment_users,0)) * (1 - treatment_conversions::float / NULLIF(treatment_users,0)) / NULLIF(treatment_users,0)
      +
      (control_conversions::float / NULLIF(control_users,0)) * (1 - control_conversions::float / NULLIF(control_users,0)) / NULLIF(control_users,0)
    ), 0
  ) AS z_score
FROM agg
WHERE control_users >= 1000
  AND treatment_users >= 1000;

Key points:

• Uses FILTER and SUM to pivot variants in one pass.
• NULLIF avoids division by zero.
• Z formula: (p_t - p_c)/sqrt(p_t(1-p_t)/n_t + p_c(1-p_c)/n_c) — appropriate for independent samples.
  Edge cases / notes:
• If you meant total users (control + treatment) >= 1000, change WHERE to (control_users + treatment_users) >= 1000.
• For very small rates or low counts consider using continuity corrections or exact tests (Fisher) / Bayesian methods.

Direct answer

In seven days, the goal isn't a polished model, it's a scoped, deployable slice that proves the concept and gives the team something real to iterate on; I'd spend most of the time on scoping and the simplest possible baseline, not on modeling sophistication.

Structured elaboration

1. Days 1 to 2: scope and data check. Nail down the target, the success metric, and confirm the data actually exists in usable form; this is where most seven-day projects actually lose time if skipped.
2. Day 3: pick the simplest defensible approach. A rule-based or simple statistical baseline first, not because it will win long-term, but because it gives you a floor to compare against and something deployable almost immediately.
3. Days 4 to 5: build and validate the baseline. Get it working end-to-end (including serving, however minimal) before spending any remaining time on a fancier model, since an undeployed sophisticated model delivers zero value.
4. Day 6: define acceptance criteria with the team. What does "good enough to ship" mean for this minimum viable product (MVP), explicitly, so the team isn't stuck in an open-ended improvement loop.
5. Day 7: ship and set the iteration plan. Deploy the simplest working version and write down the two or three things you'd improve next, so the team has a clear backlog rather than a "done" project that quietly needs more work.

Worked example

For a first version of an AutoML-versus-custom-build decision under this kind of time pressure, using an off-the-shelf AutoML tool to get a working baseline in two days, rather than hand-building a custom pipeline, is often the right seven-day call, even if a custom model would eventually outperform it, because it buys you a real deployed system to learn from immediately.

Trade-offs and pitfalls

The main risk is spending too much of the seven days on modeling sophistication and not enough on deployment, ending up with a great offline number and nothing shippable. The opposite risk is shipping something so minimal it can't actually demonstrate the value of the idea, which wastes the exercise in the other direction.

Direct answer

At the scale of thousands of experiments and hundreds of metrics each, the design has to separate metrics into tiers by decision importance (a handful of primary, ship/no-ship metrics per experiment vs. many secondary and exploratory ones), apply a stricter false discovery control to the important tier and a looser, FDR-based control to the exploratory tier, and back all of it with an immutable experiment registry so results are reproducible and auditable. Interpretability comes from tiering and clear reporting conventions, not from picking one universal statistical procedure for every metric.

Structured elaboration

Pipeline flow

flowchart LR
  A[Experiment registry: pre-registered plan, metrics, hypotheses] --> B[Metric pipeline: batch-compute per experiment]
  B --> C{Metric tier}
  C -->|Primary, ~1-3 per experiment| D[Per-experiment alpha, no correction needed within this small set]
  C -->|Secondary/exploratory, hundreds| E[Benjamini-Hochberg within the metric family]
  D --> F[Results store: adjusted + unadjusted, effect size, confidence interval]
  E --> F
  F --> G[Dashboard: verdict, q-value, effect size by segment]
  F --> H[Continuous monitoring: realized FDR, power, registry-plan mismatches]

Statistical procedures

• Primary metrics (the one to three metrics that actually decide ship/no-ship for a given experiment) are tested at the experiment's own pre-registered $\alpha$, since there are only a few of them per experiment and they were chosen before the data was seen; multiplicity correction here is unnecessary if the primary set stays small and is genuinely fixed in advance.
• Secondary and exploratory metrics (the hundreds of additional metrics reported per experiment) are controlled with Benjamini-Hochberg (BH) within that family, since the goal there is to surface promising leads for follow-up, not to make a single high-stakes decision, and FDR control keeps a bounded, known fraction of those leads spurious.
• Cross-experiment aggregation (e.g. "how many experiments this quarter showed a significant lift on metric X") is its own separate family and needs its own correction; don't let a metric's significance in one experiment's local family imply anything about a cross-experiment claim without re-testing at that level.

Data infrastructure

• An experiment registry that stores the hypothesis, the declared primary and secondary metrics, the randomization scheme, and the analysis plan before the experiment starts, and treats that record as immutable once registered.
• Raw event logging (not just pre-aggregated metrics), so any later question about what actually happened can be answered by recomputation rather than trusting a snapshot.
• Reproducible, versioned pipelines (a workflow orchestrator with pinned code, data snapshots, and seeds) so a reported result can be regenerated exactly.

Reporting conventions

• One decision-relevant verdict per primary metric per experiment (ship / hold / inconclusive), always paired with its effect size and confidence interval, not the p-value alone.
• Secondary and exploratory metrics reported with their BH-adjusted q-value alongside the raw p-value, explicitly labeled as hypothesis-generating rather than confirmatory.
• A short plain-language note translating each significant primary result into business terms, so the interpretability requirement is met at the reporting layer, not just the statistical one.

Worked example

A single experiment reports 12 exploratory secondary metrics with unadjusted p-values: $0.0009, 0.006, 0.041, 0.09, 0.14, 0.19, 0.27, 0.33, 0.41, 0.58, 0.71, 0.85$. Comparing Bonferroni (family $\alpha=0.05$) against BH (target FDR $q=0.10$):

(both computed directly from the sorted p-values and the two threshold rules above)

At platform scale, this difference compounds: across thousands of experiments each reporting hundreds of exploratory metrics, a flat Bonferroni-style correction across everything would leave the exploratory tier almost powerless, while a per-experiment BH pass on just that tier keeps a known, bounded false-discovery share without needing an ever-shrinking per-test threshold as the platform grows.

Computational scalability

• Metric computation is embarrassingly parallel per experiment and per metric; pre-aggregate with a batch or streaming pipeline, and only run the multiplicity correction step (cheap, since it's just a sort and threshold comparison) once all p-values for a family are in.
• The correction step itself is $O(m\log m)$ per family (dominated by sorting the p-values), trivial compared to the cost of computing the underlying statistics; scalability bottlenecks are almost always in the metric computation and data pipeline, not the multiplicity correction.
• Reserve heavier machinery (empirical-Bayes shrinkage, hierarchical pooling across similar experiments) for cases where the added interpretability or power is worth the added modeling and validation cost; default to plain BH within each experiment's exploratory family as the baseline that always works.

Monitoring strategies

• Backtest the pipeline on historical experiments with known outcomes (including synthetic A/A tests, where the two arms are identical) to confirm the realized false-positive rate matches what the correction claims.
• Continuously track the realized discovery rate over time; a sudden jump is a signal of either a real shift in how many effects are real, or a broken pipeline inflating false positives.
• Alert on registry-plan mismatches (a primary metric changed after the experiment started, or a metric reported that wasn't pre-registered), since that's the most common way multiplicity control gets silently defeated.

Trade-offs & pitfalls

• Applying a single strict correction (Bonferroni across everything, primary and secondary combined) is the safest failure mode but starves the exploratory tier of power at scale; applying no correction anywhere invites exactly the selective-reporting problem multiplicity control exists to prevent. The tiered approach is a deliberate middle ground, not a shortcut.
• Letting a metric drift from "exploratory" to "primary" after seeing a promising result defeats the whole design; the tier has to be fixed by the pre-registration, not re-assigned post hoc.
• BH's guarantee weakens under strong negative dependence between metrics; correlated metrics (which are common when many metrics are derived from the same few underlying events) may need a dependence-aware variant or an empirical-Bayes model rather than plain BH.
• None of this replaces basic experiment hygiene: a broken randomization or a biased sample invalidates every downstream statistical control, no matter how carefully the multiplicity correction is designed.