Amazon Data Analyst Interview Preparation Guide - Junior Level (1-2 Years)

Quick answer

Each user receiving 10 independent emails, each clicked with probability $p = 0.3$, is a Binomial$(n=10, p=0.3)$ model. The probability of at least 3 clicks is easiest to get via the complement of 0, 1, or 2 clicks, which comes out to about 61.7%. The expected number of clicks is $np = 3$, with variance $np(1-p) = 2.1$.

Setup

$X \sim \text{Binomial}(n=10,\ p=0.3)$, with PMF:

(a) $P(X \ge 3)$

Computing the complement is less arithmetic than summing 8 terms directly:

from math import comb

n, p = 10, 0.3
p0 = comb(n, 0) * p**0 * (1 - p)**10
p1 = comb(n, 1) * p**1 * (1 - p)**9
p2 = comb(n, 2) * p**2 * (1 - p)**8
p_ge3 = 1 - (p0 + p1 + p2)
print(f"P(X=0)={p0:.6f}, P(X=1)={p1:.6f}, P(X=2)={p2:.6f}")
print(f"P(X>=3)={p_ge3:.4f}")
# P(X=0)=0.028248, P(X=1)=0.121061, P(X=2)=0.233474
# P(X>=3)=0.6172

$P(X \ge 3) \approx 0.6172$: roughly 61.7% of users click at least 3 of the 10 emails.

(b) Expected value and variance

For a Binomial random variable:

Standard deviation $= \sqrt{2.1} \approx 1.449$.

Interpretation

An average user clicks 3 of the 10 emails, with a standard deviation of about 1.45 clicks, so a typical user's click count lands somewhere between about 1.5 and 4.5. The fact that over 60% of users clear the "at least 3" bar despite the average also being 3 reflects the right-skew-free, moderately spread-out shape of this particular Binomial (n=10, p=0.3 isn't close enough to either 0 or 1 to be heavily skewed).

Trade-offs and pitfalls

• Independence across the 10 emails is the load-bearing assumption. If a user who clicks one email is more likely to click the next (engagement momentum) or less likely (fatigue), the true distribution is overdispersed or underdispersed relative to Binomial, and the variance formula above will be wrong.
• $p=0.3$ is itself an estimate, not a known constant. In practice $p$ is estimated from historical click data, and the Binomial calculation above doesn't account for uncertainty in that estimate; a more careful treatment would put a Beta prior on $p$ and integrate over it (a Beta-Binomial), widening the resulting probability estimates.

Direct answer

When a user can belong to several behavioral segments at once, such as being a power user, a mobile-only user, and a member of a specific acquisition channel simultaneously, attributing a feature's adoption impact to any single segment requires either analyzing the segments' intersections directly or using a modeling approach that can separate overlapping effects, rather than reporting each segment's adoption rate independently as if the segments did not overlap.

Structured elaboration

The simplest approach is to build an intersection table: instead of reporting adoption for "power users" and "mobile-only users" as two separate rows, report adoption for each meaningful combination (power users who are also mobile-only, power users who are not mobile-only, and so on), which makes any interaction between the segments visible rather than hidden inside two overlapping averages. This works well when there are only a handful of segments and their combinations remain large enough to be statistically meaningful, but it breaks down quickly as more segments are added, since the number of combinations grows multiplicatively while each combination's sample size shrinks.

For a larger number of overlapping segments, a regression-style approach, where segment membership is entered as a set of indicator variables and the model estimates each segment's association with the outcome while holding the others constant, isolates each segment's apparent effect net of the others in a way a simple table cannot. This is still an associational, not necessarily causal, estimate unless segment membership is itself randomized or quasi-randomized, so a genuinely causal claim about a segment's effect on adoption typically requires an experiment where the feature is deliberately varied within a segment while holding the other segment memberships as similar as possible between the exposed and unexposed groups.

Worked example

Consider a feature rollout where adoption looks strong overall (35%) and looks especially strong among "power users" (60% adoption) and among "mobile-only users" (52% adoption) taken separately. If power users and mobile-only users overlap heavily, say 70% of power users are also mobile-only, a naive read might credit the feature with independently winning over two different audiences, when in fact a regression controlling for both segment memberships at once might show that being mobile-only carries almost no independent association with adoption once power-user status is accounted for, meaning the apparent "mobile-only effect" was really being driven by the fact that power users happen to skew heavily mobile-only, not by anything specific to the mobile-only experience.

Trade-offs and pitfalls

The main pitfall is exactly the scenario above: reporting overlapping segments' adoption rates independently and drawing two separate causal-sounding conclusions from what is really one underlying effect appearing twice. A second pitfall specific to the regression approach is that it can only control for the segments actually included as variables, so an unmeasured factor correlated with both segment membership and adoption (a confound not represented in the model) can still distort the isolated estimates, which is why a genuine causal claim ultimately needs an experiment rather than segment-level regression alone.

Direct answer

A novelty effect is a temporary inflation of an early treatment effect: users explore or click on something purely because it is new, and that extra engagement fades once the feature stops being novel, biasing an early readout upward. A primacy effect (sometimes called a change-aversion or resistance-to-change effect) is the opposite pattern: a change disrupts a habitual workflow, so users are temporarily worse off while they relearn it, biasing an early readout downward, then the effect climbs toward its true level as users adapt. Both biases fade over roughly the same kind of horizon, so trusting a week-one number without checking its trajectory can make you launch a fad or kill a genuine win too early.

Structured elaboration

Mechanism and direction

Diagnostics to tell a spike from a persistent effect

• Time-windowed effect plot: daily or weekly treatment effect with confidence intervals, ideally with a smoothed trend line (LOESS or a spline), not a single pooled average. A genuine effect looks like a roughly flat band around a nonzero value; novelty looks like a spike that decays toward that band; primacy looks like a trough that rises toward it.
• Exposure-age cohorts, not calendar time: plot the effect against days since each user's first exposure for a fixed cohort of users first exposed on the same day, rather than calendar date. A calendar-time plot mixes newly exposed users (still novel-biased) with long-exposed users (already stabilized) every single day, which can mask a real decay curve as a flat line.
• New vs. returning user split: novelty is usually concentrated in users encountering the feature for the first time; if the effect is similar in a segment already exposed for weeks, that argues against novelty as the explanation.
• Change-point or decay model on the daily series: fit a time-varying effect model, effect as a function of exposure age, and test whether the transient component is statistically distinguishable from zero, separately from the asymptotic (persistent) component.
• Placebo check: run the same time-windowed analysis on a pre-launch period with no real treatment; if spike-like patterns appear there too, the "decay" you see in the real experiment may just be normal week-to-week noise, not a novelty artifact.

Adjusting duration and analysis

• Pre-register the analysis window before launch rather than reading the metric the moment it looks good; a fixed rule such as "primary read is the average effect over exposure-days 21 to 35" prevents cherry-picking the peak or the trough.
• Extend the experiment until the exposure-age curve visibly plateaus, or the fitted transient component's confidence interval crosses zero, rather than for a fixed calendar duration chosen in advance.
• Report both the early-window and late-window effect side by side rather than a single blended number; a launch decision based only on the blended average silently averages a fading spike with a stabilizing floor.

Worked example

Two hypothetical (illustrative, not real study data) weekly average-treatment-effect readings for the same nominal conversion metric:

Both curves are converging toward roughly the same persistent level, one from above and one from below, which is exactly the signature that separates them from a flat, genuine effect that would show roughly the same number every week within noise.

If the transient component decays exponentially, $\Delta(t) = C + A e^{-\lambda t}$, where $A$ is the size of the initial novelty or primacy spike above the persistent effect $C$ (the extra amount present at $t=0$ that fades away over time), and the illustrative decay rate is $\lambda = 0.2$ per week, its half-life is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.2} \approx 3.5 \text{ weeks}$
That is the kind of number worth pre-registering as a decision rule: run at least three half-lives (about 10 to 11 weeks here) before reading the persistent effect $C$, rather than picking an arbitrary duration.

Trade-offs and pitfalls

• Waiting out a full decay curve costs calendar time and opportunity cost on other experiments; for low-stakes features, teams sometimes accept the risk of a novelty-inflated launch decision rather than run for months.
• Segmenting by exposure age needs per-user first-exposure timestamps captured in the assignment log; if you only log calendar-date rollups, you cannot separate calendar effects from exposure-age effects after the fact.
• A curve that looks like decay can just as easily reflect unrelated seasonality (marketing pushes, holidays) that correlates with launch timing; a decay-shaped curve is suggestive, not conclusive, on its own.
• Don't assume every early spike is novelty and every early trough is resistance to change: an early spike can be a genuine effect solving a pent-up need immediately, and an early trough can be a real bug that later gets patched. The pattern is evidence, not proof, and should be paired with qualitative checks (support tickets, session recordings) before concluding the mechanism.

Approach
I would build a date spine, which is a table of every calendar date in the reporting range, then cross join it to the product list. That guarantees zero-sales days are present. After that, I would left join the daily sales fact and use a running sum partitioned by product and month.

WITH spine AS (
  SELECT d::date AS dt
  FROM generate_series(date '2025-01-01', date '2025-01-31', interval '1 day') AS g(d)
), daily AS (
  SELECT product_id, sale_date, SUM(revenue) AS revenue
  FROM sales_fact
  GROUP BY product_id, sale_date
), base AS (
  SELECT p.product_id, s.dt
  FROM products p
  CROSS JOIN spine s
)
SELECT
  b.product_id,
  b.dt,
  COALESCE(d.revenue, 0) AS revenue,
  SUM(COALESCE(d.revenue, 0)) OVER (
    PARTITION BY b.product_id, DATE_TRUNC('month', b.dt)
    ORDER BY b.dt
  ) AS mtd_revenue
FROM base b
LEFT JOIN daily d
  ON d.product_id = b.product_id
 AND d.sale_date = b.dt;

Reference data needed

• A product dimension.
• A calendar or date spine table, or a generated date series.

Why it works
Partitioning by month resets the running total at each boundary, and the spine ensures missing product-day combinations still show as zero.

Direct answer

Escalating a stakeholder conflict to leadership is the right call when peer-level resolution has genuinely been tried and failed, the disagreement is actively blocking meaningful progress, and the decision at stake is significant or hard to reverse; framing the escalation as a request for a decision rather than a complaint about either party is what makes it land well.

Structured elaboration

1. What made escalation the right call, not just an easier one. A real attempt at resolving the disagreement directly should precede escalation; the deciding factor is usually that continued peer-level effort was unlikely to converge and the cost of continued delay was rising.
2. How the ask to leadership was framed. Presenting the situation neutrally, with both sides' reasoning represented fairly, and asking specifically for a decision on a defined question, rather than asking leadership to referee who's "right," keeps the conversation focused on unblocking progress.
3. What leadership needed to make a good call. A concise summary of the disagreement, what was tried, the trade-offs of each option, and a clear ask (a decision, a resource, an explicit priority call) gives leadership what they need without requiring them to relitigate the whole history.
4. What happened afterward. Documenting the resolution and communicating it back to both original parties closes the loop, so the escalation doesn't leave lingering resentment about how it was handled.

Worked example

Two teams couldn't agree on which of two conflicting priorities to pursue with shared, limited resources, and each had reasonable grounds for their position. After a joint conversation failed to converge over several days while the clock on both timelines kept running, escalating with a short written summary presenting both positions fairly, the trade-offs of each, and an explicit request for a priority call, let leadership make a fast, informed decision rather than re-litigating the underlying technical debate themselves.

Trade-offs and pitfalls

An escalation framed even slightly as blame-assigning, rather than decision-seeking, tends to put leadership in the position of managing your relationship with the other party instead of making the actual call needed, which is a worse outcome for everyone involved.

Direct answer: Attribute the refund to the ORIGINAL purchase month by default (so a month's revenue reflects net revenue for purchases made that month, a "vintage" view), and additionally expose a refund-date view for operational/cash-flow purposes; doing both, clearly labeled, resolves the trade-off rather than forcing a single choice that serves one audience and misleads another.

Structured elaboration:

• Attribute to original purchase month (vintage/net-revenue view): answers "how much revenue did purchases made in June ultimately generate, after all refunds," which is the more meaningful number for evaluating a specific month's SALES quality and for financial metrics like LTV that need to reflect true net value per acquisition cohort. Cost: this view changes RETROACTIVELY as later refunds come in, so "June's revenue" reported today may differ from "June's revenue" reported a month from now as more refunds land, which needs to be documented as an expected property of this view, not a bug.
• Attribute to refund month (cash/operational view): answers "how much net cash moved this month," which is what an operational cash-flow dashboard cares about; this view is stable once a month closes (refunds affect the month they occur in, not a past month), but conflates "how good was June's sales" with "how much refund activity happened in whatever month that activity landed."
• Recommendation: implement BOTH as separate, clearly named metrics (monthly_revenue_by_purchase_vintage vs monthly_net_cash_by_transaction_month) rather than picking one and hoping it serves every audience; most disputes here come from two teams silently assuming different conventions while calling the result by the same ambiguous name ("monthly revenue").

Worked example: a $100 purchase in June refunded in August: the vintage view attributes -$100 back to June (so June's TRUE net revenue, once fully settled, is 0 for that purchase), which is the right number for evaluating June's cohort quality; the cash view attributes -$100 to August (June showed +$100 at the time, August shows the -$100 offset), which is the right number for August's actual cash-flow statement.

Trade-offs & pitfalls: The vintage view requires the ETL to support RETROACTIVE updates to a previously-published month's number as refunds continue arriving (which needs the same versioned/data_version discipline discussed for other retroactively-adjusted metrics in this topic, so consumers can tell a "final" June number from a "still settling" one); implementing the vintage view without that retroactive-update discipline, and instead freezing June's number the day the month closes, silently produces an inaccurate vintage number that never gets corrected, which is arguably worse than not offering the vintage view at all.

Direct answer

Read the date column as plain strings (never let pandas guess the format), try an explicit ISO parse first, fall back to a day-first parse with errors='coerce' for whatever the ISO pass missed, and localize the whole result to UTC (Coordinated Universal Time). Detecting an ambiguous format like "01/02/2021" is a separate check: flag any D/M/Y-shaped string where both the first and second numbers are 12 or less, since either reading is grammatically valid.

Approach

1. Read the column as dtype=str so pandas does not silently infer a format on read.
2. Flag ambiguous strings with a regex plus a numeric check (both candidate day and month values in 1..12).
3. Parse with an explicit ISO format="%Y-%m-%d" first; anything that fails becomes NaT, not an exception.
4. Re-parse only the rows that failed ISO parsing, this time with dayfirst=True to correctly read "31/12/2021"-style strings; still coerce failures to NaT.
5. Combine the two parses and localize to UTC.

Worked example

import re
import pandas as pd

df = pd.DataFrame({
    "date": ["2021-12-31", "31/12/2021", "01/02/2021", "not-a-date", None],
})
s = df["date"].astype(str).str.strip()

ambig_pattern = re.compile(r"^(\d{1,2})[\/\-](\d{1,2})[\/\-]\d{2,4}$")

def is_ambiguous(date_str):
    if not isinstance(date_str, str):
        return False
    m = ambig_pattern.match(date_str)
    if not m:
        return False
    d, mo = int(m.group(1)), int(m.group(2))
    return 1 <= d <= 12 and 1 <= mo <= 12

df["date_ambiguous"] = s.apply(is_ambiguous)

parsed_iso = pd.to_datetime(s, format="%Y-%m-%d", errors="coerce", utc=True)
mask_missing = parsed_iso.isna()
parsed_dayfirst = pd.to_datetime(s[mask_missing], dayfirst=True, errors="coerce", utc=True)

parsed = parsed_iso.copy()
parsed.loc[mask_missing] = parsed_dayfirst

df["date_parsed_utc"] = parsed
df["date_parse_failed"] = df["date_parsed_utc"].isna()
print(df)

Output (verified against pandas 3.0.3):

         date  date_ambiguous           date_parsed_utc  date_parse_failed
0  2021-12-31           False 2021-12-31 00:00:00+00:00              False
1  31/12/2021           False 2021-12-31 00:00:00+00:00              False
2  01/02/2021            True 2021-02-01 00:00:00+00:00              False
3  not-a-date           False                       NaT               True
4         NaN           False                       NaT               True

"01/02/2021" is correctly flagged date_ambiguous=True (both 01 and 02 are valid day-or-month values) and, absent other context, is parsed day-first as February 1st, which the ambiguity flag exists precisely to call out for review. "31/12/2021" is unambiguous (31 cannot be a month) and needs no flag.

Key points

• Reading with dtype=str and parsing explicitly (rather than pd.read_csv(..., parse_dates=['date'])) keeps control of exactly which format is tried in which order, which matters when the column mixes formats.
• The ambiguity check and the actual parse are independent: the parse always produces a best-effort answer (or NaT), while the ambiguity flag is a signal for a human to double-check specific rows, not a gate on parsing.
• errors='coerce' at each stage means a malformed row never crashes the whole batch; it becomes NaT and is visible in date_parse_failed.

Complexity

Two vectorized pd.to_datetime passes, each O(n), plus one Python-level .apply() for the ambiguity check, which is O(n) but with a much larger per-row constant since it runs a compiled regex through the Python interpreter row by row rather than in pandas' vectorized layer; for very large data, .str.extract with a compiled pattern replaces .apply(is_ambiguous) at some cost in readability. Memory: two full-length parsed Series are held simultaneously before being combined, plus the boolean flag columns.

Edge cases

• Fully missing values (None/NaN): in pandas 3.0, string columns default to the new string dtype rather than the old object dtype, and a missing value in that column stays a real missing value through .astype(str) rather than becoming the literal text "None." A row-wise function like is_ambiguous that assumes it always receives a string will raise TypeError: expected string or bytes-like object, got 'float' on such a row; guard with an isinstance(date_str, str) check before matching, as shown above. This is a genuine pandas 3.0 currency trap: code that worked under the old object-dtype default can break silently (well, loudly, via a TypeError) once run against the new default string dtype.
• Malformed strings that match no format: land as NaT after both parse attempts and are visible via date_parse_failed.
• True ambiguity with no way to resolve it from the string alone: the flag surfaces these; resolving them (correctly) requires outside context (a known source locale, a paired timestamp field), not a smarter regex.

Trade-offs and pitfalls

• Letting pd.read_csv auto-infer the date format is convenient but silently picks one interpretation for ambiguous rows with no flag and no way to know afterward which rows were guessed; explicit, staged parsing costs a few more lines but makes every guess visible and auditable.

SQL predicates don't just evaluate to TRUE or FALSE; a third state, UNKNOWN, appears whenever a comparison involves NULL, and a WHERE clause only keeps rows where the predicate is TRUE (UNKNOWN rows are dropped, just like FALSE ones).

Structured elaboration

Truth table for col = 'a' and its negation, across the three column states:

The key insight: NOT UNKNOWN is still UNKNOWN, not TRUE. So WHERE NOT (col = 'a') does NOT select the NULL row, even though intuitively "not equal to a" feels like it should include "unknown, so possibly not a". If you want NULLs included in a "not equal to" filter, you have to say so explicitly: WHERE NOT (col = 'a') OR col IS NULL.

Worked example

Given t(col) with rows 'a', 'b', NULL: evaluating col = 'a' gives TRUE, FALSE, and NULL (SQLite/most engines represent UNKNOWN as NULL in a SELECT list) for the three rows respectively; NOT (col = 'a') gives FALSE, TRUE, and NULL. A WHERE clause on either predicate keeps only the TRUE row(s), silently dropping the NULL row from both.

Trade-offs and pitfalls

Three-valued logic is the root cause behind several other gotchas covered separately (NOT IN with NULL, = NULL returning nothing); understanding it as one unifying concept, rather than memorizing each symptom separately, is what separates a senior explanation from a list of "gotchas I've been bitten by".

Direct answer

A blended, whole-funnel conversion rate can be flat or improving while a specific segment is quietly getting worse, or a change can help one segment and hurt another in ways that cancel out in the aggregate; segmentation is what makes those cases visible. The strategy below covers eight segments (acquisition channel, device, geography, cohort age, plan type, referral status, campaign tag, feature flag), a specific, falsifiable hypothesis for how each might affect conversion, and one KPI (key performance indicator) to monitor per segment.

Structured elaboration

Why segmentation matters, concretely. Two quick examples make the point before the full strategy: a redesigned checkout page might raise the OVERALL conversion rate while quietly making mobile checkout worse, if desktop's larger improvement masks a smaller mobile regression in the blended number, only a device-segmented view catches this. Separately, a marketing campaign might report a healthy overall conversion rate while actually converting referred and organic users well and paid-display users very poorly, with the poor-performing segment's low volume hiding inside a decent-looking blended average; only a channel-segmented view surfaces which spend is actually working.

The eight segments, each with a hypothesis and a monitoring KPI:

Worked example

Take the campaign-tag segment as a concrete walk-through, since expectation mismatch is the least obvious of the eight hypotheses. A campaign advertising "instant setup" drives a large volume of clicks (high click-through rate) but underperforms on actual signup completion relative to a lower-click-through campaign advertising the product more literally, because users who clicked expecting a near-instant flow abandon as soon as they hit a multi-step setup process that does not match what the ad promised. Monitoring conversion rate ALONGSIDE click-through rate by campaign tag is what surfaces this: a campaign with high clicks and low conversion is a different problem (fix the ad copy, or fix the onboarding flow it is setting expectations for) than a campaign with low clicks and high conversion (the audience or channel targeting needs work, not the funnel itself). Reading conversion rate alone, without the click-through context, cannot distinguish these two very different diagnoses.

Trade-offs and pitfalls

• Segment interactions matter and single-dimension segmentation can hide them. A device effect might only really be a MOBILE-PLUS-PAID-CHANNEL effect (mobile users from paid campaigns specifically underperform, while mobile users from organic search do fine); segmenting on device alone and segmenting on channel alone can each look unremarkable while the intersection hides a real, fixable problem. Once a single-dimension segment looks flat but a hypothesis suggests an interaction might be at play, check the two-way cross-tab before concluding there is nothing there.
• Common mistake: treating every observed segment difference as something to "fix," rather than checking whether it reflects a genuine, fixable friction versus an expected, non-fixable population difference (enterprise-plan users SHOULD take longer to convert than self-serve users; that is not a bug in the funnel, it is a real difference in the decision being made).
• Segment sample size discipline. A segment with too little volume (a niche geography, a narrow campaign tag) produces a noisy conversion-rate estimate that can look dramatically different from the blended average purely from sampling variation; before treating a small segment's number as a real signal, check whether the segment has enough volume for the comparison to be meaningful, the same discipline a proper A/B test applies to its own sample-size planning.
• Pre-register which segments matter before looking, where possible. Segmenting exhaustively across many dimensions and reporting only the one that happens to look interesting is a version of the multiple-comparisons problem: with enough segments checked, SOME of them will show an apparently interesting difference by chance alone. The eight hypotheses above are useful specifically because each one is stated as a falsifiable claim BEFORE looking at the data, not discovered by mining for whichever cut looks most dramatic after the fact.

What data to collect first, given limited telemetry, prioritized by what's cheapest to get:

• Categorize the existing support tickets by root cause, bug, confusion or UX friction, or missing feature, using the ticket text you already have. This alone often explains a mixed NPS (Net Promoter Score, a customer loyalty metric typically ranging from -100 to 100, based on how likely customers are to recommend the product) without needing any new instrumentation.
• Pull the actual free-text comments behind the NPS scores, not just the numeric score, segmented by score band and by whether the respondent also filed a support ticket.
• Add two or three targeted funnel events to close a specific blind spot, for example first-use completion or feature discovery, deployable in days, not a full telemetry overhaul.
• Segment the sales signal itself: confirm whether "healthy sales" means new-customer acquisition, which doesn't yet reflect the launch experience, or existing-customer expansion and renewal, which would. Conflating the two is a common analysis mistake that can make the picture look more consistent than it is.

Short experiments to run:

• A small usability test, five to eight users, on the specific flow the support tickets cluster around, cheap and fast, days not weeks, to turn "tickets are up" into a specific, fixable cause.
• A lightweight in-product micro-survey triggered right after the friction point the tickets point to, to get a quick directional read on whether it's a widespread problem or a vocal minority.

Scenario planning for product changes, three branches based on what the above reveals:

• If the cause is a specific, fixable bug or UX gap with isolated blast radius: ship a targeted fix, no rollback needed, and confirm the ticket category drops afterward.
• If the cause is a genuine design mismatch for a meaningful segment that a quick patch can't fix: add an opt-out or legacy path for that segment sized to how large it actually is, while iterating on the new design.
• If the signal turns out to be a vocal minority concentrated in a small, low-value segment while the broader base is fine: make no major product change, close the specific gaps raised, and treat the sales signal as the stronger read, since it reflects a broader population than survey respondents typically do.

Criteria for go or no-go on rolling the launch back: don't use a single blanket threshold. Roll back only if the launch-caused ticket category stays above a sustained multiple of its pre-launch baseline, for example more than 3 times baseline sustained for two weeks, and it's concentrated among customers near a renewal decision, the highest-stakes segment, and no fix is deployable within that renewal window. Otherwise, keep the launch live and iterate, because rolling back a major launch has its own real cost, re-migration, team morale, lost momentum, and reacting to the loudest signal, support tickets, when it isn't necessarily the most representative one, is the exact bias this criteria set is meant to guard against.