A/B Test Design & Statistical Rigor Questions

Direct answer

Run for a whole number of full weekly cycles, decide before looking at any data how a holiday inside that window will be handled, and hold traffic allocation balanced by day-of-week (and by region and time zone if the test spans them) rather than trusting that a single aggregate 50/50 split will average out. When a weekday-positive, weekend-negative pattern shows up later, treat it as a hypothesis to falsify with three specific checks, real heterogeneity, a traffic-composition artifact, or noise, rather than reading the raw split at face value.

Structured elaboration

Duration and analysis window

Run for at least two, ideally three or more, full 7-day cycles. A partial week biases the pooled result toward whichever days happen to be over-represented, and a single week does not let you separate a real weekday/weekend pattern from that week's idiosyncrasies. If a holiday falls inside the planned window, decide up front, before seeing any results, between two options: exclude the holiday period from the primary analysis window and report a "typical week" estimate, or explicitly include it and report a distinct holiday-period estimate. Choosing between those two after looking at which one produces a better-looking result is a form of after-the-fact window selection and should be avoided; pre-register the choice in the analysis plan.

Traffic allocation and balance across time and geography

Stratify random assignment by day-of-week, and by region or time zone if the rollout spans them, so the same proportion of each arm is exposed every day and in every zone rather than relying on an aggregate split that could hide a skew. For a multi-region or multi-time-zone test, anchor "day" and "week" boundaries to each user's local time rather than a single server or UTC clock; otherwise one region's weekend gets miscounted against another region's weekday, and verify the treatment-to-control ratio stays constant across regions and hour-of-day buckets individually, not just in the combined total. Aggregate balance can look fine while a specific region or time window is quietly imbalanced, and that imbalance is exactly what later gets mistaken for a day-of-week effect.

Modeling the temporal structure instead of ignoring it

Rather than computing one pooled treatment effect and hoping seasonality washes out, fit day-of-week (and holiday, and region, if relevant) as explicit terms: outcome ~ treatment + day_of_week + treatment:day_of_week + region. This is standard regression-formula shorthand: ~ means "model the left-hand outcome using the terms on the right," so this line reads as "predict the outcome from the treatment, the day type, and the region," and treatment:day_of_week is an interaction term, a piece that lets the treatment's effect itself differ by day type rather than assuming it is the same on weekdays and weekends. The interaction term is what actually tells you whether the treatment effect differs by day type, instead of a single pooled number that could be hiding it.

Investigating a weekday-positive, weekend-negative split

Three checks, run in this order:

1. Is it real? Fit the treatment-by-day-type interaction term from the model above and check whether it is distinguishable from a null effect. This is one specific comparison, not a license to slice every available dimension until something looks significant; keep the interaction pre-specified as part of the analysis plan for exactly this reason.
2. Is it a traffic-composition artifact? Check whether the user mix itself differs by day type: a different device split, acquisition channel, or new-versus-returning ratio on weekends than weekdays. Re-run the interaction model with that covariate added and interacted; if the day-type interaction shrinks toward zero once the segment mix is controlled for, the apparent weekday/weekend split was really a segment-level pattern wearing a calendar label. Also check whether the rollout itself was staggered mid-week (a ramp that reached full exposure partway through the window) or whether an assignment-pipeline issue caused the treatment:control ratio to drift on certain days; both produce a day-type-looking artifact that has nothing to do with actual weekday or weekend behavior.
3. Is it noise? Compare the confidence interval on each day-type's estimate rather than the point estimates alone. Weekend traffic is frequently a fraction of weekday traffic, so a "negative" weekend estimate often carries a wide interval that comfortably contains the weekday estimate.

Worked example

Suppose the weekday arm has 8,000 users per group with control conversion 10.0% and treatment conversion 10.6% (a +0.6 percentage point delta), and the weekend arm has 2,000 users per group (lower weekend traffic) with control conversion 10.0% and treatment conversion 9.4% (a -0.6 percentage point delta). This is exactly the pattern in the question: positive on weekdays, negative on weekends.

Standard error of each delta, using $SE = \sqrt{\frac{p_c(1-p_c)}{n} + \frac{p_t(1-p_t)}{n}}$:

Weekday: $SE_{wd} = \sqrt{\frac{0.10\times0.90}{8000} + \frac{0.106\times0.894}{8000}} = 0.00481$, so the weekday delta's 95% interval is roughly $-0.34\text{pp}$ to $+1.54\text{pp}$, which already crosses zero.

Weekend: $SE_{we} = \sqrt{\frac{0.10\times0.90}{2000} + \frac{0.094\times0.906}{2000}} = 0.00936$, so the weekend delta's 95% interval is roughly $-2.43\text{pp}$ to $+1.23\text{pp}$, also crossing zero.

Testing whether the two deltas actually differ from each other: $z = \frac{0.006-(-0.006)}{\sqrt{0.00481^2+0.00936^2}} = \frac{0.012}{0.01052} \approx 1.14$, well under the 1.96 threshold for a two-sided 5% test. Both individual intervals already contain zero, and the two deltas are not statistically distinguishable from each other. With these particular sample sizes, the weekday-positive-weekend-negative pattern is fully consistent with noise, before ever needing to invoke a real behavioral difference or an artifact.

Trade-offs & pitfalls

• Trusting the point estimate over the interval. A sign flip between two point estimates feels meaningful; whether it survives a formal comparison of the two deltas, as above, is what actually determines whether there is anything to explain.
• Deciding the holiday treatment after seeing results. Choosing whether to include or exclude a holiday period based on which choice produces the preferred outcome is a subtle form of p-hacking through window selection, even when no single test is repeated.
• Assuming aggregate balance implies balance everywhere. A day-of-week or region-level imbalance can hide inside an aggregate 50/50 split and later masquerade as a real seasonal effect.
• Over-correcting into paralysis. Not every day-type split needs a full forensic investigation; reserve the three-check process for patterns that would actually change a rollout decision, and size the investigation to the stakes.

Direct answer

CUPED (Controlled-experiment Using Pre-Experiment Data, the variance-reduction method published by Deng, Xu, Kohavi, and Walker at Microsoft, WSDM 2013) adjusts the experiment outcome $Y$ by subtracting a scaled, pre-experiment covariate $X$ that is correlated with $Y$ but unaffected by treatment. The optimal scaling coefficient is $\theta^* = \mathrm{Cov}(X,Y)/\mathrm{Var}(X)$, and the payoff is that the adjusted outcome's variance shrinks by a factor of $(1-\rho^2)$, where $\rho$ is the correlation between $X$ and $Y$, which under a fixed minimum detectable effect and fixed power translates directly into a required sample size that shrinks by the same factor. The method breaks down if $X$ is itself affected by treatment, because then part of the true treatment effect gets subtracted out along with the noise.

Structured elaboration

Deriving the optimal coefficient

Define the adjusted outcome as $Y' = Y - \theta (X - E[X])$. Subtracting a constant, $\theta E[X]$, does not change variance, so $Y'$ has the same variance as $Y - \theta X$:
$\mathrm{Var}(Y') = \mathrm{Var}(Y) - 2\theta\,\mathrm{Cov}(X,Y) + \theta^2 \mathrm{Var}(X)$
This is a quadratic in $\theta$, minimized where its derivative with respect to $\theta$ is zero:
$\frac{d}{d\theta}\mathrm{Var}(Y') = -2\,\mathrm{Cov}(X,Y) + 2\theta\,\mathrm{Var}(X) = 0 \quad \Rightarrow \quad \theta^* = \frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}$
Subtracting $\theta(X - E[X])$ does not shift the mean of $Y'$, since $E[X - E[X]] = 0$, so this adjustment is unbiased for the mean of $Y$, and therefore unbiased for the treatment effect, as long as $X$ is unaffected by treatment (more below).

Variance reduction in terms of rho

Substituting $\theta^*$ back into the variance expression, writing $\sigma_X^2 = \mathrm{Var}(X)$ and $\sigma_Y^2 = \mathrm{Var}(Y)$:
$\mathrm{Var}(Y') = \sigma_Y^2 - 2\theta^*\,\mathrm{Cov}(X,Y) + (\theta^*)^2\sigma_X^2 = \sigma_Y^2 - \frac{\mathrm{Cov}(X,Y)^2}{\sigma_X^2}$
Using $\rho = \mathrm{Cov}(X,Y)/(\sigma_X \sigma_Y)$, so $\mathrm{Cov}(X,Y)^2 = \rho^2 \sigma_X^2 \sigma_Y^2$:
$\mathrm{Var}(Y') = \sigma_Y^2 - \frac{\rho^2 \sigma_X^2 \sigma_Y^2}{\sigma_X^2} = \sigma_Y^2(1 - \rho^2)$
CUPED reduces the variance of the outcome used in the treatment-effect estimate by exactly a factor of $(1-\rho^2)$: a covariate correlated at $\rho = 0.5$ removes $\rho^2 = 25\%$ of the outcome variance; at $\rho = 0.7$ it removes $\rho^2 \approx 49\%$. The relationship is quadratic in $\rho$, so weakly correlated covariates, $\rho$ below roughly 0.3, remove under 9% of variance and buy very little.

How required sample size scales with rho

The standard two-sample sample-size formula for a fixed minimum detectable effect (MDE), significance level, and power is proportional to the outcome variance: $n \propto \sigma^2 / \text{MDE}^2$, holding alpha and power fixed; that base relationship itself belongs to standard power-analysis mechanics, not to CUPED. Since CUPED only changes the variance term, from $\sigma_Y^2$ to $\sigma_Y^2(1-\rho^2)$, and leaves the MDE and the estimator's unbiasedness unchanged, the required sample size scales the same way:
$\frac{n_{\text{CUPED}}}{n_{\text{raw}}} = 1 - \rho^2$
A pre-experiment covariate correlated at $\rho = 0.7$ with the outcome would let you reach the same statistical power with about $1 - 0.49 = 0.51$, roughly half the sample, or equivalently run the same sample for about half the calendar duration.

What breaks if X is affected by treatment

The derivation above relies on $\theta$ being estimated from data where $X$ has no relationship to treatment assignment. If $X$ is measured after treatment starts, or is otherwise influenced by it, then $X$ itself carries part of the treatment signal, and subtracting $\theta(X - E[X])$ subtracts part of that signal out of $Y'$ along with the noise, biasing the estimated treatment effect toward zero. This is the same failure mode as conditioning on a variable that sits on the causal path between treatment and outcome: the fix is procedural, not statistical, use only strictly pre-experiment values of $X$, measured before randomization occurs, computed identically for both arms.

Worked example

Suppose a pre-experiment covariate is a user's prior 28-day spend, and analysis of historical (pre-experiment) data gives stated, illustrative moments $\sigma_X^2 = 400$, $\sigma_Y^2 = 100$, and $\mathrm{Cov}(X,Y) = 120$. Then:
$\theta^* = \frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)} = \frac{120}{400} = 0.3$
$\rho = \frac{\mathrm{Cov}(X,Y)}{\sigma_X \sigma_Y} = \frac{120}{\sqrt{400}\sqrt{100}} = \frac{120}{20 \times 10} = 0.6$
$\mathrm{Var}(Y') = \sigma_Y^2(1-\rho^2) = 100 \times (1 - 0.36) = 64$
The adjusted outcome variance drops from 100 to 64, a 36% reduction, and the required sample size for the same MDE and power drops to $1 - 0.36 = 0.64$, 64% of the original sample.

Trade-offs and pitfalls

• $\theta$ itself must be estimated, from pooled or historical data, which adds a small amount of estimation noise not captured in the idealized derivation above; in practice this is usually negligible if $\theta$ is estimated on a large enough historical sample, but it is not literally zero cost.
• CUPED buys nothing for users with no pre-experiment history, since $X$ is undefined or has to be imputed for them; teams typically report CUPED-adjusted results for existing users and a separate, unadjusted analysis for new users rather than silently imputing.
• The variance reduction is entirely a function of $\rho$; picking a covariate that is easy to compute but weakly correlated with the actual outcome metric wastes engineering effort for little statistical payoff, so the covariate should be chosen and validated on historical data before committing to it, not assumed.
• The most common real-world mistake is using a covariate window that overlaps with treatment start, even by a day, because of a timezone or pipeline lag; this silently reintroduces the treatment-affected-X bias described above while looking, superficially, like a valid pre-experiment covariate.

Direct answer

The key design choice is randomizing at a unit large enough to contain the cross-side spillover a price or fee change creates: typically a market cluster (a geography, city, or self-contained supply region) rather than individual buyers or individual sellers, because a marketplace's two sides interact through the same local supply-demand pool. Within that, you deliberately vary treatment intensity across clusters (a partial-saturation or dose-response design) so you can separate the direct pricing effect from the cross-side externality it triggers, and you plan the revenue read as a staged measurement: an early operational window for the direct effect, followed by a longer holdout-based window because churn and re-equilibration on a two-sided market take longer to surface than a single-user metric would.

Structured elaboration

Why buyer-only or seller-only randomization fails here

If you randomize individual sellers into a higher take rate, a treated seller may raise prices, list less, or churn; buyers who would have transacted with that seller instead transact with a control-arm seller in the same market. The buyer side is now indirectly treated regardless of which arm assigned it, which is the same SUTVA-style interference problem as a social feed, except the shared medium is the local marketplace instead of a social graph. Randomizing individual buyers has the mirror-image problem on the seller side. Either choice contaminates the arm you intended to leave clean.

Randomization scheme

• Unit: market cluster (geography, city, or another boundary where most matching happens locally, e.g., delivery radius). This keeps most buyer-seller matching internal to a single treatment condition.
• Design shape: partial saturation. Instead of a flat 50/50 split, assign clusters to a small number of treatment intensities (for example, no increase, a modest increase, a larger increase) rather than a single on/off arm. This lets you trace how the cross-side response scales with the size of the change, which a single treatment level cannot distinguish from a fixed step change.
• Randomize at the cluster level, stratified on baseline liquidity (existing buyer-to-seller ratio, transaction volume) so clusters that already look structurally different are balanced across intensities before you start, reducing the chance that a treatment-intensity effect is confounded with a pre-existing market difference.

Detecting and quantifying the cross-side externality

• Because clusters are internally exposed to one intensity, you can compare a directly-treated side's metric (e.g., seller take-rate exposure) against the other side's metric within the same cluster (buyer conversion, buyer price sensitivity) to see if the fee change on sellers moved buyer-side behavior, and by how much, as intensity increases.
• The partial-saturation design turns this into a dose-response check: plot the buyer-side metric against assigned intensity across clusters. A flat line across intensities is evidence of a contained direct effect; a sloped line is direct, in-cluster measurement of the spillover, not an assumption about its existence.
• Compare a cluster's realized outcome to adjacent, untreated clusters it plausibly shares supply with (e.g., neighboring cities where sellers can relist), which is a direct check for leakage across the cluster boundary itself, not just across sides within a cluster.

Estimating the long-run revenue impact

• Short-run direct revenue (immediate take-rate math: transactions times the new fee) is mechanical and available immediately, but it is not the number that matters, because it ignores behavioral response.
• The number that matters is the net of three components measured over a longer window: the direct fee revenue, minus revenue lost to seller churn or delisting, minus revenue lost to buyer-side friction from any resulting price or availability change. Each of these three needs its own time horizon: fee revenue is immediate, seller churn plays out over weeks as sellers decide whether to stay, and buyer-side effects play out over the buyer's own return cadence.
• Because of that lag structure, hold a subset of clusters as a long-run holdout past the point where you make the initial ship decision. This is what lets you catch delayed seller attrition or buyer defection that would not have shown up in an early readout, and it is a standard practice for any monetization change with a plausible slow-churn tail, not something specific to marketplaces.

Worked example

A delivery marketplace tests a take-rate increase across 40 city clusters, split into two treatment intensities plus control (roughly 13-14 clusters each, stratified on baseline order volume so the three groups start with comparable liquidity). At the direct level, transaction-weighted take-rate revenue rises with intensity, as expected mechanically. The diagnostic step is checking buyer-side order volume within the same clusters: if buyer order volume also declines with intensity (a negative slope across the three intensity levels, measured, not assumed), that decline is the in-cluster, dose-response evidence of the cross-side externality: sellers responded to the higher take rate by raising prices or delisting, and buyers responded to that. The ship decision then nets the two effects (higher unit take rate, lower volume) into an actual revenue trajectory rather than trusting the mechanical fee-revenue number alone.

Trade-offs and pitfalls

• Cluster randomization costs statistical power relative to individual-level randomization, because the effective sample size is the number of clusters, which is typically far smaller than the number of users; this needs a longer test or fewer, larger clusters, and it is a real cost you should state up front rather than discover after the fact.
• A short observation window will understate the true cost of the change, because seller churn and buyer defection both lag the price change; shipping on the early direct-revenue number alone is the single most common mistake in this design.
• Neighboring-cluster leakage (a seller in a treated city relisting in an adjacent control city) is a real risk for marketplaces with mobile supply; check it explicitly rather than assuming cluster boundaries are airtight.
• Resist reaching for a full structural or instrumental-variable model as the default; those are appropriate when randomization is genuinely unavailable, but the partial-saturation cluster design above gives a directly measured effect and should be preferred whenever you can actually randomize.

Direct answer

Design this as a pre-registered, longitudinal comparison rather than a single before/after read: fix a small number of windows relative to each user's first exposure, not launch date, in advance, fit a simple decay model to the day-by-day treatment effect, and commit to a decision rule, stated before you see the data, for what pattern of the fitted decay and asymptote counts as real and durable versus novelty that will fade. The goal is to make the durable-vs-fading call a mechanical read of a pre-specified model output, not a judgment call made after watching the curve.

Structured elaboration

Time windows to pre-specify

• Baseline (pre-treatment, roughly two weeks before exposure): confirms no pre-existing difference between the groups on the metric of interest.
• Immediate (days 0 to 7 since first exposure): captures the bulk of any novelty spike.
• Short (days 8 to 30): where a genuine novelty component should be visibly decaying.
• Long (days 91 and beyond, or as far out as the experiment can afford to run): the window whose effect is treated as the primary estimate of the persistent effect, used for the launch decision.
  These are anchored to exposure age, days since each user's own first exposure, not calendar date, so users who join on different days are all compared on the same clock. A calendar-date plot mixes freshly exposed and long-exposed users in the same daily bucket and can mask a real decay curve as a false flat line.

Modeling the decay

Fit the daily or weekly treatment effect to a two-parameter decay-to-asymptote form:
$\Delta(t) = C + A e^{-\lambda t}$
where $t$ is exposure age, $C$ is the persistent (asymptotic) effect, $A$ is the size of the transient novelty component, and $\lambda$ is the decay rate. A purely persistent effect looks like $A \approx 0$, flat from day one; a pure novelty artifact looks like $C \approx 0$, decaying to nothing; most real cases land somewhere in between, with both $A$ and $C$ meaningfully nonzero, meaning some of the early lift really does fade but a smaller durable effect remains.

The decision rule, pre-specified

Commit, before the experiment starts, to a rule such as: the effect is durable if the long-window estimate's confidence interval excludes zero and the fitted persistent component $C$'s confidence interval excludes zero, evaluated no earlier than three estimated half-lives, $3 \times \ln 2/\lambda$, after first exposure. This does three things a post-hoc read cannot: it fixes how long to wait based on the shape of the decay itself rather than an arbitrary calendar deadline, it requires the long-window effect to independently clear significance rather than trusting the fitted curve alone, and it removes the temptation to declare victory the moment the curve looks favorable.

Worked example

Suppose a fitted decay model on the immediate and short windows gives stated, illustrative parameter estimates $A = 6\%$, $C = 2\%$, $\lambda = 0.15$ per week. The half-life of the transient component is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.15} \approx 4.6 \text{ weeks}$
The pre-specified decision rule requires waiting roughly $3 \times 4.6 \approx 13.9$ weeks, call it 14 weeks, before the long-window read is treated as decisive. At that point, the transient component's contribution has decayed to:
$A \cdot e^{-\lambda \cdot 14} = 6\% \times e^{-0.15 \times 14} = 6\% \times e^{-2.1} \approx 6\% \times 0.122 \approx 0.73\%$
which is small enough that the observed effect at week 14 should be close to the true persistent effect $C$, letting the long-window confidence interval be read as a fair test of durability rather than a mix of fading novelty and true signal.

Trade-offs and pitfalls

• Waiting three half-lives before making the call costs real calendar time and delays every downstream decision riding on this experiment; for a low-stakes cosmetic change, teams often accept a shorter, less rigorous wait rather than the full 14 weeks in the worked example.
• The decay model assumes a single clean exponential; a novelty effect that itself varies by segment, a spike for new users layered with a slower-decaying resistance effect for long-tenured users, will not fit a single two-parameter curve well, and forcing the fit anyway can produce a confidently wrong half-life.
• Anchoring on exposure age rather than calendar date requires per-user first-exposure timestamps captured at assignment time; retrofitting this onto an experiment already running on calendar-date logging means exposure-age curves cannot be reconstructed after the fact.
• A pre-specified decision rule protects against motivated reasoning but is only as good as the pre-specified windows; if the true decay is much slower than assumed when the windows were chosen, day 91 may still be well inside the transient period, so a short pilot or a conservative overestimate of the likely half-life should inform window choice up front, not just the final analysis.

Direct answer

A novelty effect is a temporary inflation of an early treatment effect: users explore or click on something purely because it is new, and that extra engagement fades once the feature stops being novel, biasing an early readout upward. A primacy effect (sometimes called a change-aversion or resistance-to-change effect) is the opposite pattern: a change disrupts a habitual workflow, so users are temporarily worse off while they relearn it, biasing an early readout downward, then the effect climbs toward its true level as users adapt. Both biases fade over roughly the same kind of horizon, so trusting a week-one number without checking its trajectory can make you launch a fad or kill a genuine win too early.

Structured elaboration

Mechanism and direction

Diagnostics to tell a spike from a persistent effect

• Time-windowed effect plot: daily or weekly treatment effect with confidence intervals, ideally with a smoothed trend line (LOESS or a spline), not a single pooled average. A genuine effect looks like a roughly flat band around a nonzero value; novelty looks like a spike that decays toward that band; primacy looks like a trough that rises toward it.
• Exposure-age cohorts, not calendar time: plot the effect against days since each user's first exposure for a fixed cohort of users first exposed on the same day, rather than calendar date. A calendar-time plot mixes newly exposed users (still novel-biased) with long-exposed users (already stabilized) every single day, which can mask a real decay curve as a flat line.
• New vs. returning user split: novelty is usually concentrated in users encountering the feature for the first time; if the effect is similar in a segment already exposed for weeks, that argues against novelty as the explanation.
• Change-point or decay model on the daily series: fit a time-varying effect model, effect as a function of exposure age, and test whether the transient component is statistically distinguishable from zero, separately from the asymptotic (persistent) component.
• Placebo check: run the same time-windowed analysis on a pre-launch period with no real treatment; if spike-like patterns appear there too, the "decay" you see in the real experiment may just be normal week-to-week noise, not a novelty artifact.

Adjusting duration and analysis

• Pre-register the analysis window before launch rather than reading the metric the moment it looks good; a fixed rule such as "primary read is the average effect over exposure-days 21 to 35" prevents cherry-picking the peak or the trough.
• Extend the experiment until the exposure-age curve visibly plateaus, or the fitted transient component's confidence interval crosses zero, rather than for a fixed calendar duration chosen in advance.
• Report both the early-window and late-window effect side by side rather than a single blended number; a launch decision based only on the blended average silently averages a fading spike with a stabilizing floor.

Worked example

Two hypothetical (illustrative, not real study data) weekly average-treatment-effect readings for the same nominal conversion metric:

Both curves are converging toward roughly the same persistent level, one from above and one from below, which is exactly the signature that separates them from a flat, genuine effect that would show roughly the same number every week within noise.

If the transient component decays exponentially, $\Delta(t) = C + A e^{-\lambda t}$, where $A$ is the size of the initial novelty or primacy spike above the persistent effect $C$ (the extra amount present at $t=0$ that fades away over time), and the illustrative decay rate is $\lambda = 0.2$ per week, its half-life is:
$t_{1/2} = \frac{\ln 2}{\lambda} = \frac{0.693}{0.2} \approx 3.5 \text{ weeks}$
That is the kind of number worth pre-registering as a decision rule: run at least three half-lives (about 10 to 11 weeks here) before reading the persistent effect $C$, rather than picking an arbitrary duration.

Trade-offs and pitfalls

• Waiting out a full decay curve costs calendar time and opportunity cost on other experiments; for low-stakes features, teams sometimes accept the risk of a novelty-inflated launch decision rather than run for months.
• Segmenting by exposure age needs per-user first-exposure timestamps captured in the assignment log; if you only log calendar-date rollups, you cannot separate calendar effects from exposure-age effects after the fact.
• A curve that looks like decay can just as easily reflect unrelated seasonality (marketing pushes, holidays) that correlates with launch timing; a decay-shaped curve is suggestive, not conclusive, on its own.
• Don't assume every early spike is novelty and every early trough is resistance to change: an early spike can be a genuine effect solving a pent-up need immediately, and an early trough can be a real bug that later gets patched. The pattern is evidence, not proof, and should be paired with qualitative checks (support tickets, session recordings) before concluding the mechanism.