Product and User Behavior Analytics Questions

Direct answer

The right first step is not to jump straight to a query, but to confirm the suspicion is real: split the existing conversion metric by device type and check whether the gap between mobile and desktop is large enough, and consistent enough over time, to be worth investigating further before proposing changes.

Structured elaboration

Concretely, this means running a query that computes conversion rate separately for mobile and desktop sessions over a recent, representative window (avoiding a single unusual day), and checking the result holds across at least a few weeks rather than being a one-off blip. If a real and persistent gap is confirmed, the next step is to visualize it in a way that makes the SIZE of the gap and its trend over time legible at a glance, typically a simple time series with mobile and desktop as two separate lines, since a single point-in-time comparison cannot show whether the gap is stable, widening, or narrowing.

From there, the statistical question becomes whether the observed gap could plausibly be explained by normal variation given the sample sizes involved, which calls for a two-proportion significance test (comparing the mobile conversion rate to the desktop conversion rate as two independent proportions) rather than eyeballing the percentages, especially if one of the two groups has a meaningfully smaller sample size. Only once the gap is confirmed as both real and statistically distinguishable from noise does it make sense to bring the finding to product prioritization, since presenting an unconfirmed or statistically weak gap risks sending a team to fix a problem that may not actually exist.

Worked example

Suppose a two-week pull shows 40,000 desktop sessions converting at 4.8% and 65,000 mobile sessions converting at 3.6%, a 1.2-percentage-point absolute gap. A two-proportion test on those counts (1,920 desktop conversions out of 40,000 versus 2,340 mobile conversions out of 65,000) is the right way to check whether a gap of that size, given those sample sizes, is unlikely to be due to chance, rather than asserting significance from the percentage difference alone; with sample sizes this large, a gap of 1.2 points would typically clear a standard significance threshold, which is itself useful information; a much smaller gap, or a much smaller sample, might not. If confirmed, the visualization for product prioritization would plot the weekly mobile and desktop conversion rates as two lines over the same two-week window, making clear whether the gap has been consistent or is a recent development.

Trade-offs and pitfalls

A common mistake is treating a single day or a single small sample's device split as conclusive, when device-level conversion rates can be noisy day to day for reasons unrelated to a real UX gap, such as a marketing campaign that happened to skew heavily toward one device that day. Another common mistake is stopping at "mobile converts worse than desktop" without segmenting further, since a device-level gap is often really concentrated in a specific step of the flow (for example, a form that is hard to fill in on a small screen), and the device split alone will not reveal which step to fix.

Direct answer

HEART is a framework for choosing user-experience metrics along five categories: Happiness (attitudinal satisfaction), Engagement (frequency or depth of involvement), Adoption (how new users take up a feature or product), Retention (whether users keep coming back over time), and Task success (whether users can effectively complete what they set out to do), used to make sure a metric set covers more than just the one or two dimensions that are easiest to measure.

Structured elaboration

The value of the framework is less in the five names themselves and more in the discipline of checking a metric set against all five before shipping it, since teams left to their own devices tend to over-index on whichever category is easiest to instrument (usually engagement or task success, because both are directly observable from event logs) and under-measure attitudinal signals like happiness, which typically require a survey or in-product rating rather than a passive event. A metric set that is strong on engagement and task success but silent on happiness can miss a product that is technically being used successfully but leaving users frustrated, a gap that eventually shows up as churn with no clear signal that predicted it.

Worked example

For a messaging app, one metric per HEART category could be: Happiness, measured by a periodic in-app satisfaction survey asking "how satisfied are you with sending and receiving messages," with a threshold such as maintaining an average score above 4 out of 5; Engagement, measured by messages sent per active user per week, with a threshold such as at least 10 messages per active user per week to count as regular rather than casual, drop-in use; Adoption, measured by the percentage of new signups who send their first message within 24 hours of installing the app, with a threshold such as at least 70% of new signups reaching that first message inside the 24-hour window; Retention, measured by 7-day retention of users who sent at least one message in their first session, with a threshold such as at least 40% still active 7 days later, benchmarked against the product's own historical baseline rather than an industry-wide number; and Task success, measured by message delivery success rate (the percentage of sent messages that are confirmed delivered without error), with a threshold close to 100% (for example, 99.9%) since this is a reliability metric rather than an engagement one.

Trade-offs and pitfalls

The most common misuse of HEART is treating it as a checklist to fill in with whatever metric is easiest to compute for each category, rather than choosing the metric that best represents that category's real meaning for the specific product; a delivery-success-rate metric technically fits "task success" for a messaging app, but the same category for a project-management tool would need a metric about completing a project task, which requires genuinely different instrumentation. It is also worth remembering that not every product needs equal weight on every category: a utility-style tool used briefly but effectively might reasonably prioritize task success and retention over happiness and engagement, and forcing equal emphasis across all five can dilute focus rather than sharpen it.

Direct answer

A Bayesian hierarchical model treats each small cohort's true retention rate as drawn from a shared population-level distribution, estimates that shared distribution from all the cohorts together, and then computes each cohort's posterior estimate by combining its own limited data with the borrowed strength of the shared distribution, producing both a shrunk point estimate and a full credible interval for every cohort, including ones too small to estimate reliably on their own.

Structured elaboration

The model structure for a retention rate is naturally a beta-binomial hierarchy: each cohort's number of retained users is modeled as a binomial draw given its own true retention probability, and those per-cohort probabilities are in turn modeled as draws from a shared Beta distribution whose two shape parameters (call them alpha and beta) describe the population-level spread of retention rates across cohorts. Because the Beta distribution is the conjugate prior for a binomial likelihood (a prior is called conjugate to a likelihood when combining the two produces a posterior in that same distribution family, here another Beta, so updating the estimate is closed-form arithmetic on the two parameters rather than a simulation), this hierarchy has a closed form: each cohort's posterior is itself a Beta distribution, so a full Markov Chain Monte Carlo sampler is not strictly required for this specific case, though a fully general hierarchical model with more structure (covariates, non-conjugate priors) would typically need MCMC, the standard general-purpose way to fit such models, or a variational approximation, a faster but approximate alternative usually reached for once MCMC becomes too slow to run at the needed scale.

Calibration validation means checking that the model's stated uncertainty matches reality: if the model reports 90% credible intervals, roughly 90% of a set of held-out true values should actually fall inside their respective intervals; systematically too many or too few true values landing inside the intervals indicates the model is under- or over-confident and the prior needs to be revisited.

Worked example

import numpy as np
from scipy import stats
np.random.seed(3)

cohort_sizes = [30, 35, 40, 45, 50, 55, 60, 32, 48, 38]
observed = [8, 10, 8, 11, 16, 18, 11, 6, 7, 9]  # 10 small cohorts, true rate 25%, seed=3
raw_rates = np.array([o / n for o, n in zip(observed, cohort_sizes)])

# method-of-moments fit of a Beta prior from the pooled raw rates
m, v = raw_rates.mean(), raw_rates.var(ddof=1)
common = m * (1 - m) / v - 1
alpha0, beta0 = m * common, (1 - m) * common
print('prior alpha, beta:', round(alpha0, 2), round(beta0, 2), ' prior mean:', round(alpha0 / (alpha0 + beta0), 3))

for n, o in zip(cohort_sizes, observed):
    a_post, b_post = alpha0 + o, beta0 + (n - o)
    post_mean = a_post / (a_post + b_post)
    lo, hi = stats.beta.ppf([0.05, 0.95], a_post, b_post)
    print(f"n={n:3} obs={o:2} post_mean={post_mean:.3f} 90%_CI=[{lo:.3f}, {hi:.3f}]")

Fitting this model to 10 small cohorts (sizes 30 to 60 users, a true retention rate of 25%), method-of-moments on the pooled raw rates gives a fitted Beta prior with alpha approximately 11.58 and beta approximately 36.71, implying a prior mean retention rate of 24.0%. Combining that prior with each cohort's own observed count produces posterior means ranging narrowly from 19.3% to 28.6% (versus the raw observed range of 14.6% to 32.7%) and 90% credible intervals ranging from about 12.8 to 16.0 percentage points wide across the ten cohorts (narrower for the larger cohorts, such as n=60's 12.8-point-wide interval, and wider for the smaller ones, such as n=30's 16.0-point-wide interval), for example the 30-user cohort's posterior mean of 25.0% with a 90% interval of approximately 17.4% to 33.4%. The narrower spread of posterior means compared to the raw rates is the shrinkage effect described above; the explicit credible interval on top of that is what a simple point-estimate shrinkage approach does not provide on its own, giving a decision-maker both a better point estimate and an honest sense of remaining uncertainty for even the smallest cohorts.

Trade-offs and pitfalls

The method-of-moments prior fit used here is a fast, closed-form approximation; a fuller Bayesian workflow would place a prior on alpha and beta themselves and sample the full joint posterior (via MCMC), which better propagates uncertainty about the prior itself into each cohort's interval, at the cost of more computation and more modeling complexity than is often justified for a straightforward retention-shrinkage use case. A structural pitfall in either version is assuming all cohorts genuinely share one underlying distribution; if cohorts differ systematically by a known factor, such as acquisition channel or seasonality, pooling them into a single shared prior will shrink cohorts toward a population average that does not actually describe them, so a model with per-channel or per-period sub-groups should be considered whenever such structure is known to exist.

Direct answer

A vanity metric is one that tends to go up regardless of whether the product is actually getting better, while an actionable metric changes in response to something the team did and, when it moves, tells you clearly what to do next. The distinction matters because a metric can look impressive on a slide and still be useless for making a decision.

Structured elaboration

Vanity metrics are usually cumulative totals or simple counts that grow mechanically over time or scale with unrelated factors like marketing spend or app-store visibility: total downloads, total registered accounts, or total page views are the classic examples, because each one mostly reflects how much traffic arrived rather than whether that traffic found value. Actionable metrics are typically rates, ratios, or cohort-based measures that isolate a specific behavior change: conversion rate, day-7 retention, or the completion rate of a specific onboarding step are actionable because a team can point to a change they made and check whether the metric moved in response.

The practical test is to ask, for a given metric, "if this number went up 10% next week, would I know what to do about it, and would I trust that the change reflected something real about the product rather than just more raw traffic." A metric that fails that test belongs on a slide, not on a decision-making dashboard.

Worked example

For a consumer mobile app, "total app downloads this month" is a vanity metric: it can rise purely because a marketing campaign or a seasonal app-store feature drove more installs, with zero information about whether those new users found any value, and there is no specific action a product team can take in response to the number alone beyond "spend more on acquisition," which is a marketing lever, not a product one. By contrast, "percent of new installs that complete the core first action within 24 hours" is actionable: if that rate drops after a release, the team has a specific, testable hypothesis (something in the new-user flow broke or got harder) and a specific lever to pull (revert or fix the flow, then watch the rate recover), and the metric is normalized to new installs rather than a raw count, so it is not conflated with acquisition volume.

Trade-offs and pitfalls

A metric is not intrinsically vanity or actionable forever; total downloads becomes actionable if the team's current job is specifically to grow top-of-funnel awareness and nothing else is changing about the product, so the classification depends on what decision the metric is meant to support, not on the metric's name alone. The bigger pitfall in practice is reporting a vanity metric alongside actionable ones without labeling the difference, since a rising vanity number sitting next to a flat or declining actionable one can make a genuinely stalled product look like it is improving.

Direct answer

Three shapes come up over and over when you plot the percent of a cohort still active by day since signup. A sharp drop between day 0 and day 1 followed by a long, roughly flat tail usually points to an onboarding or first-session problem: most people who were going to bounce did so almost immediately, and the ones who got past that moment stick around. A steady, roughly exponential decay with no clear floor usually means the product has not yet found a stable core audience: every week keeps losing a fixed fraction of who is left, with no group that has settled into habitual use. A curve that starts flat and then drops later, sometimes weeks in, usually points to a trial, billing, or content-exhaustion event rather than a first-use problem: something specific happens at that later point that pushes people out.

Structured elaboration

The shape of a retention curve is really a summary of many individual user journeys, so before trusting the shape you should check that it is not an artifact: uneven event volume by day of week, a definition of "active" that changed mid-cohort, or a cohort that mixes very different acquisition sources into one curve. Once you trust the shape, read it as a sequence of questions rather than a single verdict.

• Sharp day-0 to day-1 drop, then a tail: look at the first-session experience itself. Was there a broken step, a confusing empty state, or a value moment that never arrived? A/B tests on the first-run flow are the natural next move, and the metric to watch is not overall retention but the specific step where people who never return stop taking actions.
• Steady exponential decay, no visible floor: this is a harder problem because nothing in the curve tells you which moment matters. The useful next step is usually to split the cohort by early behavior (did they complete a specific action in week 1?) rather than by acquisition channel, because a decay curve with no floor often hides a bimodal population: a small group behaving like the flat-tail case above, diluted by a much larger group that never really engaged.
• Flat, then a later drop: look for something that happens on a fixed schedule near that point, such as a trial ending, a subscription renewal, or running out of the content or inventory that brought people in. The fix usually lives outside the product surface you are looking at (pricing, content pipeline, notification cadence) rather than inside the first-run flow.

Worked example

Take a cohort of 200 signups with observed active-user counts of 200, 110, 84, 68, 58, 48, 40, 34 across weeks 0 through 7 (an illustrative cohort decaying from 100% to a 17% week-7 level). Retention by week is $110/200=55\%$, $84/200=42\%$, $68/200=34\%$, and so on down to $34/200=17\%$ at week 7. The steepest single-step loss is between week 0 and week 1 (45 percentage points), after which the week-over-week loss shrinks (13, 8, 5, 5, 4, 3 points). That shape, a large first-week loss followed by a slowly shrinking tail, is the "sharp drop then long tail" pattern: it says the product's onboarding, not its ongoing value, is where most of the addressable loss is concentrated, since the users who survive week 1 are churning at a much gentler and still-decreasing rate.

Trade-offs and pitfalls

Reading a shape from a single cohort is risky when the cohort is small: a 200-user cohort's week-7 point (34 users) already has meaningful sampling noise, so do not over-interpret small wiggles late in the curve as a real change in behavior. It is also easy to mistake a mix of two different curve shapes (some users who onboard well, some who never do) for one curve; if a decay curve looks steady but the cohort is large and heterogeneous, plotting it split by an early-behavior signal often reveals that the "steady decay" is really two flat-tail curves blended together.