Product and User Behavior Analytics Questions

Direct answer

The right first step is not to jump straight to a query, but to confirm the suspicion is real: split the existing conversion metric by device type and check whether the gap between mobile and desktop is large enough, and consistent enough over time, to be worth investigating further before proposing changes.

Structured elaboration

Concretely, this means running a query that computes conversion rate separately for mobile and desktop sessions over a recent, representative window (avoiding a single unusual day), and checking the result holds across at least a few weeks rather than being a one-off blip. If a real and persistent gap is confirmed, the next step is to visualize it in a way that makes the SIZE of the gap and its trend over time legible at a glance, typically a simple time series with mobile and desktop as two separate lines, since a single point-in-time comparison cannot show whether the gap is stable, widening, or narrowing.

From there, the statistical question becomes whether the observed gap could plausibly be explained by normal variation given the sample sizes involved, which calls for a two-proportion significance test (comparing the mobile conversion rate to the desktop conversion rate as two independent proportions) rather than eyeballing the percentages, especially if one of the two groups has a meaningfully smaller sample size. Only once the gap is confirmed as both real and statistically distinguishable from noise does it make sense to bring the finding to product prioritization, since presenting an unconfirmed or statistically weak gap risks sending a team to fix a problem that may not actually exist.

Worked example

Suppose a two-week pull shows 40,000 desktop sessions converting at 4.8% and 65,000 mobile sessions converting at 3.6%, a 1.2-percentage-point absolute gap. A two-proportion test on those counts (1,920 desktop conversions out of 40,000 versus 2,340 mobile conversions out of 65,000) is the right way to check whether a gap of that size, given those sample sizes, is unlikely to be due to chance, rather than asserting significance from the percentage difference alone; with sample sizes this large, a gap of 1.2 points would typically clear a standard significance threshold, which is itself useful information; a much smaller gap, or a much smaller sample, might not. If confirmed, the visualization for product prioritization would plot the weekly mobile and desktop conversion rates as two lines over the same two-week window, making clear whether the gap has been consistent or is a recent development.

Trade-offs and pitfalls

A common mistake is treating a single day or a single small sample's device split as conclusive, when device-level conversion rates can be noisy day to day for reasons unrelated to a real UX gap, such as a marketing campaign that happened to skew heavily toward one device that day. Another common mistake is stopping at "mobile converts worse than desktop" without segmenting further, since a device-level gap is often really concentrated in a specific step of the flow (for example, a form that is hard to fill in on a small screen), and the device split alone will not reveal which step to fix.

Direct answer

When activation requires completing several actions and those actions can happen on more than one platform, the activation rate is the fraction of new users who complete the full required set of actions within a defined window, counting a user as activated exactly once regardless of how many of the required actions happened on web versus how many happened on mobile.

Structured elaboration

The double-counting risk here is specific and easy to miss: if activation is tracked separately per platform (a "web activation" flag and a "mobile activation" flag), a single user who completes some steps on web and the rest on mobile can end up counted as activated on neither platform's flag, or worse, counted as two separate partial activations if the platforms are aggregated naively. The fix is to define activation at the USER level, not the platform-event level: track which of the required actions a given user identifier (not a device or session identifier) has completed across all platforms, and mark the user activated the moment the full set is satisfied, from whichever combination of platforms it came from.

That in turn depends on having a reliable way to tie a web session and a mobile session back to the same user identity, which is the real engineering prerequisite for a cross-platform activation metric: without identity resolution across platforms, the metric will systematically undercount activation for genuinely cross-platform users, since the actions they took on the "other" platform will not be visible to whichever platform's activation logic is being evaluated.

Worked example

Suppose activation for a note-taking app is defined as completing three actions within 7 days of signup: creating a first note, inviting a collaborator, and enabling sync. Of 1,000 new signups, 620 completed all three actions on the same platform (say, mobile), and a further 90 completed some actions on mobile and the remainder on web (for example, created the first note on mobile, then invited a collaborator and enabled sync from the web app a day later). Counting per user rather than per platform, the activation rate is $(620+90)/1000=71\%$. If instead activation were tracked per platform and a user needed to complete all three steps on one platform to count, those 90 cross-platform users would show up as incomplete on both platforms, understating the true activation rate as $620/1000=62\%$, a 9-percentage-point gap driven entirely by a measurement choice rather than a real behavior difference.

Trade-offs and pitfalls

The main pitfall is exactly the one above: silently dropping cross-platform completers because the underlying instrumentation was built per platform rather than per user. A second, subtler pitfall is choosing too generous an activation window: if the three required actions can be spread across many weeks, the metric starts to measure "eventually did these things" rather than a meaningful early-activation signal, so the window itself should be chosen based on how quickly a genuinely engaged new user is expected to reach first value, not set arbitrarily wide to inflate the rate.

Direct answer

A vanity metric is one that tends to go up regardless of whether the product is actually getting better, while an actionable metric changes in response to something the team did and, when it moves, tells you clearly what to do next. The distinction matters because a metric can look impressive on a slide and still be useless for making a decision.

Structured elaboration

Vanity metrics are usually cumulative totals or simple counts that grow mechanically over time or scale with unrelated factors like marketing spend or app-store visibility: total downloads, total registered accounts, or total page views are the classic examples, because each one mostly reflects how much traffic arrived rather than whether that traffic found value. Actionable metrics are typically rates, ratios, or cohort-based measures that isolate a specific behavior change: conversion rate, day-7 retention, or the completion rate of a specific onboarding step are actionable because a team can point to a change they made and check whether the metric moved in response.

The practical test is to ask, for a given metric, "if this number went up 10% next week, would I know what to do about it, and would I trust that the change reflected something real about the product rather than just more raw traffic." A metric that fails that test belongs on a slide, not on a decision-making dashboard.

Worked example

For a consumer mobile app, "total app downloads this month" is a vanity metric: it can rise purely because a marketing campaign or a seasonal app-store feature drove more installs, with zero information about whether those new users found any value, and there is no specific action a product team can take in response to the number alone beyond "spend more on acquisition," which is a marketing lever, not a product one. By contrast, "percent of new installs that complete the core first action within 24 hours" is actionable: if that rate drops after a release, the team has a specific, testable hypothesis (something in the new-user flow broke or got harder) and a specific lever to pull (revert or fix the flow, then watch the rate recover), and the metric is normalized to new installs rather than a raw count, so it is not conflated with acquisition volume.

Trade-offs and pitfalls

A metric is not intrinsically vanity or actionable forever; total downloads becomes actionable if the team's current job is specifically to grow top-of-funnel awareness and nothing else is changing about the product, so the classification depends on what decision the metric is meant to support, not on the metric's name alone. The bigger pitfall in practice is reporting a vanity metric alongside actionable ones without labeling the difference, since a rising vanity number sitting next to a flat or declining actionable one can make a genuinely stalled product look like it is improving.

Direct answer

A cohort heatmap or table is the right default when the audience needs to compare many cohorts against each other at a glance, a retention curve as a line chart is the right choice when the audience needs to see the shape and trajectory of one or a few cohorts over time, and a cumulative retention area chart is the right choice when the audience cares about total accumulated engagement rather than the point-in-time percentage.

Structured elaboration

• Cohort heatmap or table: rows are acquisition cohorts (for example, signup week), columns are periods since acquisition, and each cell is colored or shaded by the retention percentage. This shape is built for scanning many cohorts at once, so it is the right tool when the question is "did retention improve after we shipped X" (compare rows above and below the launch date) or "is one acquisition channel's retention worse than another's" (compare rows grouped by channel). The data needs to be a full matrix of cohort by period, in percentages, so cohorts of different sizes are visually comparable.
• Retention curve as a line chart: one or a handful of cohorts plotted as a line over days or weeks since acquisition. This is the right tool when you want the audience to see the SHAPE (steep drop, gentle decay, plateau) rather than compare many cohorts, because a line chart with more than four or five overlapping lines becomes unreadable. The data can stay in percentage form, and the choice that most affects interpretation is the y-axis: a linear axis makes a steep early drop look dramatic, while a log axis makes it easier to compare the SLOPE of decay across cohorts once the sharp early drop is past.
• Cumulative retention area chart: instead of "percent still active in period N," this shows total accumulated active-days or engagement per user over time, stacked or shaded as an area. This is the right tool when the audience question is about lifetime value or total engagement delivered rather than a snapshot retention rate, for example when justifying that a slightly lower retention percentage still delivered more total usage because the cohort was larger. The design choice that most affects interpretation here is whether the area is plotted as a raw cumulative total or normalized per user: an un-normalized cumulative area for a cohort that is simply larger will always look more impressive than a smaller, more efficiently-retained cohort, so labeling the chart clearly as per-user or per-cohort-total avoids the reader crediting cohort size for what is actually a retention-quality difference, or vice versa.

Worked example

Take a cohort of 200 users with weekly active counts of 200, 110, 84, 68, 58, 48, 40, 34 across weeks 0 through 7 (a sharp early drop followed by a flattening tail). As a heatmap row, that cohort's cells would be shaded 100%, 55%, 42%, 34%, 29%, 24%, 20%, 17%, and a second cohort's row underneath it lets a reader compare, cell by cell, whether a later cohort is retaining better at the same week-offset. As a line chart, the same eight numbers plotted against week 0 through 7 visually show the sharp early drop and the flattening tail; a design choice that matters here is fixing the y-axis at 0 to 100% across every cohort's line, since an auto-scaled axis on a single cohort can make a modest tail look like a dramatic plateau. As a cumulative area chart, the same cohort's total user-weeks of activity through week 7 is the running sum of active users each week, $200+110+84+68+58+48+40+34=642$ user-weeks across 200 signups, or 3.21 user-weeks per signup, a single number that a heatmap or line chart does not surface directly.

Trade-offs and pitfalls

A common mistake with the heatmap is coloring by absolute counts rather than percentages, which makes larger cohorts look artificially "better retained" purely because they have more surviving users; always normalize to a percentage of the cohort's own starting size. A common mistake with the line chart is defaulting to a linear y-axis when comparing decay rates across many cohorts, since the early sharp drop dominates the visual and hides differences in the later, flatter part of the curve that often matters more for long-term health. Since a candidate for this product-analytics track will often also be asked to work with off-the-shelf tools rather than building visualizations from raw data, it is worth knowing that tools like Mixpanel and Amplitude expose funnel and retention drop-off slightly differently (one leans toward cohort-table views, the other toward curve overlays), so validating that the properties and events are implemented correctly before trusting either tool's chart is a prerequisite step, not an afterthought. When presenting any of these views to a non-technical audience on a recurring cadence, pairing the chart with one sentence naming the most important change since the last update tends to matter more than the chart type itself.

Direct answer

Cohort analysis groups users by something they share at a fixed point in time, most commonly the week or month they signed up, and then tracks how that group behaves over subsequent periods, so that you are comparing like with like instead of blending users at very different points in their lifecycle into one trend line. An acquisition-date cohort is the most common type (grouped by signup date), and a behavioral cohort groups by a shared action instead, such as everyone who first used a specific feature in the same week.

Structured elaboration

The reason cohort analysis exists as a distinct technique, rather than just looking at a daily or weekly trend of an overall metric, is that an aggregate trend conflates two very different things: how existing users are behaving, and how the MIX of users is changing as new people join. If a product is growing fast, an aggregate "percent of users active today" trend can look flat or even improve while every individual cohort is actually retaining worse, purely because a large influx of very recent (and therefore still highly active) signups is diluting the picture. Reporting metrics by cohort instead removes that mixing effect and lets you ask a cleaner question: for people who joined at the same time, how does their behavior change as they age?

At least three retention metrics are typically reported for a cohort: day-1 retention (the fraction still active exactly one day after joining), day-7 retention (the same at one week), and rolling retention (the fraction active at any point on or after a given day, rather than on exactly that day), each answering a slightly different question about how quickly and how durably a cohort settles into use.

A concrete business use case: an e-commerce company noticing that customers acquired through a paid-search channel show markedly worse 30-day retention than customers acquired organically, even though both channels show similar day-1 numbers, would use that cohort comparison (not a blended trend line, which would hide the channel difference) to justify shifting acquisition budget toward organic-adjacent channels or investing in a channel-specific onboarding experience for paid-search users.

Worked example

A cohort of 200 users who all signed up in the same week produced the following illustrative weekly active counts: week 0 (signup week) 200 active, week 1: 110 active, week 2: 84, week 3: 68, week 4: 58, week 5: 48, week 6: 40, week 7: 34. The retention percentage for each period is the active count divided by the original 200, giving 100%, 55%, 42%, 34%, 29%, 24%, 20%, and 17%. If a second cohort acquired one month later, in a period when a new onboarding flow shipped, showed week-1 retention of 68% instead of 55% on a comparably sized cohort, that comparison (holding cohort size and week-offset fixed) is a much stronger signal that the onboarding change helped than comparing two different weeks' overall daily-active-user numbers, which would also move for reasons unrelated to the change, such as normal week-to-week traffic variation.

Trade-offs and pitfalls

Common pitfalls include comparing cohorts of very different sizes without normalizing to percentages (a cohort of 20 users retaining "50%" is much noisier evidence than a cohort of 20,000 doing the same), treating cohort analysis as interchangeable with a simple daily trend line when the two answer different questions, and forgetting that a cohort acquired very recently has an incomplete observation window, so its later-period numbers should not yet be compared directly against an older cohort's fully-observed numbers.