Airbnb Mid-Level Data Analyst Interview Preparation Guide

Direct answer
The core visualization best practices for non-technical executive dashboards are: match the chart type to the task, use color sparingly and only to signal something meaningful, label and annotate so the chart is self-explanatory without a caption, simplify to only what's necessary for the decision at hand, avoid misleading axes, and choose a table over a chart whenever exact numbers matter more than a visual pattern.

Structured elaboration

1. Chart-type selection: pick trend, comparison, distribution, or relationship encodings deliberately based on the task, not habitually defaulting to whatever chart type is fastest to build.
2. Color usage, including accessibility: use color to signal status or highlight the key data point, not decoratively; ensure the palette is colorblind-safe and has sufficient contrast, since an executive audience is just as likely to include colorblind viewers as any other.
3. Labeling and annotation: axis labels with units, a clear title, and a callout for the key takeaway so the chart doesn't require a live narrator to be understood if viewed later or forwarded.
4. Simplifying views: cut anything not needed for the specific decision this dashboard supports; an executive view should show fewer, more curated elements than an analyst view of the same data.
5. Avoiding misleading axes: zero-based bar charts, clearly-labeled axis ranges, and no unnecessary dual axes.
6. Tables vs. charts: use a table when the audience needs to read exact values precisely (e.g. comparing many similarly-sized numbers), and a chart when the audience needs to perceive a pattern, trend, or relative comparison at a glance.

Worked example
An executive revenue-by-region view ranks four regions on a sorted bar chart (not a pie) for fast comparison: West at $4.2M, East at $3.8M, South at $2.1M, and North at $1.4M. The axis starts at zero, the bars are a neutral gray except North, which is highlighted in a colorblind-safe red because it's the only region below its $1.8M target, and a one-line annotation on North's bar reads "$400K below target," naming the key takeaway directly rather than leaving the viewer to compute West-minus-target math themselves. A companion small table gives the exact dollar figures (and each region's individual target) for anyone who needs precision beyond what the sorted bar's visual ranking provides.

Trade-offs and pitfalls
Applying every one of these principles to every single chart can itself become a rigid checklist exercise; the underlying goal is always "can this specific audience make a fast, correct read of this specific chart," and each principle should be judged against that goal rather than applied mechanically.

Situation: An urgent pipeline incident is causing delays in daily revenue reports and downstream alerts, so analysts and engineers must act immediately.

Script to convene teams (30–60s, used in Slack/phone bridge invite):
"Hi all — we're declaring a priority incident: ETL pipeline failures impacting today's revenue reports and alerting systems. Engineers: we need a triage lead and one owner on infra and one on data model. Analysts: we need a lead to validate downstream metrics and produce a rolling impact summary. Meeting in 5 minutes on this bridge: [link]. Goals for first 15 minutes: confirm scope, assign owners, agree on mitigations and a 30‑minute stakeholder update cadence. Post here if you can’t join."

Translating business impact into engineering priorities:

• Business impact: revenue reporting delayed, customer invoices and SLA alerts risk — prioritize fixes that restore data freshness for revenue tables first.
• Engineering priorities: (1) triage and stabilize ingestion jobs (highest priority), (2) ensure data integrity for revenue aggregates (validation tests), (3) restore alerting pipelines, (4) implement temporary mitigations (replay/job-run) to minimize business disruption, (5) schedule root-cause RCA and permanent fix.
• I’ll provide analysts’ validated symptom list and failing query examples so engineers can reproduce and scope risk.

One‑sentence stakeholder update:
"We're investigating a pipeline failure affecting today's revenue reporting; engineering is triaging ingestion and integrity checks now — we'll provide a 30‑minute update with expected ETA and mitigation steps."

Direct answer
Autonomy should track demonstrated judgment in a specific domain, not tenure or title, and it should be granted and withdrawn through visible, structural mechanisms, not just a private mental model of how much you trust someone. As someone grows from junior to senior, both the default level of guidance and the criteria for changing it should become more explicit, not less.

What determines the level, not just the person's level

• Domain-specific, not global: someone can have earned full autonomy in one area (their core service) and need more guidance in an adjacent one (security-sensitive changes) they haven't touched before. Treating autonomy as a single dial per person rather than per domain misjudges both directions.
• Base it on evidence: track record of decisions in that specific domain, not just general seniority or how long they've been on the team.

The conversation isn't enough, structure it

• Guidance and autonomy shouldn't live only in how much you check in; they should be encoded in the system itself. Concretely: mandatory review gates on certain categories of change, feature flags that let risky work ship dark before it's fully trusted, and automated checks (tests, linting, policy gates) that catch the class of mistake a specific person is prone to, rather than relying on a human remembering to look for it.
• This matters especially early: a junior engineer with a mandatory review gate on production-config changes isn't being distrusted personally, the system is compensating for a domain they haven't yet built judgment in, and that's a much less fraught conversation than "I don't trust your judgment yet."

Moving the dial, in both directions

• Define, in advance, what "graduating" out of a guardrail looks like: a number of changes in that domain reviewed without a significant issue, or a specific type of decision made correctly under supervision. Vague criteria ("when I feel comfortable") makes the process feel arbitrary to the person on the other side of it.
• The dial also needs to move backward cleanly. If someone senior makes a judgment error in a domain, temporarily reintroducing a guardrail (an extra review, a smaller blast radius) shouldn't read as a permanent demotion; it should be scoped to the specific domain and have the same kind of explicit, objective path back out.

How this shifts junior to senior

• Junior: guidance is broad and mostly structural (required reviews, smaller scoped tasks, pairing), because there isn't yet enough track record to know where the real gaps are.
• Mid-level: guidance narrows to the specific domains where judgment hasn't been tested yet, while proven domains get real autonomy.
• Senior: guidance becomes mostly about the highest-blast-radius decisions (irreversible changes, cross-team commitments) rather than day-to-day execution, and the structural safeguards that remain exist because the stakes are higher, not because trust is lower.

Worked example
A mid-level engineer had strong judgment in their core service but hadn't touched the deployment pipeline before. Rather than a blanket "you need approval on everything" or "you're trusted, go ahead," the guidance was scoped to that specific gap: full autonomy on their usual work, a mandatory review plus a feature flag for anything touching the deploy pipeline, with an explicit criterion stated up front (three pipeline changes reviewed cleanly, then the mandatory review comes off for that category specifically). That made the guardrail feel like a scoped, temporary compensation for an actual gap rather than a general judgment about their competence, and removing it was a specific, visible moment rather than something that just quietly happened.

Trade-offs and pitfalls

• Treating autonomy as all-or-nothing per person, rather than per domain, either over-restricts someone who's earned trust in most areas or over-extends them into an area they haven't proven yet.
• Relying purely on personal judgment about who to trust, without structural backstops (review gates, flags, automated checks), doesn't scale past a small team and creates inconsistency that reads as favoritism.
• Leaving the criteria for regaining autonomy vague turns a guardrail into something that feels indefinite and punitive, even when it was scoped and reasonable at the start.

A distinct-count-per-period metric like Daily Active Users needs COUNT(DISTINCT user_id) specifically, not a plain COUNT, because a plain COUNT would count every event, over-counting any user who was active more than once in the same period.

Structured elaboration

SELECT DATE(occurred_at) AS day, COUNT(DISTINCT user_id) AS dau
FROM events
GROUP BY day;

COUNT(DISTINCT user_id) deduplicates within each day's group, so a user who logged three events on the same day still contributes exactly 1 to that day's count, correctly answering "how many distinct users were active", not "how many events happened". The GROUP BY-based alternative pushes the deduplication into an inner query instead of an aggregate function: SELECT day, COUNT(*) AS dau FROM (SELECT DISTINCT DATE(occurred_at) AS day, user_id FROM events) t GROUP BY day. The inner SELECT DISTINCT collapses each user down to at most one row per day first, so by the time the outer COUNT(*) runs there's nothing left to deduplicate, it's just counting already-unique rows.

Worked example

Given events for user 1 twice and user 2 once, all on 2025-01-01, and user 3 once on 2025-01-02: the query correctly returns DAU of 2 for 2025-01-01 (users 1 and 2, with user 1's repeat event not double-counted) and 1 for 2025-01-02 (user 3).

Trade-offs and pitfalls

Both forms are logically equivalent and, on the worked example below, return the identical numbers; which one an optimizer executes faster can differ by engine and data distribution, so it's worth trying both against real data rather than assuming one is always better.

COUNT(DISTINCT ...) is measurably more expensive than a plain COUNT at large scale, since the engine typically needs to sort or hash the values to deduplicate them; for very high-cardinality, very large-scale distinct counting (billions of events per day), an approximate distinct-count function (HyperLogLog-based, such as Postgres's hll extension or BigQuery's APPROX_COUNT_DISTINCT) trades a small, bounded error for dramatically better performance, a trade-off worth naming even though it's outside this topic's core scope.

For a strictly multiplicative funnel, the surprising and important result is that a fixed percentage increase in the final output requires the exact same relative lift at every step, so 'which step needs the smallest relative uplift' has a trick answer: they're identical, and the real decision criterion has to be feasibility, not the arithmetic.

Worked example

Given funnel counts: product views = 1,000,000, add-to-cart = 200,000, checkout starts = 80,000, purchases = 40,000, and a target of a 25% increase in purchases without increasing acquisition (views held fixed).

The funnel is a chain of conversion rates:
$\text{views} \to \text{cart}: r_1 = \frac{200{,}000}{1{,}000{,}000} = 0.20$
$\text{cart} \to \text{checkout}: r_2 = \frac{80{,}000}{200{,}000} = 0.40$
$\text{checkout} \to \text{purchase}: r_3 = \frac{40{,}000}{80{,}000} = 0.50$
$\text{Purchases} = \text{Views} \times r_1 \times r_2 \times r_3 = 1{,}000{,}000 \times 0.20 \times 0.40 \times 0.50 = 40{,}000$

Target purchases = $40{,}000 \times 1.25 = 50{,}000$. Solving for the new rate needed if you move ONLY $r_1$ (holding $r_2, r_3$ fixed):
$r_1' = \frac{50{,}000}{\text{Views} \times r_2 \times r_3} = \frac{50{,}000}{1{,}000{,}000 \times 0.40 \times 0.50} = 0.25$
Relative increase $= (0.25 - 0.20)/0.20 = 25%$. Repeating the same algebra for $r_2$ (new rate 0.50, relative increase 25%) and $r_3$ (new rate 0.625, relative increase 25%) gives the identical 25% relative increase in every case. This is not a coincidence: because Purchases is a pure product of independent factors, moving any single factor by a relative fraction $x$ moves the output by exactly $x$ as well.

Which step to actually target, and why

Since the required relative lift is mathematically tied, the real selection criterion is which step is CHEAPEST to move by 25% given the mechanics of that stage: a checkout-to-purchase step (0.50 to 0.625) is often the easiest to move with UX fixes (removing friction, adding trust signals, one-click checkout) because it's closer to a completed decision; a views-to-cart step (0.20 to 0.25) usually requires improving product-market fit or targeting, which is slower and less certain. Historical elasticity data (which step has moved fastest in past experiments) should break the tie, not the arithmetic.

Trade-offs and pitfalls

A candidate who jumps straight to 'target the step with the lowest current conversion rate' is making an unjustified leap; low absolute conversion doesn't imply low required relative lift or low cost to move it. Always do the algebra first, then reason about feasibility.

Direct answer

Missingness comes in three flavors that differ in whether the reason data is missing depends on unobserved information. MCAR (missing completely at random) means missingness is pure noise, unrelated to anything. MAR (missing at random) means missingness depends only on variables you did observe. MNAR (missing not at random) means missingness depends on the value that's actually missing, or on something else you never measured. The right strategy, and how defensible your fix is, changes completely across these three cases, so diagnosing which one you're likely in comes before choosing a remedy.

Structured elaboration

Definitions

• MCAR: $P(\text{missing} \mid X_{obs}, X_{miss}) = P(\text{missing})$. The missingness mechanism is independent of both observed and unobserved data.
• MAR: $P(\text{missing} \mid X_{obs}, X_{miss}) = P(\text{missing} \mid X_{obs})$. Missingness can depend on things you measured (e.g. users on older app versions are less likely to have a field logged), but not on the missing value itself once you condition on what's observed.
• MNAR: missingness depends on the unobserved value itself (e.g. users who would have reported very low satisfaction are less likely to complete the survey at all).

Strategy per type

Diagnostics to justify the assumption

1. Compare observed-variable distributions between missing and non-missing groups. Run a two-sample test (t-test for continuous, chi-square for categorical) on each observed covariate, split by whether the target field is missing. Systematic differences argue against MCAR and toward MAR (or possibly MNAR, since this test can't distinguish MAR from MNAR).
2. Fit a model predicting missingness from observed covariates. If observed variables predict missingness well, that's consistent with MAR (or MNAR); if they predict it poorly and no other systematic pattern is found, MCAR is more plausible, though this is never proof.
3. Little's MCAR test, a formal global test of whether missingness patterns are consistent with MCAR across multiple variables jointly. Useful as a first pass, but has limited power and doesn't rule out MNAR.
4. Sensitivity analysis under MNAR. Vary an assumed MNAR mechanism (e.g. "missing values are systematically X% lower than the observed distribution suggests") and see how much the conclusion moves. If conclusions are stable across a reasonable range of assumptions, that increases confidence even without proving MAR.
5. External validation, where available. If you can get a small, cost-justified sample where missingness is forced to zero (e.g. a mandatory follow-up survey to a random subsample), compare it to the model's implied distribution.

Worked example

Suppose a product logs "time to first purchase" but 15% of records are missing this field. A logistic regression predicting missing (1/0) from observed covariates (signup channel, device type, account age) shows a large, significant coefficient on device type: mobile-web signups are missing this field at 3x the rate of app signups (a plausible instrumentation gap, since a purchase event on mobile-web wasn't always logged with the field). That pattern is evidence for MAR conditional on device type, not MCAR: the observed covariate (device type) explains the missingness. The fix is not to drop these rows (that would systematically undercount mobile-web purchasers) but to run multiple imputation for time to first purchase that includes device type as a predictor, or to compute the metric within device-type strata separately and combine with appropriate weights. If, after controlling for device type, missingness were still correlated with the (unobserved) purchase amount itself, e.g. large purchases are less likely to be logged because they route through a different, less-instrumented checkout flow, that would be an MNAR signature no amount of covariate adjustment can fix without further data.

Trade-offs & pitfalls

• You cannot statistically prove MAR from the observed data alone. MAR and MNAR are observationally indistinguishable without either an untestable assumption or external ground truth; the diagnostics above build a case, not a proof.
• Little's test has limited power, especially with many variables or a small missingness fraction; a non-significant result is not strong evidence for MCAR.
• Complete-case analysis under MAR (not MCAR) is a common silent bug. It's fine under MCAR but introduces bias under MAR, since the missingness is now systematically related to something you kept.
• Multiple imputation needs the imputation model to include everything the analysis model uses, including the outcome variable itself, or it will bias the very relationship you're trying to estimate.
• Sensitivity analysis under MNAR is about honesty, not certainty. The deliverable is "here's how much the conclusion could shift under plausible mechanisms," not a single corrected number.

Direct answer

When a client wants to independently verify your result, the deliverable is not just the finding, it's a minimal, self-contained package they can rerun themselves: the code that produced the numbers, a description of how to get equivalent data (or a synthetic/sanitized stand-in for it), and clear enough documentation that someone outside your team can execute it without you in the room.

Structured elaboration

1. Decide what is actually reproducible versus what has to stay described.
The code and the analysis logic should always be reproducible in full. The underlying data usually cannot be handed over as-is if it contains customer PII (personally identifiable information, such as names, emails, or account numbers), proprietary business data, or anything covered by a data-sharing agreement. The fix is not to skip reproducibility, it's to separate 'reproduce the LOGIC exactly' from 'reproduce the DATA exactly,' and hand over an artifact for each: the real code, plus either (a) a clearly labeled synthetic dataset with the same schema and similar statistical properties, or (b) precise instructions for how the client can pull the equivalent data from their own systems if they have access to comparable sources.

2. Package the minimal artifact set.
At minimum: the analysis code itself (scripts or a notebook, not just a slide describing the method), a requirements/environment specification (exact library versions, because 'it ran on my machine' is not reproducible), a data dictionary describing every column the code expects, and a short README describing the exact sequence of steps from raw input to the reported number. Anything beyond this (internal dashboards, ad hoc exploration) is noise the client did not ask for and should not be included.

3. Protect sensitive data without breaking reproducibility.
Three common patterns, useful in combination: synthetic data generation that preserves the schema and rough distributional shape of the real data without being traceable to real records; a sanitized sample where identifying fields are removed or hashed but the analytical structure is intact; or a documented data-access pattern (exact query, exact filters, exact time window) the client can run against their own copy of the data if they already have access to it. State explicitly which of these you used and why, so the client understands they are validating the LOGIC, and, if they used the synthetic data, that the specific NUMBERS may differ from production.

4. Document execution steps as if the reader has never seen the project.
A numbered list of exact commands (not prose describing what to do) is the standard that actually gets used: install X, run script Y with these arguments, expect output Z. Include the expected output or a checksum/summary statistic so the client knows immediately if their run matches yours or has diverged (and if it diverges, what that would signal, e.g. an environment or data mismatch rather than a code bug).

Worked example

A consulting analytics team tells a retail client that a new pricing rule increased average order value by 6.2%. The client's finance team is skeptical and wants to run it themselves. The team delivers: (1) the exact SQL/Python transformation code that computes average order value pre- and post-change, version-pinned to specific library versions in a requirements file, (2) a synthetic transactions dataset of 50,000 rows generated to match the real schema and the real data's approximate order-value distribution (mean and spread matched, no real customer identifiers), with a clear README stating this is synthetic and will not reproduce the exact 6.2% figure, only the method, (3) a one-page data dictionary defining every column, and (4) a documented query pattern the client's own analysts can run against their live warehouse, with the exact date range and filters used, so they can reproduce the real 6.2% figure against their own data if they choose to. The client's team runs the synthetic-data version, confirms the logic matches what was described, and separately reruns the documented query against their own warehouse to confirm the real number.

Trade-offs and pitfalls

• The most common failure is handing over a notebook that ran once on someone's laptop with no environment pinning; without exact versions, 'reproducible' code frequently produces silently different results months later.
• Do not confuse 'gave them the code' with 'gave them something they can run'; if the client needs data access, credentials, or infrastructure you did not describe, it is not actually reproducible for them.
• Synthetic data is a compromise, not a substitute for the real validation path; always be explicit that synthetic-data reruns validate the METHOD, and offer the real-data query pattern as the path to validating the actual NUMBER.
• Over-scoping the package (handing over your entire internal codebase or every exploratory notebook) creates a support burden and a larger attack surface for something to go wrong; keep the package to exactly what reproduces the stated result.

An outlier is a real, legitimate data point that happens to be extreme; an anomaly is a data point that exists because something went wrong. The two categories overlap in appearance but call for opposite treatment.

Distinguishing them

A customer whose spending is genuinely 50 times the median because they run a large business is a legitimate outlier: keep it in the data, since a model or report that silently drops it distorts reality by pretending large customers don't exist. A sensor reading of `-999` because a device reset to its sentinel default is an anomaly: it doesn't describe anything real and should be quarantined or corrected, not treated as valid signal.

Worked example

A single day's revenue spiking 5x during a well-publicized flash sale is an outlier (a real business event); the exact same 5x spike appearing because a batch job accidentally double-counted every transaction that day is an anomaly (a data-quality defect) with an identical statistical signature but an opposite correct response.

Trade-offs and pitfalls

Because the two can look statistically identical, the fastest reliable triage checklist is: check for known instrumentation failure modes first (a clock reset, a sentinel default value, a timestamp misalignment between two systems), since these have a distinctive, checkable signature (repeated identical sentinel values, an impossible timestamp) that a genuine business event won't share. Only once those specific failure modes are ruled out should you treat an extreme value as a probably-legitimate outlier. Getting this triage wrong in either direction is costly: quarantining a real outlier hides a genuine business signal from decision-makers, while keeping a genuine anomaly in the data corrupts every downstream aggregate that includes it.

Direct answer

Strip currency symbols and separators with a rule that explicitly handles the ambiguous cases (which character is a thousands separator versus a decimal point, and parenthetical negatives), and treat anything you cannot confidently parse as NULL rather than guessing, since a silently wrong number is worse than a visibly missing one.

Structured elaboration

1. Strip presentation, keep meaning: remove currency symbols and any character that isn't a digit, separator, or sign, before attempting to parse the number itself.
2. Resolve the thousands-vs-decimal ambiguity: 1,234.56 (US-style, comma is thousands, period is decimal) and 1.234,56 (many European locales, reversed) look similar but mean different things; a heuristic based on which separator appears last and how many digits follow it can resolve the common cases, but truly ambiguous inputs (like a bare 1,234 with no decimal at all) may need an explicit locale hint from the source rather than a guess.
3. Parenthetical negatives: accounting notation writes a negative amount as (500.00); detect the wrapping parentheses and negate the value, don't let the parentheses become stray characters that break the parse.
4. NULL and invalid markers: an explicit NULL string, an empty string, or a genuinely unparseable value should all become a real NULL in the output, not a zero (zero is a real, different value) and not a parse exception that halts the batch.
5. Fail closed on ambiguity: when a format truly cannot be disambiguated (e.g. a space-separated European thousands format combined with a comma decimal, without knowing the source locale), the defensible choice is to flag it for review rather than silently picking an interpretation that might be wrong half the time.

Worked example

-- Postgres-style; strip symbols/commas, handle parenthetical negatives
SELECT
  CASE
    WHEN raw_amount IS NULL OR TRIM(raw_amount) IN ('', 'NULL') THEN NULL
    WHEN raw_amount LIKE '(%)' THEN
      -1 * REPLACE(REGEXP_REPLACE(raw_amount, '[^0-9.]', '', 'g'), ',', '')::NUMERIC
    ELSE
      REPLACE(REGEXP_REPLACE(raw_amount, '[^0-9.,]', '', 'g'), ',', '')::NUMERIC
  END AS amount_numeric
FROM transactions;

Verified in Python against representative cases: '$1,234.56' correctly parses to 1234.56; '(500.00)' correctly parses to -500.00 (negative); 'NULL' and '' correctly become NULL rather than 0 or an error; a plain '1234' correctly parses to 1234.0. A space-separated European format like '1 234,56' is flagged as a documented gap in the simple heuristic version above, exactly the kind of ambiguous input a careful answer calls out explicitly rather than silently mis-parsing.

Trade-offs and pitfalls

• A "best-effort" parser that never fails closed will, sooner or later, silently mis-parse an ambiguous format as a plausible-looking but wrong number; state explicitly which formats you're confident about and which you're not.
• Regex-based stripping of "everything except digits and separators" can accidentally strip a genuinely meaningful character if the input contains an unexpected symbol (a stray currency code letter, for instance); validate the result's shape (does it look like a number at all?) before casting, not just after.
• This same parsing competency generalizes directly to multi-currency normalization (converting the now-clean numeric amount into a single reporting currency using a historical exchange rate), which is a related but separate concern once the raw string is safely a number.

Direct answer

Small cohorts produce noisy retention rates because a rate computed from a handful of users has wide sampling variation even when nothing about underlying behavior has changed, and two standard techniques address this directly: bootstrapped confidence intervals, which quantify how much a small cohort's rate could plausibly have varied by chance, and empirical Bayes (beta-binomial) smoothing, which pulls each small cohort's estimate partway toward a shared, more stable estimate borrowed from other similar cohorts.

Structured elaboration

A bootstrapped confidence interval works by resampling the observed users within a cohort with replacement many times, recomputing the retention rate on each resample, and using the spread of those resampled rates to build an interval around the observed rate; it does not change the point estimate itself, only communicates how much uncertainty surrounds it, which is often enough on its own to stop a team from over-reacting to a single small cohort's number. Empirical Bayes smoothing goes further and actually adjusts the point estimate: it treats each cohort's true retention rate as drawn from a shared distribution across cohorts, estimates that shared distribution's parameters from the pooled data, and then shrinks each individual cohort's noisy observed rate toward the pooled mean, weighted by how much data that cohort has (a cohort with very few users gets pulled hard toward the pooled estimate; a cohort with many users is barely adjusted).

A minimum cohort-size threshold is the simplest and cheapest of the three: below some chosen size, a cohort's individual rate is simply not reported on its own, and is instead rolled into a combined bucket with neighboring cohorts until the combined size clears the threshold.

Worked example

import numpy as np
np.random.seed(3)

true_rate = 0.25
cohort_sizes = [30, 35, 40, 45, 50, 55, 60, 32, 48, 38]
observed = [np.random.binomial(n, true_rate) for n in cohort_sizes]
raw_rates = [o / n for o, n in zip(observed, cohort_sizes)]

pooled_rate = sum(observed) / sum(cohort_sizes)
k = 20  # prior strength, in "pseudo-users"
shrunk_rates = [(o + k * pooled_rate) / (n + k) for o, n in zip(observed, cohort_sizes)]

print('raw_rate range:', round(min(raw_rates), 3), '-', round(max(raw_rates), 3))
print('raw std:', round(np.std(raw_rates), 4), '  shrunk std:', round(np.std(shrunk_rates), 4))
print('pooled_rate:', round(pooled_rate, 3))

Simulating 10 small cohorts (sizes 30 to 60 users) that ALL share the same true 25% retention rate, with a fixed seed, the raw observed rates range from 14.6% to 32.7% purely from sampling noise, a spread of over 18 percentage points despite there being zero real difference in the underlying rate. Applying beta-binomial shrinkage (pooling the 10 cohorts to estimate a shared prior, then blending each cohort's own count with that prior) reduces the standard deviation of the estimates across the 10 cohorts from 0.058 to 0.041, and every shrunk estimate moves closer to the pooled rate of 24.0%, which itself is close to the true 25% rate used to generate the data. A team looking only at the raw rates might have concluded the 32.7% cohort was meaningfully outperforming the 14.6% cohort; the shrinkage result shows both were consistent with the same underlying rate once sampling noise is accounted for.

Trade-offs and pitfalls

Displaying a smoothed value rather than the raw observed rate is appropriate when the audience's decision depends on comparing many small cohorts against each other or against a benchmark, since the raw numbers would otherwise mislead by amplifying noise into apparent differences; displaying the raw value alongside a wide confidence interval is more appropriate when the audience needs to see the actual observed data and understand its uncertainty rather than a blended estimate. A pitfall specific to shrinkage is applying it uniformly without disclosing that it has happened, since a stakeholder unaware that a number has been smoothed may draw conclusions as if it were the raw observed rate.