Metrics and KPI Design Questions

Direct answer

Building a defensible composite score means treating it like a small production model: put every input on the same scale, choose weights from a mix of business priors and evidence rather than a guess, decide explicitly how to handle missing components, validate the score against a real outcome before trusting it, and monitor it continuously for drift and gaming. A composite that isn't monitored will eventually be optimized by whoever's compensation depends on it.

Structured elaboration

1. Normalization. Orient every component so higher is always better, for example inverting defect rate to 1 - defect_rate. Use a bounded scale (min-max over a reasonable percentile range, such as the 1st to 99th percentile, to avoid a single outlier compressing everything else) or a robust z-score using the median and IQR (interquartile range) instead of mean and standard deviation, since it resists outliers.

2. Choosing weights. Start from business-informed priors, for example retention weighted highest because it is closest to revenue durability. Refine with evidence: model a real outcome (churn, revenue growth) on the normalized components and let the fitted relationship pull the priors toward what actually predicts the outcome, rather than starting from data alone, which overfits to whatever happened to correlate last quarter.

3. Handling missing values. Impute with a cohort-level baseline (a similar segment's median) and flag the record as imputed so downstream users can discount it. If several components are missing for a record, renormalize the weights across only the present components rather than treating a missing value as zero.

4. Validation. Check that the composite predicts a real business outcome (churn, revenue growth, support load) out of sample, not only in the data used to fit the weights. Check calibration: scores in the same band should correspond to roughly the same real outcome rate.

5. Monitoring for drift and gaming. Track each component's distribution over time, not just the composite, so a single gamed input doesn't hide inside an otherwise stable average. Watch for a component moving in isolation from the others it should move with, for example NPS (net promoter score) climbing while every other input is flat or falling.

The same construction pattern, different labels. This five-step pattern is identical regardless of what the composite is called: a customer engagement score, a customer satisfaction composite, an engagement-score-weight-validation exercise, a "golden homepage" score built to predict retention, or a hierarchical multi-product KPI model. The one structural difference is the hierarchical multi-product case: there, a single-product composite is built first with these five steps, and a second aggregation layer rolls per-product composites into one portfolio score, weighting each product's composite by its revenue share or active-user share rather than averaging unweighted, since an unweighted average lets a small product swing the portfolio score as much as the flagship product.

Worked example

This month's four normalized inputs, using min-max scaling over an assumed reasonable range for each:

$\text{Retention} = \frac{72-50}{90-50} = \frac{22}{40} = 0.55 \quad (72\%,\ \text{range } 50\%\text{-}90\%)$
$\text{NPS} = \frac{35-(-20)}{60-(-20)} = \frac{55}{80} = 0.6875 \quad (+35,\ \text{range } -20\text{ to }+60)$
$\text{Revenue per user (RPU)} = \frac{42-20}{60-20} = \frac{22}{40} = 0.55 \quad (\$42,\ \text{range }\$20\text{-}\$60)$
$\text{Defect (inverted)} = 1 - \frac{3-0}{10-0} = 1 - 0.30 = 0.70 \quad (3\%,\ \text{range }0\%\text{-}10\%)$

Using priors retention 0.4, NPS 0.2, RPU 0.3, defect 0.1:
$\text{PHS} = 0.4(0.55) + 0.2(0.6875) + 0.3(0.55) + 0.1(0.70)$
$= 0.22 + 0.1375 + 0.165 + 0.07 = 0.5925$

The product health score (PHS) for the month is approximately 0.59 on a 0-to-1 scale, or 59 if rescaled to 0-100. Recomputing this same calculation next month, with the same weights and ranges, is what makes the score comparable over time; changing the ranges or weights should be logged as a version change.

Trade-offs & pitfalls

• A single blended score is easy to communicate but destroys diagnostic information; always ship the components alongside the composite, not instead of it.
• Weights chosen once and never revisited go stale as the business changes, for example NPS becoming less predictive of retention as the product matures; schedule periodic re-validation, not a one-time fit.
• Anyone whose bonus depends on the composite has an incentive to game its weakest-verified component; monitoring is not optional polish, it is the control that keeps the score honest.
• In the hierarchical case, an unweighted average across products silently gives a low-usage product the same influence as the flagship product; weight by a real business quantity such as revenue or active users, not equally.

Direct answer

For an online marketplace, the platform's own revenue is gross merchandise value (GMV, the total value of transactions flowing through the platform) multiplied by the take rate the platform keeps as commission; GMV itself decomposes into how many customers are active, what share convert, how much they spend, and how often. Each term in that chain has its own measurement approach and its own way of quietly going wrong.

Structured elaboration

$\text{Platform Revenue} = GMV \times \text{Take Rate}$
$GMV = N_{active} \times CR \times AOV \times F$

where $N_{active}$ is active customers, $CR$ is conversion rate, $AOV$ is average order value, and $F$ is purchase frequency over the period. Separating platform revenue from GMV matters specifically for a marketplace: the buyer's total spend and the platform's actual take are different numbers, and conflating them overstates what the business itself earns.

Worked example

Suppose, for one period: 10,000 active customers, a 4% conversion rate, $60 average order value, and 1.5 orders per active customer.

$GMV = 10{,}000 \times 0.04 \times \$60 \times 1.5$

Working left to right: $10{,}000 \times 0.04 = 400$ converting customers; $400 \times $60 = $24{,}000$; $$24{,}000 \times 1.5 = $36{,}000$.

$GMV = \$36{,}000$

If the platform's blended take rate on this category is 15%:

$\text{Platform Revenue} = \$36{,}000 \times 0.15 = \$5{,}400$

The buyers collectively spent $36,000 through the marketplace, but the platform itself only earned $5,400 of that; a decomposition that stops at GMV and calls it "revenue" overstates the business's actual earnings by more than six times in this example.

Trade-offs & pitfalls

The multiplicative decomposition assumes the four GMV terms move independently, but they often do not: a promotion that raises conversion rate frequently lowers average order value at the same time (customers converting on smaller, promo-priced baskets), so a healthy-looking conversion number can mask a shrinking GMV if AOV is not checked in the same breath. Take rate is the term most often left out of a customer-facing decomposition entirely, which is the core risk for a marketplace specifically: reporting buyer GMV growth as if it were platform revenue growth misrepresents the business, especially when take rate itself is drifting down due to a mix shift toward lower-commission categories or negotiated seller discounts. Finally, purchase frequency and active-customer counts both depend on identity resolution; if that resolution logic changes (a new login system, say), every term in the decomposition can shift for reasons that have nothing to do with real buyer or seller behavior.

Direct answer

For a two-sided marketplace, I would recommend successful transactions per active user (STPAU) as the north-star metric (NSM), tracked alongside Gross Merchandise Value (GMV) as a business-outcome metric rather than the NSM itself. STPAU captures liquidity and repeat use on both sides of the marketplace, while GMV alone rewards raw transaction value in ways that can be earned without the marketplace actually getting healthier.

Structured elaboration

Candidate north-star metrics and their trade-offs

Guarding against perverse incentives

GMV alone creates a perverse incentive: a team can grow it by pushing toward fewer, larger, or more expensive transactions, for example relaxing seller quality bars to allow higher-priced listings, without improving whether a typical user can complete a transaction at all. Pairing GMV with STPAU and a small set of supporting metrics (match rate, dispute or refund rate) closes that loophole, because GMV cannot rise on the back of stagnant or declining STPAU without those supporting metrics flagging it.

Decomposition into leading indicators

• Demand-side: active buyers per week; search-to-message rate; view-to-purchase-intent conversion.
• Supply-side: active sellers per week; listings per active seller; time-to-first-response.
• Transaction flow: match rate (views to contacts); checkout abandonment rate; payment success rate.
• Quality: post-transaction ratings; dispute or refund rate.

Cross-team alignment, including a B2B buyer/supplier marketplace variant

For product and growth teams, tie roadmap items and acquisition-channel prioritization to specific leading indicators (for example, checkout improvements targeting abandonment rate), and share cohort analyses showing which channels bring high-STPAU users. For operations, attach service-level targets to the supply-side indicators (onboarding speed, moderation throughput).

In a B2B buyer/supplier marketplace, alignment looks different from a consumer marketplace: accounts (not individuals) are the unit of activity, sales cycles are longer, and relationships are managed account-by-account rather than through anonymous consumer transactions. STPAU is computed at the account level there, and growth and operations teams organize around account tiers (key accounts versus long tail) with a quarterly account-health review alongside sales, rather than around paid-acquisition-channel dashboards, since B2B liquidity is driven more by relationship management and catalog completeness than by paid acquisition.

Worked example

Baseline month: 50,000 monthly active buyers, 120,000 successful transactions, average transaction value $85.

$$\text{STPAU} = \frac{120{,}000}{50{,}000} = 2.4 \text{ transactions per active buyer}$$ $$\text{GMV} = 120{,}000 \times \$85 = \$10{,}200{,}000$$

Now suppose the growth team instead pushes listings toward higher price points, raising average transaction value to $95, while the same 50,000 buyers complete slightly fewer transactions each (2.3 instead of 2.4, because higher prices deter some purchases):

$$\text{New transactions} = 50{,}000 \times 2.3 = 115{,}000$$ $$\text{New GMV} = 115{,}000 \times \$95 = \$10{,}925{,}000$$ $$\text{GMV change} = \frac{10{,}925{,}000}{10{,}200{,}000} - 1 \approx +7.1\%$$ $$\text{STPAU change} = \frac{2.3}{2.4} - 1 \approx -4.2\%$$

GMV rose 7.1% even though the marketplace's actual liquidity, STPAU, fell 4.2% and total transaction count fell too. That divergence is exactly the gaming risk that using STPAU as the NSM (with GMV as a supporting business metric, not the target) is designed to catch.

Trade-offs & pitfalls

• No single blended metric captures both sides fairly; track buyer-side and seller-side STPAU separately, since one side can be starved while the blended average still looks fine.
• Match rate can be gamed by loosening what counts as a "match" (for example, counting a low-quality contact); pair it with a downstream conversion-to-transaction check.
• Cohort retention is the most honest signal of durable health but too slow for weekly prioritization; reserve it for quarterly reviews rather than day-to-day decisions.
• Aligning incentives requires changing how growth teams are actually compensated (often tied to acquisition volume), not just publishing a new dashboard; otherwise the perverse incentive persists at the individual level even when the company-level number is correct.

Direct answer

Map each AARRR (Acquisition, Activation, Retention, Referral, Revenue, commonly called "pirate metrics") stage to one or two channel-attributable KPIs (key performance indicators) and one visualization built for a fast, non-technical read: a stacked bar for acquisition volume and cost by channel, a funnel for activation drop-off, a cohort heatmap for retention, and a combined trend-plus-bar for revenue. The brief here focuses on acquisition, activation, retention, and revenue; referral would slot in the same way (referral rate, one viral-loop chart) if the marketing manager later asks for it.

Structured elaboration

Include a channel filter and a date-range picker so the same dashboard supports both a monthly executive glance and a campaign manager's weekly diagnostic dig, and annotate major campaign launches directly on the charts so a metric move has an obvious candidate explanation attached.

Worked example

For one channel with $10,000 spent in a month producing 500 new signups:

$$\text{CPA} = \frac{10{,}000}{500} = \$20 \text{ per acquired user}$$

Of those 500 signups, 350 complete the defined activation action (say, creating a first project):

$$\text{activation rate} = \frac{350}{500} = 70\%$$

Of the original 500 signups, 140 are still active on day 7:

$$\text{D7 retention rate} = \frac{140}{500} = 28\%$$

A marketing manager reading the funnel sees the full chain at a glance: 500 acquired at $20 each, 70% activate, and 28% of the original cohort are still around a week later, which is a very different story than any single one of those numbers shown alone.

Trade-offs & pitfalls

A single shared dashboard that tries to serve every stage equally for every audience tends to serve none of them well; an executive wants the revenue trend and retention heatmap, while a campaign manager wants the acquisition breakdown and activation funnel drilled down by specific channel and creative. The fix is one dashboard with role-based default views, not one static layout for everyone. A common mistake is choosing acquisition volume as the headline number without pairing it with cost or downstream activation, which rewards channels that bring cheap, low-intent traffic that never activates. Cohort heatmaps communicate retention trends well but can be misread by a non-technical viewer as a simple gradient rather than a time-since-acquisition comparison; a one-line caption stating what the axes mean avoids that misreading.

Direct answer

Absolute churn rate (also called customer or logo churn) is the percentage of customers lost in a period; revenue churn rate is the percentage of recurring revenue lost in that period, usually measured against starting MRR (monthly recurring revenue). They diverge whenever the customers who leave are not representative of the average customer: losing a few high-paying accounts can shrink revenue churn even when the customer count looks fine, and vice versa. A team that tracks only customer churn can miss a revenue problem building up in its largest accounts.

Structured elaboration

Definitions

• Customer churn rate = customers lost in period / customers at start of period.
• Revenue churn rate = MRR lost in period (from cancellations and downgrades) / MRR at start of period.

Because customer churn counts every account equally, it is blind to price. Revenue churn weights each departure by its dollar size, so it reacts to who left, not just how many. The gap between the two metrics is itself a diagnostic: rising revenue churn alongside flat or falling customer churn usually points at concentration risk in high-value accounts, expansion not offsetting churn among large accounts, or a pricing and segmentation shift underway.

Worked example

Start of Month 1: 100 customers. 20 large accounts at $2,000/month = $40,000 MRR, and 80 small accounts at $750/month = $60,000 MRR. Total starting MRR = $100,000 (20 x $2,000 + 80 x $750 = $40,000 + $60,000).

Month 1: 8 small accounts cancel.
$\text{Customer churn}_1 = \frac{8}{100} = 8\%$
$\text{Revenue churn}_1 = \frac{8 \times \$750}{\$100{,}000} = \frac{\$6{,}000}{\$100{,}000} = 6\%$

Month 2 (same starting base: 100 customers, $100,000 MRR): only 4 large accounts cancel, no small-account churn.
$\text{Customer churn}_2 = \frac{4}{100} = 4\%$
$\text{Revenue churn}_2 = \frac{4 \times \$2{,}000}{\$100{,}000} = \frac{\$8{,}000}{\$100{,}000} = 8\%$

Result: customer churn fell (8% to 4%) while revenue churn rose (6% to 8%), because churn shifted from small, low-ARPU (average revenue per account) accounts to large, high-ARPU accounts.

Interpretation: fewer customers left overall, but the ones who left carried disproportionate revenue weight. This points at concentration risk in the largest accounts and possibly a pricing or segmentation issue, and argues for splitting churn dashboards by ARPU band or account tier rather than reporting one blended number.

Trade-offs & pitfalls

• Reporting only customer churn understates real financial exposure when the base has a long tail of small accounts and a few large ones.
• Reporting only revenue churn can look artificially healthy if a company over-serves a few large accounts while quietly losing many small customers, masking a broader satisfaction problem.
• A common pitfall is computing revenue churn against ending MRR instead of starting MRR, or mixing expansion revenue into the same ratio; keep churn and expansion as separate line items so this ratio measures loss only.
• Net revenue churn (which nets expansion/upsell against contraction and cancellations) is a different, usually more favorable, signal than gross revenue churn; naming the exact variant matters when comparing figures across teams.