Product Metrics and Health Questions

Designing and using product specific metrics to measure user experience product health and business impact. Topics include identifying a north star metric and supporting metrics at company product and feature levels, measuring activation adoption engagement retention daily active users and monthly active users feature adoption rates and time to value, using product telemetry experimentation and funnel analysis to measure feature impact, and connecting product metrics to monetization and strategic objectives. Candidates should be able to propose metrics for new features justify trade offs instrument tracking and explain how product metrics inform prioritization roadmap and stakeholder alignment.

EasyTechnical

98 practiced

Explain cohort analysis at a high level and describe a practical use case where cohort analysis reveals information hidden by aggregate metrics. What are typical cohort dimensions (e.g., signup week, acquisition channel) and retention windows you would choose for a subscription product?

HardTechnical

96 practiced

Design a robust, versioned event schema and identity strategy for a mobile app that must support funnel analysis, experimentation, and cross-device users. Include example event definitions and properties, a plan for schema versioning, how to handle anonymous events and later stitch to user_id, and how to backfill or transform legacy events.

Sample Answer

Requirements & goals:- Support funnel analysis, A/B experimentation, cross-device user stitching, GDPR/consent, low-latency analytics.- Product + engineering need clear, versioned schema and identity rules so metrics are stable.

Event schema (examples — canonical fields present on every event):- event_name (string) — canonical enum- event_version (int) — schema version- event_id (uuid) — unique- timestamp_utc (ISO8601)- client_platform (enum: ios/android, web)- sdk_version- device_id (nullable) — anonymous device identifier- user_id (nullable) — persistent user identifier after login or identity resolution- session_id- properties (object) — event-specific payload- consent_flags (object) — analytics_opt_in: bool, marketing_opt_in: bool

Example event definitions:1) "signup_complete" v1- properties: {method: "email|google|apple", campaign_id, plan: "free|paid", referrer}2) "product_view" v2- properties: {product_id, category, price_usd, variant, position}3) "checkout_start" v1- properties: {cart_value_usd, items_count, coupon_code}

Schema versioning plan:- Semantic event_version per event_name (major integer). Changes that break downstream semantics -> increment major.- Add changelog file in repo and immutable JSON schema for each event_name+version.- Validate at ingestion with schema registry (reject or flag mismatches).- Deprecation policy: support old versions for 90 days, emit warnings to producers.

Identity & stitching strategy:- Primary keys: user_id (persistent, server-issued or auth provider) and device_id (client-generated stable per install).- Anonymous flow: emit events with device_id + anonymous_id (uuid) before login; user_id null.- On login/identify event: emit "identify" event with user_id, device_id, anonymous_id, timestamp. In ingestion pipeline, run deterministic join to attribute prior anonymous events to user_id by backfilling user_id for session window (e.g., events from same device_id or cookie in last 30 days).- For cross-device: use hashed deterministic identifiers for matching when users link accounts (email hash with salt stored server-side). Do not store raw emails in analytics.

Privacy & security:- Hash PII (email) with salt before sending. Respect consent_flags: drop/skip sending when opted out.- TTL for device_id and anonymous data per privacy rules.

Backfill & legacy transform:- Maintain mapping table: legacy_event_name -> canonical_event_name + transformation script.- Create ETL jobs to: - Parse legacy events, map fields, populate missing canonical fields (e.g., set event_version=1). - Emit synthetic "identify" events where server auth logs show join between device identifier and user_id. - Backfill historical aggregates into data warehouse with provenance tags (source_version, transform_job_id).- Validate with checksum comparisons and replay sampled legacy logs to a staging analytics pipeline.

Governance & operationalization:- Analytics MAINTAINER role approves event additions/changes via PR with example payloads and business semantics.- Provide SDK helper methods to enforce canonical fields and automatic identify() call on login.- Automated tests: unit schema checks, integration tests that replay sample events, and daily monitoring of event volumes and schema drift.- Documentation: living event catalog (searchable), examples, ownership, downstream impact matrix.

Why this works:- Versioning protects analytics consumers from breaking changes.- Deterministic identify + server-side confirmation creates robust cross-device stitching while preserving privacy.- Backfill strategy preserves historical comparability and creates traceable transformations so experiments and funnels remain reliable.

MediumTechnical

88 practiced

Define SLOs for product health for an app with 100k daily active users. Propose Service Level Objectives and alerting thresholds for: (1) signup success rate, (2) activation within 7 days, and (3) API error rate. For each SLO give a target, an alerting threshold, and recommended immediate and follow-up actions.

Sample Answer

Situation summary: For a 100k DAU consumer app, product health SLOs should balance user experience, business impact, and operational noise. Below are pragmatic SLOs, alert thresholds, and runbook actions for signup success, 7‑day activation, and API error rate.

1) Signup success rate- SLO (target): 99.5% of signup attempts succeed (weekly rolling window). (~0.5% failure tolerated)- Alert threshold: Page alert at 99.0% (warning); P1 alert at 98.0% sustained >10 minutes or >1% drop vs baseline over 5m.- Immediate actions: (on-call) verify recent deploys, check auth/DB errors, rollback if obvious regression, open incident ticket, enable bypass flow if available.- Follow-up actions: Root-cause analysis, fix deployment/process, add automated E2E signup tests, instrument more granular metrics (step-level failure rates), update runbook.

2) Activation within 7 days (new user activation: key event e.g., first purchase or core action)- SLO (target): 60% of new signups activate within 7 days (measured per cohort weekly).- Alert threshold: Warning if 7-day activation drops 10% relative to previous 4-week moving average; P2 if drop >20% or absolute <50%.- Immediate actions: Check onboarding funnel telemetry (dropoff steps), review recent messaging/experiments, pause new experiments that could affect flows.- Follow-up actions: A/B test onboarding tweaks, qualitative research with dropped users, prioritize product fixes in roadmap, monitor segmented cohorts (device, geo, channel).

3) API error rate (5xx + client-impacting 4xx on core APIs)- SLO (target): <0.2% error rate across core production APIs (p95 window daily).- Alert threshold: Warning at 0.5% sustained >5m; P1 at >1% sustained >2m or latency spike p95 >2s.- Immediate actions: Throttle non-essential traffic, failover to healthy region/replica, identify noisy endpoints, engage SRE to scale or patch.- Follow-up actions: Capacity planning, add retries/exponential backoff, improve circuit breakers, add synthetic canaries and per-endpoint alerts, postmortem and schedule engineering work.

Notes: Choose measurement windows (rolling 7d/28d) aligned to business cadence; tier alerts to reduce noise; tie SLOs to business impact (e.g., revenue per activation) when prioritizing fixes.

EasyTechnical

91 practiced

List and justify company-level, product-level, and feature-level supporting metrics for an e-commerce marketplace. For each metric explain what it signals, how it is calculated, possible ways it can be gamed or misinterpreted, and the likely product or business actions you might take if the metric trends down.

Sample Answer

Company-level metrics1) Gross Merchandise Value (GMV)- Signals: overall marketplace health and demand.- Calculation: sum of transaction value (orders) in a period.- Gaming/misinterpretation: promotions, returns, or fake orders inflate GMV; high GMV with low margin can be misleading.- If down: investigate traffic, conversion, high-level retention; assess marketing funnels, seller supply, pricing strategy; run cohort analysis and competitor pricing check.

2) Take Rate / Revenue- Signals: monetization effectiveness.- Calculation: platform fees + ads + other revenue ÷ GMV.- Gaming: shifting revenue recognition or one-off deals; discounting increases GMV but lowers take rate.- If down: evaluate pricing, fee structure, ad product performance; A/B test fee changes; diversify revenue streams.

3) Monthly Active Users (MAU) & DAU/MAU ratio- Signals: reach and engagement.- Calculation: unique users who interacted in month; DAU/MAU shows stickiness.- Gaming: bots, multiple accounts, broad definition of “active.”- If down: audit acquisition channels, onboarding, re-engagement campaigns, product-market fit signals.

Product-level metrics1) Conversion Rate (visits → purchase)- Signals: product experience, listing quality, checkout friction.- Calculation: purchases ÷ unique sessions (or purchases ÷ visits).- Gaming: fake purchases, low-quality buyers; discounts can temporarily boost conversion.- If down: run funnel analysis (search → view → add-to-cart → checkout), AB test UI/checkout fixes, improve listings, optimize search relevance.

2) Average Order Value (AOV)- Signals: spend per transaction.- Calculation: GMV ÷ number of orders.- Gaming: bundling to inflate AOV, refunds reduce effective AOV.- If down: cross-sell/upsell, bundle recommendations, free-shipping thresholds, pricing experiments.

3) Listing Availability / Fill Rate- Signals: supply-side health and selection.- Calculation: % of user searches with at least one available listing.- Gaming: stale listings or front-loaded inventory.- If down: recruit sellers, improve seller onboarding, incentivize restocking, adjust marketplace policies.

Feature-level metrics (examples tied to specific features)1) Search Relevance Score (click-through on top 5 results)- Signals: how well search surfaces relevant items.- Calculation: clicks on top 5 results ÷ impressions for query.- Gaming: SEO manipulation, title stuffing causing irrelevant clicks.- If down: refine ranking algorithm, incorporate behavioral signals, improve query understanding.

2) Add-to-Cart Rate for a product page- Signals: product appeal and price fit.- Calculation: add-to-cart events ÷ product page views.- Gaming: accidental clicks, bots.- If down: improve photos/descriptions, price adjustments, trust signals (reviews), A/B test CTA placement.

3) Checkout Abandonment Rate- Signals: friction in checkout or trust/payment issues.- Calculation: initiated checkouts not completed ÷ initiated checkouts.- Gaming: staged checkouts or testing behavior.- If down: simplify checkout, add payment methods, show costs earlier, retry flows, address error rates.

For each metric, always pair with qualitative research (user/session recordings, seller interviews) and segment the metric (new vs returning, cohort, channel) to avoid chasing noise. Prioritize actions by impact and ease of implementation and set experiments to validate fixes before broad rollouts.

EasyTechnical

76 practiced

Explain the difference between retention and churn. Provide formulas for computing a simple period retention rate and churn rate. Given MAU numbers Month1=1000, Month2=900, Month3=850, calculate month-over-month retention and churn for Month2 and Month3 and interpret whether retention is improving or worsening.

Unlock Full Question Bank

Get access to hundreds of Product Metrics and Health interview questions and detailed answers.

Join thousands of developers preparing for their dream job.