OKR and Metric Definition Questions

Covers translating strategic objectives into measurable key results and operational metrics that drive the right behaviors. Topics include writing clear objectives and specific, measurable key results, distinguishing leading and lagging indicators, defining primary metrics versus guardrail metrics, selecting absolute versus relative targets, avoiding perverse incentives, and ensuring metric hygiene through reliable instrumentation and data quality checks. Also addresses setting appropriate targets and time horizons, monitoring cadence, dashboard design and alerting for metric deviations, and using metrics to inform prioritization and continuous improvement without encouraging gaming.

EasyTechnical

82 practiced

You're given the Objective: 'Improve customer retention over the next quarter' for a SaaS product. Write three specific, measurable Key Results (KRs) that avoid ambiguity. For each KR include metric name, exact target, baseline, time window, and whether it is leading or lagging, plus one-sentence rationale.

MediumSystem Design

87 practiced

Design an alerting strategy for the daily revenue metric: define statistical and business-based thresholds, alert types (info/warning/critical), noise reduction techniques (smoothing, cooldowns, correlated suppression), escalation paths, and an on-call/runbook approach for investigating alerts. Explain how to avoid unnecessary pager noise.

Sample Answer

Requirements & goals:- Detect meaningful deviations in daily revenue (both drops and spikes) with low false positives, route the right severity to appropriate recipients, and enable fast troubleshooting and resolution.

Statistical vs business thresholds- Statistical (anomaly detection): - Rolling baseline: 28-day seasonally-aware baseline with day-of-week adjustments and holiday flags. - Compute z-score or Median Absolute Deviation (MAD). Alert when |z| > 3 (critical), 2–3 (warning). - Use EWMA for short-term smoothing to reduce single-day volatility.- Business thresholds: - Absolute SLAs: e.g., Revenue drop > 15% vs same weekday last 4-weeks → warning; >30% → critical. - Minimum floor: revenue < $X (cashflow risk) → critical regardless of z-score. - Upper-bound anomalies (fraud/bug): spike > +100% or unusual payment method pattern → alert.

Alert types & routing- Info: Minor anomalies (z 1.5–2 or <10% delta). Sent to BI slack channel and logged—no pager.- Warning: Intervention likely needed (z 2–3 or 15–30% drop). Email to product/ops and on-call BI during business hours.- Critical (Pager): z >3 or >30% drop or below floor. Paged to on-call revenue owner (phone/pager) 24/7.

Noise reduction techniques- Smoothing: EWMA with alpha tuned to preserve signal (e.g., 0.3).- Cooldowns: Suppress duplicate alerts for same root cause for N hours (e.g., 4–12h).- Correlated suppression: If downstream system (payment gateway) reports outage, suppress per-payment-type alerts and surface dependency incident instead.- Composite rules: require multiple signals before paging (e.g., revenue drop + increase in failed payments or drop in transactions).- Adaptive thresholds: widen during known events (sales, marketing campaigns) via calendar integration.

Escalation paths & SLAs- Critical: Page primary on-call (15 min response), if no ack → escalate to secondary (15 min), then to Engineering Manager (30 min).- Warning: Email + slack; if not acknowledged in 4 hours during business days, escalate to Product Ops.- Info: No escalation; retained in dashboard and weekly review.

On-call & runbook- On-call responsibilities: acknowledge, triage, mitigate, document.- Runbook steps: 1. Acknowledge alert, note timestamp & severity. 2. Check dashboard (daily revenue trend, transactions count, average order value, payment failures). 3. Check upstream systems (ETL pipeline jobs, data freshness, payment gateway, A/B tests, deployments). 4. Query raw tables (transactions per hour, top SKUs, refunds) with saved SQL snippets. 5. Apply quick mitigations (rollback deploy, alert payments team, disable campaign). 6. Postmortem if critical: timeline, root cause, corrective actions, update thresholds/runbook.- Provide quick SQL snippets and links in runbook for the BI on-call.

Avoiding unnecessary pager noise- Reserve pager for incidents with clear business impact (cashflow, regulatory, major customer experience).- Use graduated alerts (info → warning → critical) and require corroborating signals for paging.- Tune thresholds with historical backtesting and periodically review false positives.- Allow scheduled muting for planned events (campaigns, holidays) and calendar-driven threshold adjustments.- Monitor alert metrics (MTTA, false-positive rate) and iterate.

Operationalize- Implement in monitoring stack (Looker/PowerBI alerts + Datadog/Prometheus/CloudWatch for infra signals) or a single alerting engine that ingests BI metrics.- Store alert history and feedback loop: annotate false positives to retrain rules.

HardTechnical

75 practiced

Marketing and Product dashboards report different 'monthly active users'. Propose an automated reconciliation process: define metric contracts, nightly reconciliation queries with tolerances, automated alerts on drift, a triage workflow for root cause, and a reconciliation dashboard that tracks history, owners, and actions taken.

Sample Answer

High-level approachDefine a single source-of-truth (SSOT) MAU spec, run automated nightly reconciliations that compute both systems’ MAU per contract, detect tolerances/drift, alert owners, and provide a triage + audit dashboard to close the loop.

1) Metric contract (MAU spec)- Definition: “Unique user_id with >=1 qualifying event between month_start and month_end.”- Data sources: enumerate event tables, filters, user join keys, timezone, dedup rules, bot filters, backfill rules.- Aggregation rules: rolling 30-day vs calendar-month, cohort inclusion/exclusion, primary key.- Owners: product analytics (owner), marketing analytics (consumer), data platform (schema owner).- Versioning & tests: store contract in Git + metadata table with version, effective date, unit tests (row counts, sample queries).

2) Nightly reconciliation- Job computes contract-compliant MAU in SSOT and in marketing & product tables. Use SQL jobs (Airflow/Prefect).- Output per day: {date_window, SSOT_count, product_count, marketing_count, delta_abs, delta_pct, contract_version}.

3) Tolerances & alerting- Tolerance thresholds: warning if delta_pct > 2% or delta_abs > 500; critical if >5% or >2000 (adjustable per segment).- Automated alerts: push to Slack/email and create incident if critical. Include links to query results and top-10 contributing user segments.

4) Triage workflow- First responder: owner indicated in contract (e.g., product analytics) acknowledges alert in 15m.- Triage steps: a) Re-run queries with debug flags (show sample user_ids present in one source but not another). b) Check upstream pipelines / last successful ETL timestamps. c) Compare event-level counts, user-level dedupe differences, timezone/backfill windows. d) If code/contract mismatch, create PR to contract or pipeline fix; if data incident, create incident ticket with RCA SLA.- Decision matrix: if root cause is contract drift → update contract; if data bug → patch ETL + backfill.

5) Reconciliation dashboard- Key views: daily reconciliation trend, heatmap by cohort/region/device, top root-cause categories, open incidents, contract versions timeline.- Per-incident panel: owner, severity, detection time, acknowledgement, actions taken, resolution time, post-mortem link.- Features: ability to drill to user-level sample, rerun queries on-demand, export CSV, and annotate events.- Metrics tracked: MTTA/MTTR, frequency of contract changes, percent of days within tolerance.

Why this works- Contracts remove ambiguity; automated nightly checks catch regressions quickly; tolerances reduce noise; clear ownership and triage steps speed remediation; dashboard creates audit trail and continuous improvement loop.

EasyTechnical

81 practiced

What is a 'perverse incentive' in metric design? Provide two historical or realistic examples where a metric produced harmful behavior, describe the unintended results, and propose concrete metric redesigns or guardrails to prevent recurrence.

MediumTechnical

75 practiced

Product plans to ship 'In-app Messaging'. As BI analyst, draft an instrumentation spec listing event names, required properties and types (e.g., message_id TEXT, sender_id INT, delivered BOOL), suggested user identifiers, and derived metrics (open rate, CTR, retention lift). Describe minimal events required to compute KRs and how you'd version the spec.

Sample Answer

Instrumentation Spec — In-App Messaging (BI Analyst)

Events (names, required properties & types)1. iam_message_sent- message_id TEXT, campaign_id TEXT, template_id TEXT, sender_id INT|null, recipient_user_id INT|null, recipient_device_id TEXT|null, channel TEXT (e.g., in-app, push) TEXT, send_time TIMESTAMP, audience_segment TEXT, is_personalized BOOL, priority INT

2. iam_message_delivered- message_id TEXT, recipient_user_id INT|null, recipient_device_id TEXT|null, delivered BOOL, delivered_time TIMESTAMP, platform TEXT

3. iam_message_viewed- message_id TEXT, recipient_user_id INT|null, recipient_device_id TEXT|null, view_time TIMESTAMP, view_duration_seconds FLOAT, screen_context TEXT, autoplay BOOL

4. iam_message_clicked- message_id TEXT, recipient_user_id INT|null, recipient_device_id TEXT|null, click_time TIMESTAMP, click_target TEXT, target_url TEXT|null

5. iam_message_dismissed- message_id TEXT, recipient_user_id INT|null, recipient_device_id TEXT|null, dismiss_time TIMESTAMP, dismiss_type TEXT (user, timeout, nav-away)

6. iam_action_converted- message_id TEXT, recipient_user_id INT|null, event_name TEXT, event_time TIMESTAMP, value NUMERIC|null

Suggested user identifiers (order of preference)- user_id (INT) primary- anonymous_id / device_id (TEXT) for unauthenticated- session_id (TEXT) for session-scoped analysesInclude consistent client_timestamp and server_received_timestamp.

Derived metrics (definitions & formulas)- Delivery rate = count(delivered where delivered=TRUE) / count(sent)- Open rate (view rate) = distinct users who viewed / distinct users who received (delivered)- CTR = clicks / views OR clicks / delivered (specify both)- Conversion rate = conversions (iam_action_converted) / clicks (or delivered)- Retention lift = (DAU or 7-day retention of exposed cohort) − (DAU or 7-day retention of matched control cohort) using cohort / A/B assignment- Revenue per message = sum(value where event conversion)/messages_sent

Minimal events to compute KRs- iam_message_sent, iam_message_delivered, iam_message_viewed, iam_message_clicked, iam_action_converted plus reliable user identifiers and timestamps.

Versioning the spec- Use semantic versioning: iam_spec v1.0.0. Document changes in a changelog with date, author, reason, migration notes. Include backward-compatible additions (patch/minor) vs breaking changes (major). Embed spec_version property in each event payload (e.g., spec_version: "1.0.0") so downstream ETL can handle parsing rules and migrations.

Quality notes- Enforce required fields server-side, validate schema in ingestion (e.g., dbt tests), capture both client and server timestamps, and include sampling flags if rollout uses sampling.

Unlock Full Question Bank

Get access to hundreds of OKR and Metric Definition interview questions and detailed answers.

Join thousands of developers preparing for their dream job.