Learn to design dashboards that support decision-making. Organize metrics by user role: executives care about North Star and business outcomes; team leads care about specific channel or feature metrics. Include trends, comparisons, and targets. Practice explaining the purpose of each dashboard tier and how it enables specific decisions.
EasyTechnical
24 practiced
Explain the difference between monitoring dashboards and diagnostic dashboards. For a high-traffic checkout process, provide two examples of monitoring metrics a BI Analyst would display and two diagnostic dashboards or queries you would prepare to investigate an alert.
Sample Answer
Monitoring dashboards vs diagnostic dashboards:- Monitoring dashboards: high-level, real-time views that track health and KPIs to detect issues quickly. They prioritize clarity, thresholds, and alerts (e.g., green/yellow/red) so stakeholders can see system status at a glance.- Diagnostic dashboards: investigation tools used once an alert fires. They expose detailed breakdowns, drill-downs, correlations and time-series to root-cause the problem.For a high-traffic checkout process — two monitoring metrics a BI Analyst would display:1. Checkout Conversion Rate (sessions that complete purchase) — 5m/1h rolling, baseline + threshold alert when drop >10%. Display trend and cohort compare (desktop vs mobile).2. Payment Success Rate & Failure Rate by gateway — real-time percentage and transactions per minute, with alert on sudden spike in failures or latency over threshold.Two diagnostic dashboards/queries to investigate an alert:1. Transaction-level Failure Drilldown dashboard: - Filters: time window, payment gateway, country, device, browser, error code. - Visuals: stacked bar of error codes over time, table of recent failed transaction IDs with payment provider error messages, geographic heatmap. - Purpose: identify whether failures cluster by gateway, geo, or client.2. Latency and Dependency Trace query/dashboard: - Time-series of step-level latency (checkout page load, cart calculation, payment authorization, order confirmation) and upstream service error rates. - Query to join logs/metrics: SELECT step, avg(latency), p99, count_errors FROM checkout_steps WHERE ts BETWEEN ... GROUP BY step. - Purpose: pinpoint which step introduced latency or errors (e.g., third-party auth slowdown).These provide quick alerts plus targeted, actionable investigation paths for ops and product owners.
HardTechnical
18 practiced
Design a governance process to manage metric changes such as definition updates, renames, and backfills that minimizes user disruption. Describe the steps for proposal, review, testing, communication, a deprecation timeline, automated metadata updates in dashboards, and a rollback mechanism if downstream issues arise.
Sample Answer
Situation: Metric changes (definition updates, renames, backfills) routinely break dashboards and trust. We need a governance process that prevents disruption while allowing needed evolution.Proposal (intake)- Submit a Metric Change Request (MCR) via a template (metric name, owner, business rationale, change type, affected datasets, proposed SQL/logic, start/end times for backfill, urgency).- Tag stakeholders: metric owner, BI maintainer, data engineering, product/finance consumer groups.Review & approval- Triage by BI steward within 48 hours for completeness.- Technical review by Data Engineering (schema/backfill feasibility) and BI lead (downstream impact).- Business review with metric consumers for semantic approval.- Approvals recorded in ticketing system (Jira) with required sign-offs.Testing- Implement change in a sandbox environment; run automated regression tests comparing historical values and distribution checks.- Create a “canary” dashboard view that surfaces both old and new metric values and row-level provenance for sampling.- Add unit tests for metric SQL and integration tests for dashboards - fail the CI pipeline if thresholds exceeded.Communication- Publish change summary, impact matrix, and rollout plan to mailing list + workspace channel. Include migration timeline, examples, query snippets to update SQL, and a migration FAQ.- Host a short walkthrough for major consumers.Deprecation timeline- Announcement (T0): MCR approved.- T+2 weeks: Opt-in period where dashboards can switch to new metric via metadata toggle.- T+8 weeks: Read-only deprecation: old metric available but flagged deprecated in catalog.- T+12 weeks: Old metric removed; final backfill completed. Emergency extensions require executive sign-off.Automated metadata updates- Maintain a metrics catalog (e.g., dbt docs, Data Catalog) as source of truth with API access.- BI tool integrations: use the catalog API to push metadata updates and push deprecation flags and suggested replacements to dashboards (Looker/Power BI metadata connectors).- Provide a compatibility layer: alias mapping so old metric identifiers map to new ones until final removal.Rollback mechanism- Version metrics in code (git) and use feature flags/toggles for rollout.- If downstream anomalies occur, toggle back to previous metric version immediately; CI/CD rollback re-applies prior SQL and triggers backfill if needed.- Post-rollback: incident brief, root-cause analysis, adjust tests or timeline before retry.Governance operating model- Monthly metrics council (BI, Data Eng, key consumers) to review changes, prioritize backlog, and maintain SLAs.- KPIs for the process: time-to-approve, number of broken dashboards post-change, consumer satisfaction.This process balances agility and safety through clear ownership, automated testing, staged rollout, proactive communication, and fast rollback capability.
MediumTechnical
24 practiced
Design a dashboard panel that compares acquisition channels to targets while weighting channels by revenue contribution. Explain how you'd compute a 'weighted target attainment' metric, which visualizations best convey per-channel attainment and contribution, how to normalize channels with different scales, and how to treat newly launched channels with sparse data.
Sample Answer
Compute weighted target attainment- Define for each channel i: Revenue_i (actual), Target_i (revenue target), ContributionWeight_i = Revenue_i / TotalRevenue (period).- Weighted Target Attainment (WTA) = sum_i (ContributionWeight_i * Attainment_i) where Attainment_i = Revenue_i / Target_i.- Equivalent formula: WTA = (1 / TotalRevenue) * sum_i (Revenue_i * (Revenue_i / Target_i)).- Interpretation: channels that drive more dollars influence the overall score more; WTA ∈ [0, ∞), often expressed as %.Visualizations to convey per-channel attainment and contribution- Primary: Dual-axis bar+line per channel: bar = Revenue contribution (absolute $ or % of total), line = Attainment (%) on secondary axis. This shows both scale and target success.- Alternative: Scatter plot (x = Attainment %, y = ContributionWeight or Revenue), bubble size = Target size or CPA; adds quick identification of high-revenue, low-attainment outliers.- Stacked bar (by channel) showing achieved vs remaining to target gives intuitive progress toward targets.- KPI tiles: overall WTA, TotalRevenue vs TotalTarget, top 3 over/under-performing channels.- Use color: green/orange/red based on attainment thresholds.Normalization across different scales- Use percent-based metrics (Attainment = Revenue/Target) and contribution share (Revenue/TotalRevenue) so channels with small absolute volumes are comparable.- For trends, use indexed series (index = value / value_at_period0 *100) so growth rates are comparable.- Use log scale only when distributions are highly skewed and absolute comparisons are needed.Treating newly launched channels with sparse data- Flag channels with insufficient history (e.g., < N conversions or < X days) and show confidence/volume annotation.- Use shrinkage / Bayesian prior: blend observed Attainment_i with a baseline prior (e.g., company average attainment) weighted by sample size: AdjAttainment = (n * Attainment_i + k * Prior) / (n + k).- Show separate “early-stage” cohort visuals and avoid over-weighting them in WTA: either use minimum-revenue threshold for ContributionWeight or cap their weight until they pass a volume threshold.- Provide dotted forecast lines with uncertainty bands (bootstrap or simple growth projection) rather than firm conclusions.Implementation notes (BI tool tips)- Precompute Revenue, Target, Attainment, ContributionWeight in ETL or a semantic layer for performance.- Use parameterized thresholds and filters for timeframe and cohorting; expose hover tooltips with raw counts, sample size, and confidence.- Add annotations for channel launches and attribution model used.This approach balances dollar impact with target performance while making statistical uncertainty explicit for new channels.
MediumSystem Design
21 practiced
Design a dashboard to report A/B test results for product managers and non-statistical stakeholders. Requirements: support multiple concurrent experiments, show primary metric, effect size, 95% confidence interval, p-value, sample size, test duration, uplift vs baseline and recommended action (stop/ship/continue). Describe layout, visual elements, how you'd annotate sequential monitoring warnings, and guards against misinterpretation.
Sample Answer
Requirements (clarify): support many concurrent experiments, surface primary metric + lift/effect size, 95% CI, p-value, n, duration, uplift vs baseline, and a recommended action (stop/ship/continue). Audience: PMs / non-stat stakeholders — clear, actionable, minimal statistical jargon.High-level layout:- Top bar: filters (product area, owner, status: running/complete/paused, date range).- Left column (Experiment list): compact cards for each concurrent experiment with name, status badge (running/paused/complete), start date, sample size, primary metric, quick verdict icon (stop/ship/continue).- Main panel (selected experiment detail): 1. Header: experiment name, hypothesis, owner, start/end dates, sample size per variant. 2. Key summary strip (prominent numbers): baseline value, variant value, absolute & relative effect (uplift %), 95% CI, p-value, test duration, power achieved. Each number has a tooltip with plain-language explanation. 3. Visuals: - Time-series of metric by cohort (baseline vs variants) with shaded 95% CI bands; allow smoothing toggle. - Bar chart of effect size with CI whiskers centered on uplift % (horizontal). - Cumulative sample size and conversion curves below. - Table of secondary metrics (sortable) with small effect+CI. 4. Decision card: automated recommendation engine shows recommended action with reasoning bullets (e.g., "p=0.02, uplift 3.2% (95% CI 1.0–5.4%); power 85% → Ship") and confidence level badge. 5. Notes & audit trail: experiment notes, sequential checks performed, analyst comments, and link to raw analysis notebook.Sequential monitoring and warnings:- Display a clear banner when sequential/interim looks were allowed: "Interim looks allowed — alpha spending: 0.03 used; adjusted p-values shown." Show both nominal and adjusted p-values and adjusted CI method (e.g., O’Brien-Fleming, Pocock) in the summary strip.- If early stopping threshold crossed, color the decision card and include timestamped alert + rationale, with link to underlying test of stopping rule.- Provide an "analysis assumptions" panel: randomization integrity checks, baseline balance p-values, sample ratio mismatch alert, and data freshness.Guards against misinterpretation:- Show both absolute and relative effects; never show p-value alone — pair it with effect size and CI.- Display plain-language explanations (one-liners) for p-value, CI, and what “statistical significance” means for business impact.- Add mandatory confirmation modal before acting on "stop" recommendations with checklist (minimum sample, power, SNR, no suspicious traffic spikes).- Provide experiment-level simulations: estimate expected business impact (revenue lift) given effect and CI ranges.- Use visual cues: green/yellow/red for practical significance thresholds (configurable), and disable “ship” if power < threshold or sample ratio mismatch detected.Data pipeline & implementation notes:- ETL job aggregates daily by variant, stores metrics in analytical DB (partitioned by experiment id).- Use Looker/Tableau/Power BI dashboards with embedded SQL models or in-tool derived tables for CI/p-value calculations, or precompute in scheduled analytics job to ensure reproducibility.- Versioned analysis code + automated tests for metric definitions; store audit logs of who ran what and when.- Access controls: PMs see summary; analysts see raw stats and adjustment details.This design balances clarity for PMs with rigorous statistical transparency for analysts, prevents misuse via clear warnings/adaptive p-value display, and supports multiple concurrent experiments with scalable filtering and drilldowns.
EasyTechnical
20 practiced
Define the concept of a 'North Star' metric for a product. As a Business Intelligence Analyst for a subscription-based SaaS company, outline a repeatable process to select the North Star. Include at least three candidate metrics (for example: MRR, 28-day active users, engaged-sessions-per-user), the selection criteria (measurable, leading vs lagging, aligned to customer value, ownership), and how you would validate the choice with stakeholders and data.
Sample Answer
A North Star metric is a single, clear KPI that captures the core value your product delivers to customers and that correlates with long-term business health. As a BI Analyst for a subscription SaaS, I’d use a repeatable process to select it:Process1. Align: Gather stakeholders (PM, Sales, CS, Exec) and agree success outcomes (growth, retention, expansion).2. Inventory: List candidate metrics and data sources.3. Filter by criteria (below) to shortlist 2–3 candidates.4. Quantitatively validate: run correlation and causal/lag analysis vs business outcomes (MRR growth, churn, LTV) over historical data.5. Qualitatively validate: user research and stakeholder workshops to confirm metric reflects customer value.6. Decide & govern: nominate owner, define formula/segment rules, dashboard, and review cadence (monthly).7. Re-evaluate quarterly or on major product changes.Candidate metrics- MRR (Monthly Recurring Revenue) — business-facing, lagging, tied to revenue.- 28‑day Active Users (DAU28) — engagement proxy, leading for retention.- Engaged‑sessions‑per‑user — measures depth of value delivery, leading indicator.Selection criteria- Measurable & reliable (clear definition, instrumented, low noise)- Leading vs lagging (prefer leading predictors of retention/revenue)- Aligned to customer value (reflects core job-to-be-done)- Ownership & actionability (team can move it and has levers)- Sensitive but stable (detectable changes without excessive volatility)Validation with stakeholders & data- Correlation: compute Pearson/Spearman with MRR growth and churn at cohort level.- Lag analysis: show whether changes in candidate precede revenue or retention changes.- A/B / experiment signals: check if product experiments that moved the metric also moved outcomes.- Present findings in a one-pager plus dashboard, run a pilot (30–90 days) where teams use the metric to prioritize work, and measure impact. If validation fails, iterate using the governance cadence.
Unlock Full Question Bank
Get access to hundreds of Dashboard Structure & Actionability interview questions and detailed answers.
