Situation: Our Sales Ops team wasn't using a newly launched Power BI executive dashboard; weekly active users were 18% of target and average time-to-insight (time from opening to answering a key question) was ~22 minutes.
Task & Hypothesis: I hypothesized that low adoption stemmed from two issues — slow initial load/performance and unclear entry points to answer top questions. Improving load speed and adding guided entry points would increase active use and reduce time-to-insight.
Experiment design:
- A/B test over 4 weeks with two cohorts of Sales managers (randomized, ~120 users each).
- Variant A (performance): optimized queries, incremental refresh, removed heavy visuals to cut initial load.
- Variant B (guidance): retained original performance but added a "Start Here" landing page with 3 pre-set bookmarks/filters and short tooltips for common questions.
- Control: original dashboard.
Metrics:
- Primary: weekly active users (WAU) and median time-to-insight for 3 predefined questions (revenue by region, pipeline risk, closed-won trend).
- Secondary: session length, number of interactions (filters/bookmarks), user satisfaction (1–5 in pop-up survey).
Results:
- Performance-only cohort: WAU +28%, median time-to-insight -15% (from 22 to 18.7 minutes).
- Guidance-only cohort: WAU +42%, median time-to-insight -40% (to 13.2 minutes), higher interactions and satisfaction (+0.8).
- Control showed no significant change.
Action & Integration:
- Rolled out the guidance-first approach company-wide, combined with targeted query optimizations to preserve gains.
- Instituted a dashboard design pattern: landing page with 3 task-focused bookmarks, concise tooltips, and default filters aligned to user role.
- Added instrumentation to track time-to-insight and a quarterly A/B cadence for future UX changes.
Learning: Users value clear, task-oriented entry points more than marginal performance gains. Combining lightweight performance tuning with guided workflows yields the best adoption and faster decision-making.
