Direct answer: Treat qualitative and quantitative evidence as answering different questions, not as competing sources of the same fact: quantitative results tell you what changed and by how much; qualitative evidence tells you why, and whether the "why" is one you actually want. When they disagree, investigate the disagreement as a signal rather than picking whichever source is more convenient.
Structured elaboration
- Use quantitative results to establish the size and direction of an effect with statistical rigor; use qualitative signals (support tickets, user interviews, in-app feedback, session recordings) to explain the mechanism behind that effect and to surface things the quantitative metrics were never designed to catch.
- When the two agree (a positive quantitative result accompanied by positive qualitative sentiment), that convergence is strong evidence the feature is genuinely working for the reason you think it is.
- When they disagree (a positive quantitative result but negative or confused qualitative sentiment, or vice versa), do not average them into a vague "mixed" verdict; instead investigate which specific mechanism explains the gap. A common real pattern: the quantitative metric improved because the feature makes an action easier, but qualitative feedback reveals users feel manipulated or confused while doing it, meaning the metric captured a behavior change without capturing whether that change is something you actually want to have caused.
- Weigh the two by scope and reliability, not by which is more recent or more convenient: a quantitative result from a well-powered experiment on the actual metric you care about should not be casually overridden by a handful of vivid but unrepresentative qualitative complaints, but a qualitative signal that surfaces a genuine mechanism the quantitative metric cannot see (a dark-pattern-like feeling, a trust concern) should not be dismissed just because it lacks a p-value.
Worked example: A subscription-cancellation flow redesign shows a statistically significant 15% reduction in completed cancellations (a quantitative win by the metric the team set out to move). Qualitative signals, though, show a spike in support tickets and negative app-store reviews specifically describing the cancellation flow as confusing or intentionally obstructive. Investigating the mechanism reveals the reduction is partly coming from users who wanted to cancel giving up in frustration rather than being retained through genuine reconsideration, a distinction the quantitative metric alone could not make. The team concludes the quantitative win is real but achieved partly through an unwanted mechanism, and revises the flow to keep the improvements that reduce accidental/uninformed cancellations while removing the friction that frustrated users who genuinely wanted to leave.
Trade-offs and pitfalls: The most common failure is treating a clean quantitative result as the whole story and never checking qualitative signals at all, which misses exactly the "right metric, wrong mechanism" case above. The opposite failure is letting a small number of vivid, negative qualitative anecdotes override a well-powered quantitative result without first checking whether those anecdotes represent a real, sizeable pattern or a loud but unrepresentative minority.
