Exploratory Data Analysis and Data Quality Questions

Direct answer

Confirm suspected drift with a two-pronged check: a Kolmogorov-Smirnov (KS) test for a statistically significant difference in distribution shape, and a Population Stability Index (PSI) for a practically-scaled measure of how much the distribution has shifted, alongside a simple side-by-side plot so you can see what's actually changed, not just a p-value.

Reading the two statistics together

A KS test answers "is this difference statistically distinguishable from noise," which on a large enough sample will often come back significant even for a genuinely tiny, practically irrelevant shift. PSI answers a different, more practical question: how large is the shift, on a scale where common industry rules of thumb treat a value under about 0.1 as no real concern, 0.1 to 0.2 as a moderate shift worth watching, and above 0.2 as a substantial shift worth escalating. Using both together avoids two separate failure modes: trusting a KS test alone (which can flag statistically-real but practically-irrelevant differences at scale) or trusting PSI alone without confirming the difference isn't just sampling noise in a smaller dataset.

Worked example

Comparing 5,000 training-time observations against 5,000 recent production observations for a feature whose mean shifted from about 50 to about 54 and whose spread widened somewhat: the KS test comes back highly significant (a KS statistic around 0.16, with a p-value far below any conventional threshold), confirming the distributions are genuinely different, not just noise. The PSI comes out to about 0.18, landing right at the boundary between "moderate, worth watching" and "substantial, worth escalating" by common conventions, which is itself useful information: this is a real, meaningful shift, but not yet in the most severe range, informing how urgently to escalate versus continue monitoring.

Trade-offs and pitfalls

This EDA-level confirmation is the analyst's first pass, not the final word: once you've confirmed a real and meaningful shift, escalate to whoever owns the production monitoring and retraining decision rather than deciding unilaterally whether to retrain, since that decision usually needs to weigh retraining cost and risk alongside the size of the shift you've just confirmed.

Direct answer

Treat this as an evidence-gathering exercise for each flagged point, not a blanket rule: check whether the value is physically plausible, whether it correlates with a known legitimate cause (a VIP account, a bulk order, a specific promotion), and whether removing or capping it would change a downstream decision. Only remove a point once you have a specific reason to distrust it, not just because it's far from the rest.

A decision process

1. Check plausibility first. Is the value even possible given the business (a negative quantity, an age of 300)? If so, it's very likely an error, not a legitimate extreme value.
2. Look for corroborating signal. Does the account have a history of similarly large orders? Is there a promotion or bulk-purchase flag active that explains it? A genuine VIP purchase usually has supporting evidence elsewhere in the row or the customer's history; a data-entry error usually doesn't.
3. Consider the downstream use. If you're reporting a sum (total revenue), an error needs fixing but a legitimate large value should stay in. If you're reporting a "typical" value (median order size), a genuine extreme value can often stay in the data untouched, since the median won't be swayed by it anyway.
4. Choose the least destructive fix that solves the actual problem. Removing a row throws away information permanently; winsorizing (capping at a threshold) keeps the row but limits its influence; a log transform reduces the influence of all large values proportionally rather than singling one out. Reach for the most surgical option that addresses what's actually wrong.

Worked example

Say a $50,000 order shows up in a dataset where the 99th percentile is $1,200. Checking the account: it belongs to a corporate customer with three other orders over $30,000 in the past year, and the order line items list 500 units of the same product, consistent with a bulk purchase. That's a legitimate outlier, kept as-is for a sum, and probably fine to leave untouched for a median too since it won't move it much. Contrast that with a second $50,000 order from a first-time individual customer buying a single unit of a $12 product: no supporting evidence, and a quick check shows the quantity field was likely fat-fingered as 4,000 instead of 4. That one gets corrected (or excluded and flagged), not because it's large, but because there's a specific, checkable reason to distrust it.

Trade-offs and pitfalls

The temptation to "just remove outliers because they skew the numbers" is a real trap: skewing the numbers is sometimes the entire point, if the numbers are genuinely skewed. Document whatever decision you make and why, since a silent removal is invisible to anyone auditing the analysis later, and a wrong call here (removing real revenue, or keeping a fat-fingered order) directly changes a business number.

Direct answer

An unexpected insight from EDA is only worth pivoting a project on once you've ruled out the mundane explanations first: a data artifact, a small sample, or a seasonal effect masquerading as a genuine pattern. The STAR shape here is: notice something surprising in the data, verify it's real before trusting it, and only then communicate the pivot with the evidence that convinced you, not just the surprising headline.

Verifying before pivoting

The instinct when something surprising shows up is excitement, but the discipline that matters is treating a surprising EDA finding exactly like any other hypothesis: check it against a different slice of the data, a different time window, or a different measurement of the same underlying idea, before restructuring a project around it. A finding that survives that kind of cross-check is worth acting on; one that evaporates under a second look was a data artifact, not an insight.

Worked example

While exploring engagement data ahead of a planned feature launch, a pattern emerges: a specific existing feature, thought to be a minor one, correlates with dramatically higher long-term retention. Before recommending the team pivot toward investing in that feature, the check: does this hold across multiple cohorts and time windows, or just the one you happened to look at first; is it explained by a confound (maybe the kind of user who uses that feature was already more engaged for unrelated reasons); and does the effect size hold up once you account for that confound. In this case, the pattern held across three separate cohorts even after controlling for prior engagement level, which was enough evidence to bring to the team, along with the specific checks that had been run, not just the headline number.

Trade-offs and pitfalls

Documenting and communicating the pivot means being explicit about what you checked and what you didn't, since a stakeholder acting on this finding deserves to know its actual evidentiary weight (three cohorts and a confound check) rather than hearing it presented with more confidence than the underlying verification actually supports.

Direct answer

Mahalanobis distance measures how far a point is from the center of a multivariate distribution, accounting for both the SCALE of each variable and the CORRELATION between them, unlike a per-column z-score, which only ever looks at one variable in isolation. A point can be entirely unremarkable on every individual axis and still be a genuine multivariate outlier if it breaks the correlation structure the rest of the data follows.

Why per-column checks miss this

A per-column z-score asks "is this value far from the mean of ITS OWN column," which says nothing about whether the COMBINATION of values across columns is plausible given how those columns normally relate to each other. Mahalanobis distance directly incorporates the covariance structure, so a point that lands in a "hole" where the correlated variables normally never combine gets flagged, even if neither value individually looks unusual.

Worked example

Simulating 200 points from two strongly positively correlated variables (correlation about 0.8), a new point at (2, -2) has per-axis z-scores of roughly 2.0 and -1.9 respectively, both comfortably under the usual |z| > 3 threshold that a univariate check would flag on. But its Mahalanobis distance comes out to about 6.5, standing out sharply, because this point combines a high value on one variable with a low value on the other, exactly the combination that never happens given how strongly and positively these two variables normally move together. A univariate scan of each column separately would let this point through entirely.

Trade-offs and pitfalls

Mahalanobis distance itself relies on the mean and covariance, which are sensitive to outliers in exactly the way a plain z-score is, so it's common (and often necessary on real data) to use a robust covariance estimator instead of the plain sample covariance when outliers might already be present in the data used to fit it. Model-based approaches like Isolation Forest or Local Outlier Factor are worth reaching for when the relationships among variables are more complex than a single global covariance structure can capture (for example, when "normal" combinations differ meaningfully across subgroups or clusters).

Direct answer

Run exploratory analysis to propose a concrete, measurable definition rather than accepting the vague request as-is: inspect what data sources plausibly relate to "satisfaction" (support tickets, survey responses, churn, product usage), look at their distributions and how they relate to each other, and sanity-check any candidate definition against cases where you already have strong intuition about whether a customer was satisfied or not.

Building toward a defensible definition

Start by asking what decisions this score is meant to inform, since that shapes what "satisfaction" should even mean operationally. Then explore candidate inputs (survey scores if they exist, support-ticket sentiment or volume, usage trends, churn or renewal behavior) individually: their distributions, how much missing data each has, and whether they move together or tell conflicting stories. Before publishing, sanity-check the proposed score against a handful of accounts where you already have strong outside knowledge of whether the customer is happy or not, since a definition that contradicts obvious cases needs revisiting before it goes on a dashboard.

Worked example

Exploring three candidate signals: an NPS-style survey (Net Promoter Score, only 15% response rate, but where it exists it's directly interpretable), support-ticket volume normalized by account size, and month-over-month usage trend. The survey alone is too sparse to use on its own; ticket volume alone conflates a heavy but happy power-user with a genuinely frustrated one. Combining usage trend with ticket volume (weighted more heavily since it covers 100% of accounts) and validating the resulting score against ten accounts the account management team already has strong opinions about shows reasonable agreement in eight of ten cases; the two disagreements turn out to be accounts with unusually low usage for legitimate seasonal reasons, prompting a seasonal adjustment before the definition ships.

Trade-offs and pitfalls

A metric built this way should ship with its definition and known limitations attached, not just a number, since "customer satisfaction score" invites people to treat it as an objective fact rather than the constructed proxy it actually is.