Exploratory Data Analysis and Data Quality Questions

Direct answer

Before anything else, get the shape of the problem: how many rows and columns, what each column's type actually is (not what it claims to be), how much is missing, and whether the target or key business columns look plausible. That first pass tells you whether you can trust anything else you compute.

A repeatable process

1. Load and inspect the skeleton. Row count, column count, dtypes, memory footprint, and the first/last few rows. Check that dtypes match what the column names imply (a "date" column stored as a string is the most common surprise).
2. Quantify missingness and duplication. Null rate per column, exact and near-duplicate row counts. A column that's 60% null changes everything downstream.
3. Univariate pass. For numeric columns: min, max, mean, median, standard deviation, and a histogram or boxplot. For categorical columns: cardinality and the top values by frequency. This is where you catch impossible values (negative ages, dates in the future) and near-constant columns.
4. Bivariate pass. Correlations among numeric features, and cross-tabs or grouped summaries for categorical-versus-target relationships. Look for anything suspiciously strong, since a feature that predicts the target almost perfectly is often leakage rather than a genuine signal.
5. Sanity-check the story. Do the aggregate numbers match what you'd expect from domain knowledge? If revenue should be roughly flat month over month and it triples, that's worth chasing before you build anything on top of it.
6. Write it down. A short profile (what you checked, what you found, what's still unresolved) is the artifact that lets someone else pick up where you left off, and it's the difference between EDA that informs a decision and EDA that lives only in your head.

For non-tabular data (text, images, audio), the same shape applies at a different unit: token-length or file-size distributions and corrupt-file checks stand in for the numeric-column checks above.

Worked example

Say you're handed a 50,000-row CSV of orders. In the first pass you find: order_date is stored as text with three different formats, 8% of rows are missing customer_id, and amount has a maximum value 40 times the 99th percentile. None of that tells you the final answer yet, but it tells you exactly what to chase next: normalize the date formats, decide whether the missing customer_id rows are guests or a logging bug, and pull the handful of extreme amount rows to see whether they're real (bulk orders) or data-entry errors (a misplaced decimal point).

Trade-offs and pitfalls

The most common failure is treating EDA as a checklist you complete once and never revisit: as you clean and re-derive columns, re-run the same univariate and bivariate checks on the new versions, because a fix to one column can introduce a new artifact in another (for example, dropping "impossible" rows can quietly shift a category's proportions). The other common failure is skipping straight to modeling-relevant checks (correlations, class balance) before confirming the basics (types, missingness, duplicates), which means you build early conclusions on data you haven't actually verified yet.

Direct answer

For a single numeric column, start with a histogram (or a KDE, a kernel density estimate) for shape, a boxplot for a compact five-number summary and outlier flags, and the standard descriptive statistics (mean, median, standard deviation, min, max). For a single categorical column, start with a bar chart of value counts and the distinct-value count itself, since cardinality changes what "summarizing" even means.

What each reveals that the others don't

A histogram shows the full shape: modality, skew, and gaps that a single summary number can't convey. A boxplot compresses that shape into quartiles and flags candidate outliers, which is more compact but hides multimodality (two distinct humps can produce a perfectly ordinary-looking boxplot). Descriptive statistics give you exact numbers to reference or compare across columns, but on their own they can be actively misleading on a skewed or multimodal distribution (the classic case: a dataset can have identical mean, variance, and correlation as another while looking completely different when plotted). For categorical data, a bar chart of value counts shows both the shape (is one category dominant, or is it fairly even) and immediately reveals cardinality problems (hundreds of near-unique values where you expected a handful of categories).

Worked example

For a price column: the histogram shows a long right tail with most values clustered under $50 and a scattering up past $500; the boxplot confirms a compact box around $20 to $45 with several points flagged above the upper fence; and the mean ($68) sitting noticeably above the median ($38) confirms the skew numerically. For a payment_method column: value counts show credit_card at 71%, paypal at 24%, and four other methods splitting the remaining 5%, with a distinct count of 6, telling you immediately this is a genuinely low-cardinality categorical column you can treat straightforwardly.

Trade-offs and pitfalls

Don't stop at descriptive statistics alone for a column you haven't looked at before: they're fast to compute but can hide exactly the shape information (multimodality, heavy tails, a spike of a single repeated value) that a chart would reveal in seconds. For categorical columns, always check the distinct count before choosing a visualization, since a bar chart of value counts is useless once cardinality climbs into the hundreds or thousands.

Direct answer

The z-score method flags a point as an outlier when it's more than roughly 3 standard deviations from the mean; it assumes the data is approximately normal and is itself sensitive to the very outliers it's trying to detect, since both the mean and standard deviation shift when an extreme value is present. The IQR (interquartile range, Tukey's fences) method flags anything outside $[Q_1 - 1.5 \times IQR,\ Q_3 + 1.5 \times IQR]$; it doesn't assume normality and is more robust because it's based on the middle 50% of the data, which extreme values barely touch. A robust alternative, the median absolute deviation (MAD), replaces both the center and the spread with median-based versions, making it even less sensitive to the outliers you're trying to find.

How they behave differently

Because the z-score uses the mean and standard deviation, a single very extreme value can inflate the standard deviation enough to hide itself and other, smaller outliers, a known "masking" effect. IQR and MAD-based approaches don't have this problem, because the median and quartiles barely move even when a handful of extreme values are added. On a skewed distribution, z-scores are especially unreliable, since normality is baked into the interpretation of "3 standard deviations."

Worked example

Take the values 21, 23, 24, 22, 25, 23, 22, 24, 21, and 95. $Q_1 = 22$, $Q_3 = 24$, so $IQR = 2$ and the fences are $[19, 27]$: the IQR method correctly flags only 95. The z-scores, computed from the mean and standard deviation of all ten points including 95, come out to roughly $-0.4$ to $-0.3$ for the normal-looking values and only about $2.8$ for 95 itself, which falls just short of a typical $|z| > 3$ threshold: the outlier's own presence inflated the standard deviation enough to mask itself. A MAD-based robust z-score, by contrast, gives 95 a value around 48, unmistakably flagged, while every other point stays under 1.5.

Trade-offs and pitfalls

IQR assumes a reasonably unimodal distribution; on a distribution with multiple genuine peaks it can flag a whole second mode as "outliers" when it's really a distinct subpopulation worth splitting out rather than removing. None of these methods tell you WHY a point is extreme, only THAT it is: the judgment about whether it's an error or a genuine extreme value always requires looking at the actual case.

Direct answer

A histogram is the default first look at a numeric distribution's overall shape; a boxplot compresses that into a five-number summary plus outlier flags and is best for quickly comparing many groups side by side; a violin plot adds a smoothed density outline around the boxplot, recovering the shape information a boxplot alone hides, at the cost of needing more data to look stable; a KDE (kernel density estimate) shows a smooth version of the histogram's shape without being sensitive to bin-width choices, but can smooth away genuinely sharp features like a hard floor or ceiling in the data.

What each emphasizes

Histograms show gaps, multiple modes, and the exact shape, but the picture changes with bin width, sometimes substantially. Boxplots are compact and comparable across many categories at once (ten boxplots side by side is readable; ten histograms side by side usually isn't), but they can make a genuinely bimodal distribution look like an ordinary unimodal one with some spread. Violin plots recover that bimodality by showing the density shape directly, which matters when you specifically suspect two or more distinct subpopulations hiding inside one column. KDEs are the smoothest and most visually appealing for a single distribution, but the smoothing bandwidth is a hidden parameter that can understate or overstate how sharp a real feature is.

Worked example

Exploring session_duration for two user segments: a boxplot side by side for "free" and "paid" users quickly shows the paid segment has a higher median and a longer whisker. But a violin plot for the free segment on its own reveals two distinct humps, a very short-session group and a long-session group, that the boxplot's single box completely hid. That distinction (two subpopulations rather than one spread-out population) is exactly the kind of finding that changes what you investigate next.

Trade-offs and pitfalls

For a technical audience comparing many groups at once, boxplots (or violin plots if bimodality is suspected) scale better than a wall of histograms. For a single distribution you're exploring in depth yourself, a histogram alongside a KDE is usually the most informative combination. For a non-technical audience, a single clean histogram or a simple boxplot is usually more legible than a violin plot, whose density-shape encoding takes explanation.

Direct answer

Data leakage is when information that wouldn't actually be available at decision time sneaks into your analysis or dashboard, making a result look better or more certain than it really is. Two common flavors: temporal leakage, where a field reflects something that happened AFTER the point in time you're supposedly making the decision at, and schema-derived leakage, where a field is mechanically DERIVED from the very outcome you're trying to explain or predict.

Why it misleads whoever consumes the result

Leakage inflates apparent performance or apparent insight in a way that evaporates the moment the analysis meets reality, because the leaking information simply won't exist yet when someone tries to act on the finding. A dashboard or model built on leaked data looks great in testing and then quietly fails (or never actually helps) in production, since the shortcut it learned isn't available going forward.

Worked example

Temporal leakage: a churn dashboard includes "total number of support tickets filed," computed using ALL tickets a customer ever filed, including ones filed after they already churned (often a spike of complaints right around cancellation). The dashboard will show "high ticket count strongly predicts churn," but that's a symptom recorded after the fact, not a usable early-warning signal. Schema-derived leakage: a "days until first purchase" field is included as a predictor of "will this customer make a purchase," but the field is undefined (or null) for anyone who never purchases, meaning the field mechanically encodes the answer.

Trade-offs and pitfalls

The fix is always to ask "would this exact value have been available and true at the moment I'd need to act on it," not just "does this field correlate with the outcome." A field can be perfectly legitimate for a retrospective report ("how many tickets did churned customers file, in total") while being leakage if the same field is used to predict or dashboard something forward-looking.