Exploratory Data Analysis and Data Quality Questions

Direct answer

Don't assume causality from timing alone: a release-then-drop pattern is consistent with the release causing it, but also with an unrelated event that happened to coincide. Rule out the alternatives systematically: check instrumentation first (did anything in the release change how the conversion event is even logged), then segment by device, geography, and user cohort to see whether the drop is universal or concentrated, then compare against the same period from a prior cycle to separate a genuine drop from ordinary seasonal variation.

Building the investigative plan

Start with the cheapest, highest-value check: verify the metric itself is being measured the same way before and after the release, since a broken or changed tracking pixel produces a "drop" that has nothing to do with user behavior at all. If instrumentation checks out, segment the conversion drop by device, platform version, geography, and acquisition channel; a universal drop across every segment points toward something release-wide, while a drop concentrated in one segment (say, only users on the release's new platform version) points toward a specific defect rather than the release's core change. A pre/post comparison against the same calendar period from a prior cycle (last month, or the same weekday last week) helps separate a genuine step-change from normal week-to-week noise. Only once instrumentation, segmentation, and normal-variation checks come back clean does a genuinely release-caused drop become the most plausible remaining explanation, and even then, a proper causal claim usually still wants a controlled comparison (a holdback group, or a staged rollout) rather than a before/after look alone.

Worked example

A 20% drop in conversion follows a release. Instrumentation check: the conversion event definition didn't change, ruling out a tracking artifact. Segmentation: the drop is heavily concentrated in mobile users on the newest app version, while desktop and older app versions show only a normal weekly dip. That narrows the investigation from "the whole release" to "something specific to the new mobile build," which turns out, on further digging, to be a checkout button rendering below the fold on smaller screens. The release DID cause the drop, but the segmentation step was what made that conclusion trustworthy rather than a coincidence-driven guess.

Trade-offs and pitfalls

Resist the pressure to declare a cause the moment timing lines up: the segmentation and instrumentation checks take real time, but skipping them is how a team ends up "fixing" the wrong thing while the actual defect (or a completely unrelated cause) goes unaddressed.

Direct answer

Before analyzing a new dataset, write down assumptions about label or field quality, whether the data is stationary over the time window you're using, what you expect key distributions to look like, how much missingness you can tolerate, and whether a deduplication key or timestamp normalization actually holds. For each one, pair it with a concrete, checkable test, since an assumption you never verify is just a guess wearing a more confident outfit.

Why this matters as its own step

Every downstream number implicitly rests on a stack of unstated assumptions (that a user ID uniquely identifies a person, that timestamps are all in the same timezone, that a product code never gets reused). Surfacing them explicitly, before you start computing anything, turns invisible risk into a short checklist you can actually verify, and it means a wrong assumption gets caught in minutes instead of discovered months later when a number looks strange.

Worked example

Before computing a "monthly active users" metric: assumption one, a user_id uniquely identifies one person (checked by looking for the same email or device fingerprint under multiple IDs); assumption two, event timestamps are normalized to a single timezone (checked by confirming the schema and spot-checking a few rows against known local activity patterns); assumption three, the definition of "active" hasn't silently changed partway through the data's history (checked by confirming the event name and its logic haven't been redefined). On a real dataset, that second check once revealed timestamps were stored in UTC for one data source and local time for another, which would have silently shifted a meaningful share of "daily" activity into the wrong day.

Trade-offs and pitfalls

Writing down assumptions is only valuable if you actually test them: a list of assumptions that never gets checked is no better than not writing them down at all, and it's tempting to stop at the list because it feels like due diligence even when the actual verification step gets skipped.

Direct answer

On a small sample table with age missing for one user and country missing for another, look at what else is known about each row before concluding anything: if the missingness seems tied to specific OTHER observed fields (say, both missing values belong to users with very short last_active_days, suggesting incomplete signups rather than a random gap), that points toward missing at random (MAR) rather than missing completely at random (MCAR); if there's no such pattern at all, MCAR is the more plausible working assumption, though a sample this small can't rule out much with confidence either way.

Reading the indicators in a small sample

The judgment here is necessarily provisional: with only a handful of missing values, you're looking for suggestive patterns, not statistical proof. Check whether the missing values cluster on particular values of other observed columns (a specific signup channel, a specific date range, a specific last_active_days value) rather than spreading randomly across the table; a cluster is a hint worth chasing further, while a scattered, unpatterned handful of missing values in an otherwise unremarkable small sample is consistent with MCAR, though genuinely inconclusive at this size.

Worked example

In a four-row sample: age is missing for the user with last_active_days of 5, and country is missing for the user with last_active_days of 30. If this were the ENTIRE evidence, it's genuinely ambiguous at n=4, but the diagnostics worth running on a larger version of the same table would be: does age missingness correlate with recency of signup or activity level across many more rows (pointing toward MAR via activity level as the observed explanation), or does it look scattered across every level of activity (more consistent with MCAR)? The two concrete diagnostics to propose here are a missingness-versus-last_active_days crosstab and a simple bar chart of missingness rate by signup cohort, both cheap to run once you have more than four rows to work with.

Trade-offs and pitfalls

Resist over-interpreting a genuinely small sample: the honest answer at n=4 is "here's a hypothesis and here's how I'd test it on the full dataset," not a confident MCAR-versus-MAR conclusion drawn from four rows.

Direct answer

Organize an EDA project so raw data, processed/derived data, notebooks, and outputs (charts, summary tables) each live in their own clearly separated folder, with notebooks kept thin (orchestration and narrative) while the actual logic lives in importable scripts or functions. Version both the code and, where practical, a snapshot or hash of the dataset you ran against, so someone else (including future you) can regenerate exactly what you found.

What makes it actually reproducible

The core discipline is separating "what changes every run" (raw inputs, generated charts) from "what defines the analysis" (code, parameters). Notebooks are great for narrative and iteration but terrible as the single source of truth for logic that needs to be re-run reliably, since cell execution order can silently diverge from the file's visual top-to-bottom order. Moving real logic into scripts or a small package, and having the notebook just call into it, means the analysis can be re-run from a clean state and get the same answer. Recording a data snapshot (a hash, a date-stamped extract, or a fixed query with a pinned date filter) closes the other common gap: source data changing under you between when you ran the analysis and when someone tries to reproduce it.

Worked example

A concrete layout: data/raw/ (untouched extracts, never edited by hand), data/processed/ (derived tables, each generated by a named script so you know how they were built), notebooks/ (numbered, narrative-focused, each one calling into shared code rather than redefining logic inline), src/ (the actual transformation and profiling functions, unit-testable and importable), and outputs/ (charts and summary tables, each named with the date and the script that generated it). A lightweight CI check that re-runs the core profiling script on a small fixture dataset catches silent breakage before it becomes someone's afternoon mystery.

Trade-offs and pitfalls

Don't over-engineer this for a one-off, half-day exploratory look, the ceremony of scripts, packages, and CI is worth it once an analysis is going to be revisited, shared, or built upon, not for genuinely throwaway exploration. The most common failure mode is a notebook that "worked" once, in an execution order nobody wrote down, and can't be reliably re-run six months later when someone needs to check the numbers again.

Direct answer

Treat this as an evidence-gathering exercise for each flagged point, not a blanket rule: check whether the value is physically plausible, whether it correlates with a known legitimate cause (a VIP account, a bulk order, a specific promotion), and whether removing or capping it would change a downstream decision. Only remove a point once you have a specific reason to distrust it, not just because it's far from the rest.

A decision process

1. Check plausibility first. Is the value even possible given the business (a negative quantity, an age of 300)? If so, it's very likely an error, not a legitimate extreme value.
2. Look for corroborating signal. Does the account have a history of similarly large orders? Is there a promotion or bulk-purchase flag active that explains it? A genuine VIP purchase usually has supporting evidence elsewhere in the row or the customer's history; a data-entry error usually doesn't.
3. Consider the downstream use. If you're reporting a sum (total revenue), an error needs fixing but a legitimate large value should stay in. If you're reporting a "typical" value (median order size), a genuine extreme value can often stay in the data untouched, since the median won't be swayed by it anyway.
4. Choose the least destructive fix that solves the actual problem. Removing a row throws away information permanently; winsorizing (capping at a threshold) keeps the row but limits its influence; a log transform reduces the influence of all large values proportionally rather than singling one out. Reach for the most surgical option that addresses what's actually wrong.

Worked example

Say a $50,000 order shows up in a dataset where the 99th percentile is $1,200. Checking the account: it belongs to a corporate customer with three other orders over $30,000 in the past year, and the order line items list 500 units of the same product, consistent with a bulk purchase. That's a legitimate outlier, kept as-is for a sum, and probably fine to leave untouched for a median too since it won't move it much. Contrast that with a second $50,000 order from a first-time individual customer buying a single unit of a $12 product: no supporting evidence, and a quick check shows the quantity field was likely fat-fingered as 4,000 instead of 4. That one gets corrected (or excluded and flagged), not because it's large, but because there's a specific, checkable reason to distrust it.

Trade-offs and pitfalls

The temptation to "just remove outliers because they skew the numbers" is a real trap: skewing the numbers is sometimes the entire point, if the numbers are genuinely skewed. Document whatever decision you make and why, since a silent removal is invisible to anyone auditing the analysis later, and a wrong call here (removing real revenue, or keeping a fat-fingered order) directly changes a business number.