I follow a repeatable, prioritized workflow so dashboards are trustworthy and reproducible.1) Quick profiling (discover)- Goal: understand size, schema, distributions, missingness.- Tools: pandas: df.info(), df.describe(), df.nunique(), df.isnull().sum(); SQL: SELECT COUNT(*), COUNT(DISTINCT col), AVG/STD, SUM(CASE WHEN col IS NULL THEN 1 END).- Example:python
# pandas
df.info()
df.describe(include='all')
df.isnull().sum().sort_values(ascending=False).head(10)
2) Schema & type fixes- Enforce types early: dates, numeric, categorical.- pandas: pd.to_datetime(df.col, errors='coerce'), pd.to_numeric(..., errors='coerce'), df[col]=df[col].astype('category')- SQL: CAST/TRY_CAST, e.g. TRY_CAST(col AS DATE)3) Missing data handling- Distinguish MCAR/MAR/not missing: impute, flag, or drop.- pandas: df.dropna(subset=['key']), df['col'].fillna(value) or create is_missing flag- Excel: filter blanks, use IFERROR/IFNA, Data Validation lists4) Deduplication & key integrity- Detect duplicates: df.duplicated(subset=['pk','ts'], keep=False); drop with df.drop_duplicates()- SQL: row_number() OVER (PARTITION BY pk ORDER BY updated_at DESC) = 1 to keep latest- Validate joins with merge(..., indicator=True) to find unmatched rows.python
merged = df1.merge(df2, on='id', how='left', indicator=True)
merged['_merge'].value_counts()
5) Outliers & domain validation- Use business rules (negative sales, unrealistic ages). Treat via winsorize, cap, or mark for review.- SQL patterns: WHERE col < 0 OR col > max_expected6) Consistency / standardization- Normalize categorical labels (strip, lower, mapping dictionaries), address formatting, currency conversions.7) Documentation & reproducibility- Record transformations in SQL views, dbt models, or a pandas script/notebook with tests.- Add data-quality checks: row counts, null thresholds, foreign key checks automated in CI (dbt tests, Great Expectations).Common SQL snippets:sql
-- dedupe keep latest
WITH ranked AS (
SELECT *, ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY updated_at DESC) rn
FROM raw.users
)
SELECT * FROM ranked WHERE rn = 1;
Why this order: profiling informs which fixes matter; types first prevent downstream casting issues; dedupe/join integrity preserve accurate aggregates; validation ensures business-safe reporting. This workflow keeps BI reports accurate and auditable.