Approach: produce a reproducible, privacy-safe, representative dataset by stratified, time-aware sampling from production logs, then augment/curate for edge cases and tests.
Sampling:
- Define population window (e.g., last 6–12 months). Partition by time (monthly/weekly) and sample proportionally per partition to preserve seasonality.
- Stratify by label and key covariates (user cohort, region, device) to maintain feature distributions.
Handle class imbalance & rare slices:
- Keep an unbiased holdout that reflects production class ratio for evaluation.
- For training/test fixtures, use targeted oversampling (SMOTE or bootstrapping) and importance-weighted sampling for minority class.
- Extract and upsample rare slices (e.g., <0.5% groups) into a “slice set” so tests always include them; document slice definitions.
Anonymize/PII:
- Remove direct identifiers; hash IDs with salted one-way hashes (salt stored separately, not in test artifacts).
- Replace free-text PII with realistic synthetic placeholders or use differential privacy tools for aggregates.
- Mask or tokenise sensitive numeric fields (partial redaction) while preserving distributional properties.
Edge cases & temporal partitions:
- Inject/retain examples for missing values, extreme values, malformed inputs, and adversarial inputs.
- Create time-based partitions: train/dev/test by time (e.g., train up to T0, val T0–T1, test T1+), plus a rolling-validation set to catch concept drift.
Unit & integration test suitability:
- Produce small deterministic fixtures (seeded sampling) covering typical, rare, and failure cases for unit tests.
- For integration tests, provide larger sampled snapshots with realistic distributions but no secrets.
- Store metadata: sampling seed, provenance, schema, statistics, and slice definitions.
Secrets & reproducibility:
- Strip API keys, environment variables, and any tokens.
- Keep salts/keys in secure vaults (not in dataset repo). Provide scripts that can reproduce sampling when authorized.
- Validate with data quality checks (schema, ranges, label leakage) and automated privacy scans before any dataset is used in CI.
