Situation & goal: We must rename a column and split its value into two new columns across millions of rows without downtime and while clients continue to read/write.Strategy overview (safe, incremental):1. Compatibility layer- Add new columns (col_a, col_b) nullable.- Keep old column (col_old) until cutover.- Implement adapter in application/service: - Writes: dual-write — on every write, write both col_old and computed split into col_a/col_b. - Reads: prefer new columns if non-null; fallback to parsing col_old.Pseudocode:python
def read_row(row):
if row.col_a is not None or row.col_b is not None:
return (row.col_a, row.col_b)
return parse_old(row.col_old)
def write_row(val):
col_a, col_b = split(val)
db.update(..., col_old=val, col_a=col_a, col_b=col_b)
2. Incremental backfill- Backfill in small batches (e.g., 1k–100k rows) with id-range or timestamp cursor.- Use idempotent updater: only write rows where col_a IS NULL to avoid clobbering concurrent writes.- Throttle and parallelize: tune worker concurrency, measure DB load; exponential backoff when contention rises.SQL backfill example:sql
UPDATE table
SET col_a = split_part(col_old,':',1), col_b = split_part(col_old,':',2)
WHERE col_a IS NULL
AND id BETWEEN $start AND $end;
3. Testing strategy- Unit tests for adapter logic and parsing/splitting.- Integration tests exercising dual-write/read semantics; simulate partial backfill.- Staging: run full backfill on production-sized snapshot; run traffic replay against staging to validate behavior and performance.- Dry-run: backfill to a shadow table or add flagging column backfill_done to validate values before switch.4. Performance considerations- Add necessary indexes carefully; avoid creating heavy indexes during peak traffic — build CONCURRENTLY where supported.- Use non-blocking schema changes (ADD COLUMN is usually fast); avoid locking operations (no ALTER TYPE/RENAME that locks).- Monitor metrics: replication lag, CPU, I/O, query latencies, deadlocks.- Tune batch size, parallel workers, and transaction size to keep transaction log growth acceptable.5. Cutover & rollbackCutover steps:- Ensure backfill percent ~100% and adapter reads returning new columns for a sample.- Switch reads to read-only-from-new (flip feature flag to prefer new columns).- Stop writing col_old in a controlled release; keep ability to write both for safety.- Remove adapter fallback after a stabilization period.Rollback plan:- Feature-flag controlled adapter so we can revert to using col_old immediately.- If backfill introduced corruption, reverse by using backups or re-running backfill from consistent snapshot; keep pre-migration backup and binlog/replication window.- Don’t drop col_old until rollback window passes.Key trade-offs & notes:- Dual-write increases write latency; monitor and quantify.- Adapter + feature flags gives safest, fastest rollback.- Prioritize non-blocking schema ops and idempotent backfill to avoid locks and inconsistent states.This approach ensures zero-downtime, safe validation, measurable performance impact, and quick rollback capability.