**Overview (goal):** perform zero-downtime schema change on a hot Postgres table (~100k QPS) with no API interruption and safe rollback.**Phase 1 — Clarify & prepare**- Requirements: exact column changes (add/drop/rename/type), SLAs, retention, replication lag tolerance.- CI / staging run of full plan on prod-sized copy (rds snapshot -> staging).**Phase 2 — Schema changes (online-safe)**- Prefer additive, non-blocking changes: ADD COLUMN with DEFAULT NULL, no immediate backfill.- Use:sql
ALTER TABLE events ADD COLUMN new_col TEXT;
- Avoid ALTER TYPE or large table rewrites. If type change needed, create new column.**Phase 3 — Backfill**- Use batched, transactional updates to avoid bloat/locks. Example script (psql + pg_sleep):bash
# backfill_batches.sh
limit=10000; offset=0
while true; do
psql -c "WITH t AS (
SELECT id FROM events WHERE new_col IS NULL ORDER BY id LIMIT $limit
)
UPDATE events SET new_col = compute(old_col) FROM t WHERE events.id = t.id;"
break_if_no_rows
sleep 0.2
done
- Alternatively use pg_repack or pt-osc style tool for heavy migrations.**Phase 4 — Read/write compatibility**- Deploy application changes in two phases: 1. Read from existing column; write to both old_col and new_col (dual-write). 2. Read from new_col if present; fallback to old_col.- Feature-flag dual-write via config and gradual traffic switch.**Phase 5 — Data verification**- Consistency checks:sql
SELECT count(*) FROM events WHERE coalesce(new_col,'') != coalesce(compute(old_col),'');
- Row-sample checks, CRC hashes, checksum tool:sql
SELECT md5(string_agg(id||':'||coalesce(new_col,''),',')) FROM events;
- Verify replication lag (pg_stat_replication), table bloat (pg_stat_user_tables).**Phase 6 — Cutover**- Once backfill + verification pass, switch reads to new_col behind a flag; monitor errors.- After 24–72 hours of healthy ops, drop old_col in low-traffic window:sql
ALTER TABLE events DROP COLUMN old_col;
(Only if safe; consider keeping for longer for rollback speed.)**Rollback plan**- If issues during dual-write: flip feature flag to read old_col only; stop writing new_col.- If after cutover and problems: re-enable reads from old_col; re-populate old_col from new_col if it was dropped — keep backups / WAL for PITR.- Pre-step: take immediate snapshot before destructive steps.**Monitoring & alerts**- Metrics: error rate, latency (p95/p99), QPS, replication lag, deadlocks, table bloat, vacuum stats.- Tools: Prometheus + Grafana, PagerDuty, Datadog APM, pg_stat_activity queries.- Health checks: synthetic requests hitting read and write paths, compare responses.**Coordination & communication**- Notify dependent teams and analytics: timetable, expected read schema, migration flags, migration channel.- Provide migration runbook, downtime plan (none expected), rollback triggers, contact list.- Downstream analytics: coordinate for schema changes in materialized views/ETL; pause/validate ETL. Provide SQL mapping and sample queries.**Automation & safety**- Use CI pipelines to run migration scripts with dry-run mode.- Use transaction-safe batches, maintain WAL retention and point-in-time recovery window.- Keep audit logs of writes (dual-write marker) for replay if needed.This plan emphasizes additive changes, dual-write/read compatibility, batched backfill, automated verification, monitoring, and clear communication to ensure zero-downtime and fast rollback.