Approach (methodical steps)1. Reproduce & measure: capture query plan (EXPLAIN/EXPLAIN ANALYZE), runtime, I/O, row estimates and hotspots.2. Push predicates early: ensure WHERE filters are applied as soon as possible (predicate pushdown) and on partition columns for partition pruning.3. Replace expensive nested loops/subqueries with joins or window functions where appropriate.4. Pre-aggregate or materialize expensive work if reused (materialized view / aggregated table).5. Add or adjust indexes (covering, filtered) only after verifying plan benefits.6. Iterate: test cost and runtime for each change.Example: nested correlated subquery -> window functionSlow pattern (per-row correlated subquery):sql
SELECT f.user_id,
f.event_time,
(SELECT COUNT(*) FROM fact f2
WHERE f2.user_id = f.user_id
AND f2.event_time >= f.event_time - interval '7 days'
AND f2.event_time <= f.event_time) AS cnt_7d
FROM fact f
WHERE f.event_time BETWEEN '2025-01-01' AND '2025-02-01';
Transforms to window aggregate (single scan, streaming):sql
SELECT user_id, event_time, cnt_7d
FROM (
SELECT user_id, event_time,
COUNT(*) OVER (PARTITION BY user_id
ORDER BY event_time
RANGE BETWEEN INTERVAL '7 days' PRECEDING AND CURRENT ROW) AS cnt_7d,
ROW_NUMBER() OVER () rn
FROM fact
WHERE event_time BETWEEN '2025-01-01' AND '2025-02-01' -- predicate pushdown, enables partition pruning
) t;
When window RANGE is unsupported or data is huge, pre-aggregate using a rolling precomputation:- Materialize daily counts per user: daily_user_counts(user_id, day, cnt)- Join or run a fast sliding sum on that much-smaller table.Partition pruning & predicate pushdown- Partition the 2TB fact by event_date (range) or by (event_date, tenant_id) if multi-tenant.- Ensure queries filter on event_date so the planner prunes partitions; push filters into subqueries/CTEs (use inline or temporary tables if planner fails).Materialized aggregates & pre-aggregation- If queries repeatedly need weekly/monthly aggregates, build materialized views or aggregated tables refreshed incrementally (daily upserts).- Use incremental ETL to maintain aggregates (e.g., Spark job that compacts last N days).Indexes & physical tuning- Create covering indexes for frequent point lookups: e.g., (user_id, event_time, metric) INCLUDE (other cols).- Use filtered/partial indexes for common WHERE clauses (e.g., status = 'active').- For wide tables, consider columnar storage (Parquet/ORC) and query engines (Redshift, BigQuery, Snowflake) with clustering on frequently filtered columns.Plan-guided trade-offs- Window functions and joins reduce CPU vs correlated subqueries but may increase memory. Test with EXPLAIN.- Materialized aggregates reduce query latency at the cost of storage and ETL complexity—use for frequent patterns.- Indexes speed selective reads but slow writes; choose based on read/write balance.Key takeaways- Always start from the plan and metrics.- Push filters to enable partition pruning and reduce scanned data.- Replace per-row subqueries with set-based window functions or joins.- Pre-aggregate or materialize if work is repeated; choose incremental updates.- Validate each change with EXPLAIN/ANALYZE and regression tests.