Geospatial and Time-Series Data Questions
Specialized data shapes and their stores: geospatial data types, spatial indexing, and location queries; and time-series data with high-ingest, retention, downsampling, and cardinality management. Covers storage-engine tuning such as compression, chunking, and indexing for range queries, and when purpose-built extensions or databases beat general-purpose stores for these workloads. A niche but recurring topic for location- and telemetry-heavy systems.
In Python (you may reference scikit-learn or scipy), implement or describe a function that returns the k nearest drivers to a rider location when n can be up to 100k drivers. Explain coordinate choices (use BallTree with haversine vs KDTree on projected coordinates), memory/time complexity, and how you would support dynamic updates when driver positions change.
Sample Answer
Direct answer
For 100k drivers in one process, build a scikit-learn BallTree over driver positions using the haversine metric (great-circle distance on a sphere), converted to radians with latitude first, and call query(k=...). Build costs O(n log n), a query is roughly O(log n + k) on spread-out 2-D data, and memory is O(n), a few MB. Because these trees are static, handle moving drivers by keeping live positions in a plain array, rebuilding the tree as a snapshot every couple of seconds, and over-fetching candidates from the snapshot so they can be re-ranked by where each driver is NOW.
Terms used
- k-nearest-neighbour (kNN) query: "give me the k points closest to this one."
- Haversine distance: the shortest distance between two points along the surface of a sphere (the great-circle distance). It needs angles in radians.
- KDTree: a tree that repeatedly splits space with axis-aligned cuts (first by x, then by y, and so on). It assumes a flat, Euclidean space.
- BallTree: a tree that groups points into nested "balls" (a centre and a radius). It only needs a valid distance function, so it works with haversine.
- Projection: converting (lat, lon) degrees into flat x/y metres so ordinary Euclidean distance is meaningful.
Approach
- Pick coordinates that make the distance honest. Degrees of longitude shrink with latitude: at 37.75° N one degree of longitude is about 111.2 × cos(37.75°) ≈ 87.9 km, while one degree of latitude is about 111.2 km. Feeding raw (lat, lon) degrees into a Euclidean KDTree therefore ranks neighbours with a stretched ruler. Three correct options exist; this answer commits to the first (BallTree + haversine) and uses it for the rest of the code below, the other two are useful context, not the default:
- BallTree + haversine on
[lat, lon]in radians. Exact on the sphere, works everywhere, including across the 180° meridian (the antimeridian). - KDTree on projected metres (for example a local equirectangular projection centred on the city: x = R·lon·cos(lat0), y = R·lat). Fast and fine for one metro area, but distortion grows as you move away from the centre latitude, and it breaks at the antimeridian.
- KDTree on 3-D unit vectors (x, y, z on the unit sphere), a niche trick for when only a plain Euclidean KDTree library is available. Straight-line chord distance (a straight segment connecting two points, cutting through the Earth's interior rather than following its curved surface) through the Earth grows monotonically (it only ever increases, never decreases, as the true great-circle distance increases) with great-circle distance, so the neighbour ORDER is exact globally. Convert back with angle = 2·asin(chord/2).
- BallTree + haversine on
- Scale the result.
BallTree(metric="haversine")returns distances in radians on a unit sphere; multiply by the Earth radius (6,371,008.8 m, the mean radius) to get metres. - Verify against brute force. For a new index, always compare with an O(n) exact scan on a sample. It catches the classic bug of passing
[lon, lat]or degrees.
Code: static kNN and the coordinate choices
Requires numpy and scikit-learn.
import numpy as np
from sklearn.neighbors import BallTree, KDTree
EARTH_RADIUS_M = 6_371_008.8
rng = np.random.default_rng(42)
# 100k synthetic drivers in a San Francisco Bay Area box, columns = [lat, lon] in degrees
N = 100_000
drivers_deg = np.column_stack([
rng.uniform(37.60, 37.90, N),
rng.uniform(-122.55, -122.20, N),
])
def build_index(latlon_deg):
# haversine metric in scikit-learn expects [lat, lon] in RADIANS, latitude first
# leaf_size caps how many points sit in a leaf before the tree stops splitting
# and falls back to a linear scan inside it; it trades memory for query speed
# (memory drops roughly in proportion to leaf_size) without changing which
# neighbours are returned. 40 is scikit-learn's own default.
return BallTree(np.radians(latlon_deg), metric="haversine", leaf_size=40)
def k_nearest(tree, rider_lat, rider_lon, k=5):
q = np.radians([[rider_lat, rider_lon]])
dist_rad, idx = tree.query(q, k=k) # distances come back in radians (unit sphere)
return idx[0], dist_rad[0] * EARTH_RADIUS_M # scale by Earth radius to get meters
def brute_force(latlon_deg, rider_lat, rider_lon, k=5):
lat, lon = np.radians(latlon_deg[:, 0]), np.radians(latlon_deg[:, 1])
p, l = np.radians(rider_lat), np.radians(rider_lon)
a = np.sin((lat - p) / 2) ** 2 + np.cos(p) * np.cos(lat) * np.sin((lon - l) / 2) ** 2
d = 2 * EARTH_RADIUS_M * np.arcsin(np.sqrt(np.clip(a, 0, 1)))
order = np.argsort(d)[:k]
return order, d[order]
tree = build_index(drivers_deg)
rider = (37.7749, -122.4194)
idx, dist_m = k_nearest(tree, *rider, k=5)
bf_idx, bf_d = brute_force(drivers_deg, *rider, k=5)
print("BallTree ids:", idx.tolist())
print("BallTree m :", np.round(dist_m, 1).tolist())
print("brute force ids match:", idx.tolist() == bf_idx.tolist(), "| max |diff| m:", float(np.max(np.abs(dist_m - bf_d))))
# KDTree cannot use haversine; it only supports Minkowski-family metrics
# (distance formulas built from raw coordinate differences, such as Euclidean
# or Manhattan distance, that assume a flat space)
print("'haversine' in KDTree.valid_metrics:", "haversine" in KDTree.valid_metrics)
# Alternative 1: local equirectangular projection to meters, then KDTree (Euclidean)
lat0 = np.radians(rider[0])
def project(latlon_deg):
lat, lon = np.radians(latlon_deg[:, 0]), np.radians(latlon_deg[:, 1])
return np.column_stack([EARTH_RADIUS_M * lon * np.cos(lat0), EARTH_RADIUS_M * lat])
kd = KDTree(project(drivers_deg))
kd_d, kd_i = kd.query(project(np.array([rider])), k=5)
print("KDTree(projected) ids match:", kd_i[0].tolist() == bf_idx.tolist(), "| max |diff| m:", round(float(np.max(np.abs(kd_d[0] - bf_d))), 3))
# Alternative 2: 3D unit vectors + KDTree. Chord length is monotonic in great-circle angle,
# so the neighbor ORDER is exact anywhere on Earth (poles, antimeridian included).
def to_xyz(latlon_deg):
lat, lon = np.radians(latlon_deg[:, 0]), np.radians(latlon_deg[:, 1])
return np.column_stack([np.cos(lat) * np.cos(lon), np.cos(lat) * np.sin(lon), np.sin(lat)])
kd3 = KDTree(to_xyz(drivers_deg))
chord, i3 = kd3.query(to_xyz(np.array([rider])), k=5)
arc_m = 2 * np.arcsin(chord[0] / 2) * EARTH_RADIUS_M
print("KDTree(xyz) ids match:", i3[0].tolist() == bf_idx.tolist(), "| max |diff| m:", round(float(np.max(np.abs(arc_m - bf_d))), 6))
# Antimeridian check: two drivers either side of 180 degrees, rider at lon 179.99
edge = np.radians([[0.0, 179.995], [0.0, -179.995], [0.0, 179.0]])
edge_tree = BallTree(edge, metric="haversine")
d, i = edge_tree.query(np.radians([[0.0, 179.99]]), k=3)
print("antimeridian order:", i[0].tolist(), "meters:", np.round(d[0] * EARTH_RADIUS_M, 1).tolist())
Output:
BallTree ids: [49604, 87224, 30155, 78708, 38563]
BallTree m : [37.4, 48.0, 72.1, 94.9, 106.4]
brute force ids match: True | max |diff| m: 0.0
'haversine' in KDTree.valid_metrics: False
KDTree(projected) ids match: True | max |diff| m: 0.0
KDTree(xyz) ids match: True | max |diff| m: 0.0
antimeridian order: [0, 1, 2] meters: [556.0, 1667.9, 110083.1]
All three correct methods agree with the brute-force scan, and KDTree confirms it has no haversine metric. The antimeridian test puts the driver at longitude -179.995 (1,667.9 m away) ahead of the one at 179.0 (110 km away), which a naive "subtract the longitudes" approach gets backwards.
Supporting dynamic updates
scikit-learn's BallTree/KDTree and SciPy's KDTree have no insert, delete or move operation: changing a point means rebuilding. With drivers sending a GPS ping every 4 s, 100,000 drivers produce 100,000 / 4 = 25,000 updates per second, so updating the tree per ping is not an option. The design I would commit to for a single-city, in-process service:
- Live state is an array, one row per driver. An update is an O(1) write; availability (online, not on a trip) is a boolean per driver.
- The tree is a snapshot rebuilt on a timer (every 2 s here) off the request path, then swapped in atomically, so a reader never sees a half-built tree.
- Queries over-fetch and re-rank. Ask the snapshot for k + 20 candidates, drop unavailable drivers, then rank those candidates by haversine distance from their CURRENT positions.
import threading
import numpy as np
from sklearn.neighbors import BallTree
EARTH_RADIUS_M = 6_371_008.8
def haversine_m(lat1, lon1, lat2, lon2):
lat1, lon1, lat2, lon2 = map(np.radians, (lat1, lon1, lat2, lon2))
a = np.sin((lat2 - lat1) / 2) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin((lon2 - lon1) / 2) ** 2
return 2 * EARTH_RADIUS_M * np.arcsin(np.sqrt(np.clip(a, 0, 1)))
class DriverIndex:
"""Live positions are updated in O(1); the BallTree is a periodic snapshot.
Queries over-fetch from the snapshot, then re-rank the candidates by their
CURRENT position, so a moved driver is scored where it is now.
"""
def __init__(self, latlon_deg):
self.pos = latlon_deg.copy() # live [lat, lon] per driver id
self.available = np.ones(len(latlon_deg), bool)
self._lock = threading.Lock()
self.rebuild()
def update(self, driver_id, lat, lon): # O(1): no tree touched
self.pos[driver_id] = (lat, lon)
def set_available(self, driver_id, flag):
self.available[driver_id] = flag
def rebuild(self): # O(n log n), run on a timer off the hot path
# (the latency-sensitive code that handles each
# incoming request; kept separate so a query is
# never blocked on this rebuild)
snapshot = self.pos.copy()
tree = BallTree(np.radians(snapshot), metric="haversine")
with self._lock: # atomic swap (the tree reference changes in one
# indivisible step): readers never see a half-built tree
self._tree = tree
def k_nearest(self, lat, lon, k=5, overfetch=20):
with self._lock:
tree = self._tree
_, cand = tree.query(np.radians([[lat, lon]]), k=k + overfetch)
cand = cand[0][self.available[cand[0]]] # drop drivers who went offline or got a trip
live = self.pos[cand]
d = haversine_m(lat, lon, live[:, 0], live[:, 1])
order = np.argsort(d)[:k]
return cand[order], d[order]
rng = np.random.default_rng(7)
N = 100_000
drivers = np.column_stack([rng.uniform(37.60, 37.90, N), rng.uniform(-122.55, -122.20, N)])
index = DriverIndex(drivers)
# Between two rebuilds every driver moves up to 30 m (15 m/s for 2 s) in a random direction
step_m = rng.uniform(0, 30, N)
heading = rng.uniform(0, 2 * np.pi, N)
dlat = np.degrees(step_m * np.cos(heading) / EARTH_RADIUS_M)
dlon = np.degrees(step_m * np.sin(heading) / (EARTH_RADIUS_M * np.cos(np.radians(drivers[:, 0]))))
for i in range(N):
index.update(i, drivers[i, 0] + dlat[i], drivers[i, 1] + dlon[i])
index.set_available(123, False)
riders = np.column_stack([rng.uniform(37.65, 37.85, 1000), rng.uniform(-122.50, -122.25, 1000)])
for overfetch in (0, 20):
exact = 0
for lat, lon in riders:
got, _ = index.k_nearest(lat, lon, k=5, overfetch=overfetch)
d = haversine_m(lat, lon, index.pos[:, 0], index.pos[:, 1])
d[~index.available] = np.inf
truth = np.argsort(d)[:5]
exact += got.tolist() == truth.tolist()
print(f"stale snapshot, overfetch={overfetch:2d}: exact top-5 for {exact}/1000 riders")
index.rebuild()
exact = sum(
index.k_nearest(lat, lon, 5)[0].tolist()
== np.argsort(np.where(index.available, haversine_m(lat, lon, index.pos[:, 0], index.pos[:, 1]), np.inf))[:5].tolist()
for lat, lon in riders
)
print(f"fresh snapshot, overfetch=20: exact top-5 for {exact}/1000 riders")
Output:
stale snapshot, overfetch= 0: exact top-5 for 586/1000 riders
stale snapshot, overfetch=20: exact top-5 for 1000/1000 riders
fresh snapshot, overfetch=20: exact top-5 for 1000/1000 riders
Why 20 extra candidates is enough here (and when it is not)
The simulated box is 0.30° × 111.2 km ≈ 33.4 km tall by 0.35° × 87.9 km ≈ 30.8 km wide, about 1,027 km², so density ρ ≈ 100,000 / 1,027 ≈ 97 drivers per km². For randomly scattered points, a circle of radius r centred on the query point holds on average rho x pi x r^2 drivers (density times the circle's area); setting that equal to k and solving for r gives the distance to the k-th nearest neighbour, roughly
rk≈πρkwhich gives about 128 m for k = 5 and about 286 m for k = 25. A driver can only be wrongly left out if it sat beyond the 25th candidate at snapshot time and moved inside the true top 5 before the next rebuild. The gap is about 286 − 128 ≈ 158 m, and in 2 s at 15 m/s (54 km/h) a driver moves at most 30 m, so the over-fetch margin is roughly five times what motion can consume. Without over-fetch, only 586 of 1,000 riders got the exact top 5. If drivers are sparser (suburbs) or the rebuild interval is longer, re-derive the over-fetch from the same inequality rather than keeping 20.
Complexity
| Operation | Time | Memory |
|---|---|---|
| Build BallTree / KDTree | O(n log n) | O(n): 100,000 × 2 float64 = 1.6 MB of coordinates plus 0.8 MB of int64 index array and node bounds |
| kNN query | about O(log n + k) for spread-out low-dimensional data; O(n) worst case when points are pathologically clustered (nearly all crammed into one tiny region, so the tree's branching gives it no leverage) | O(k) |
| Brute-force check | O(n) per query | O(n) temporary distances |
| Position update (snapshot design) | O(1) | none extra |
| Query with over-fetch m and re-rank | O(log n + k + m), plus O((k + m) log(k + m)) to sort | O(k + m) |
Edge cases
- Order and units: haversine in scikit-learn wants
[lat, lon]in radians. Passing degrees or[lon, lat]returns wrong neighbours silently. - Fewer than k drivers (or fewer than k + m after filtering):
queryraises if k exceeds n; clamp k to the number of points and return what exists. - Antimeridian and poles: fine with haversine or 3-D vectors; broken with a flat projection.
- Ties and duplicates (drivers queued at an airport at the same coordinate): order among equals is arbitrary; add a tie-breaker such as time since last trip if fairness matters.
- Stale drivers: a driver whose last ping is older than, say, 3 missed intervals should be marked unavailable, not matched at an old position.
Trade-offs and pitfalls
- Straight-line distance is a candidate filter, not the answer. A driver 300 m away across a river or on the wrong side of a divided highway may be 10 minutes out. Production dispatch takes the kNN shortlist (say 20 to 50) and ranks it by road-network ETA (estimated time of arrival) from a routing engine.
- Projected KDTree vs BallTree: inside one city a projected KDTree gives the same neighbours (shown above) and is simpler to reason about in metres. Choose BallTree or 3-D vectors when the service spans many latitudes or the antimeridian (Fiji, or New Zealand together with its Chatham Islands).
- When to leave the in-process tree: once positions must be shared across many stateless API servers, or the fleet spans many cities, move the live index out of process. Common choices are a grid index (bucket drivers by geohash or H3 cell, where an update is an O(1) move between buckets and a query scans the rider's cell plus neighbouring cells), or Redis GEO commands (
GEOADDto upsert a position,GEOSEARCH ... BYRADIUS ... ASC COUNT kto query). The snapshot-and-re-rank idea still applies to any index that is cheaper to rebuild than to update. - Rebuild cost is O(n log n) per interval, so doubling the fleet or halving the interval roughly doubles background CPU. Measure it on your hardware before choosing the interval rather than assuming.
A rideshare analytics warehouse needs to answer ad-hoc questions like 'find trips that started within 500 meters of these coordinates, during this one-hour window' at petabyte scale, with sub-minute response for analysts running exploratory queries. Design the schema and query strategy: how would you make a query like this fast without a full scan, and what would you trade off between query latency, storage overhead, and the extra work you do at write time to keep the index maintained?
Sample Answer
Direct answer
Do not try to index the petabyte. Build a narrow trip-starts table (one row per trip: ID, start time, start latitude/longitude and a start-cell key), partition it by start hour, and sort the data inside each partition by a spatial cell key such as a precision-6 geohash. A query then prunes on two axes before reading anything: the time predicate selects one hour's partition, and the cell predicate (start_cell IN the 9 cells around the point) lets the engine skip every file or block (the physical chunk a partition is split into on disk, called a file in an open table format such as Iceberg or Delta, or a block in a warehouse's own storage; both are pruned the same way) whose min/max cell range cannot match. Only the remaining candidates get the exact 500 m distance check. The costs are a derived column computed at write time, a periodic re-sort (compaction) of recently written partitions, and a small duplicate table; in return, the bytes scanned per query drop by several orders of magnitude.
How warehouses make "without a full scan" possible
Key terms.
- Columnar storage stores each column separately, so a query reading 5 columns of a 100-column table reads only those 5.
- Partition pruning: the table is split into partitions by a column (here, start hour); a filter on that column skips whole partitions.
- Clustering / sort order and zone maps: inside a partition, data is written in files or blocks, and the engine keeps each block's minimum and maximum per column (a zone map, also called min/max statistics). If rows are sorted by a key, a filter on that key skips every block whose range excludes it. BigQuery clustering, Snowflake clustering keys, Redshift sort keys and the sort orders of open table formats (a metadata layer over plain data files that tracks each file's schema, partitions and per-column min/max statistics, so a query engine can plan which files to read without opening them) such as Apache Iceberg and Delta Lake (which manage Parquet columnar files) are all versions of this.
- A geohash turns a location into a string where each extra character is a smaller cell and nearby points usually share a prefix, so sorting by it keeps nearby trips in the same blocks.
The trick is to turn a 2D spatial predicate into a 1D key the engine can prune on, because zone maps work on one column at a time: min/max on latitude and longitude separately prunes poorly (a thin latitude band spans the whole planet in longitude).
The design
Tables.
| Table | Contents | Role |
|---|---|---|
trips (the petabyte) | Full trip records including GPS traces, fares, events | Source of truth; queried only by trip_id joins or broad batch jobs |
trip_starts | trip_id, start_ts, start_lat, start_lon, start_geohash6, city_id | The ad-hoc search surface |
trip_starts partitioned by start_ts truncated to the hour (or by day with hour as the leading sort key if the engine caps partition counts: most engines slow down or refuse to plan a query once a table has too many partitions, commonly somewhere in the thousands), then sorted by (start_geohash6, start_ts).
Query shape.
SELECT t.trip_id, t.start_ts
FROM trip_starts t
WHERE t.start_ts >= TIMESTAMP '2026-06-12 17:00:00'
AND t.start_ts < TIMESTAMP '2026-06-12 18:00:00'
AND t.start_geohash6 IN (:c1, :c2, :c3, :c4, :c5, :c6, :c7, :c8, :c9)
AND ST_DWITHIN(ST_GEOGPOINT(t.start_lon, t.start_lat),
ST_GEOGPOINT(:lon, :lat), 500);
(ST_GEOGPOINT and ST_DWITHIN are BigQuery's spellings; each warehouse has equivalents.) A search on only the point's own cell would miss a trip that started a few metres away but just across a cell edge, so the query expands to a 3x3 block: the point's precision-6 geohash cell and its 8 neighbours, computed by the client. A precision-6 cell is about 0.61 km tall everywhere and, at 40 degrees latitude, about 0.94 km wide (360 / 2^15 degrees of longitude times cos(latitude)), so a 500 m circle around any point inside the centre cell stays inside the 3x3 block; the general rule is that a 3x3 block only fully covers the search circle when the radius is no larger than the cell's shortest side. The width is 1.222 km times cos(latitude), which drops below 500 m above about 65.8 degrees latitude; beyond that the client should switch to precision 5.
Worked example: bytes scanned
Inputs (stated assumptions, not measurements): 3 years of history in trips totalling 1 PB; 20,000,000 trips per day; trip_starts rows of about 60 bytes compressed.
With the hour partition selected and blocks sorted by geohash, the 9 target cells are a tiny slice of a global service's cells, so the engine reads the one or few blocks whose cell ranges overlap them. Estimate: a compacted 50 MB hourly partition typically lands in on the order of 10 to 20 files of a couple of megabytes each; because file order follows the geohash sort, the 9 adjacent target cells usually fall inside one file, occasionally two if they straddle a file boundary, so the engine reads on the order of a few megabytes instead of 50 MB, and many orders of magnitude less than a petabyte. Exact numbers depend on the engine's block size, so measure them with the engine's "bytes processed" figure on a real query rather than trusting this estimate.
Wider windows scale linearly: a one-week window reads 168 hourly partitions, still pruned by cell within each.
Trade-offs: latency vs storage vs write-time work
| Lever | Query latency | Storage overhead | Write-time work |
|---|---|---|---|
Nothing (full scan of trips) | Minutes to hours; scans up to a petabyte per query | None | None |
| Time partitioning only | Reads 38 GB per hour queried | None | Trivial |
trip_starts + hour partition + geohash sort (committed) | Seconds; reads megabytes | About 1.2 GB per day, a rounding error beside the traces | Compute a geohash per row (cheap), plus re-sorting recent partitions as late data lands |
| Engine-managed search index on geography (for example Snowflake search optimization, a paid Snowflake feature that builds and maintains a search index automatically) | Fast point lookups | Index storage billed separately | Background maintenance billed as compute |
| Pre-aggregated cell-by-hour counts | Milliseconds, but only for counting questions, never "which trips" | Small | A stream job |
The committed option wins because it serves both the time and the space predicate with the engine's cheapest mechanism (block skipping) and costs almost nothing to store. The write-time price is clustering maintenance: rows arriving late or out of order land in new, unsorted files, which widens their min/max ranges and silently degrades pruning. Schedule compaction that re-sorts each partition once it stops receiving writes (for example 24 hours after the hour closes), and monitor the bytes-scanned trend for a fixed benchmark query.
Pitfalls
- Sorting by latitude then longitude looks spatial but prunes badly, because every longitude sits inside a narrow latitude band.
- Cell key without the exact check returns false positives up to the block's corners; the
ST_DWITHINrefinement is required for correctness. - Wrapping the column in a function (
WHERE ST_GEOHASH(point) IN ...computed at query time) defeats pruning; the key must be a stored column. - Time zones: partition on UTC and convert analyst input, or a local-time "one hour" straddles two partitions.
- What would change the design: if analysts mostly asked city-level or polygon questions rather than 500 m point questions, cluster by
city_idthen time instead; if the dominant query were "trips passing near a point" rather than starting there, index route cells (a trip-to-cells table) instead of start cells.
Write an efficient PostGIS SQL query to find up to 5 nearest available listings within 5 km for each search coordinate in 'searches(lat, lon, search_time)'. Return listing_id, distance_km, price_usd, and compute the percentage of searches per city with at least 3 results. Assume 'listings(listing_id, lat, lon, is_active, price_usd)'. Use spatial indexes and lateral joins to keep the query performant.
Sample Answer
Direct answer
Store (or index) each listing as a geography point, put a partial GiST index on it covering only active listings, and drive the per-search lookup with a LEFT JOIN LATERAL subquery that filters with ST_DWithin(..., 5000) and orders by the <-> distance operator with LIMIT 5. The lateral join runs one small index probe per search instead of comparing every search against every listing. The city percentage is a second aggregation over the per-search result counts. One gap in the stated schema has to be closed first: searches has no city column and no key, so the answer adds a search_id and assigns a city with a point-in-polygon join (a join whose condition is "this point falls inside this polygon", here testing whether a search's coordinate lands inside a city's boundary shape) against a cities boundary table.
Why each piece is there
Key terms.
- A spatial index lets the database skip rows that are nowhere near the query point. PostGIS uses GiST (Generalized Search Tree), which stores a bounding box per row in a tree, so "which boxes overlap this 10 km square" is answered by walking a few tree pages.
geographyvsgeometry:geometrytreats coordinates as flat x/y, so distances come out in degrees.geographymeasures on the Earth's surface, soST_DWithin(a, b, 5000)means 5,000 metres. With rawlat, loninput and a kilometre requirement,geographyis the correct type.- SRID 4326 is the identifier for plain WGS 84 (World Geodetic System 1984) longitude/latitude, the coordinate system GPS reports.
- A lateral join (
LATERAL) is a subquery that may reference columns of the rows to its left, so it runs once per search row. That is exactly the "top N per group" shape. - KNN (k-nearest-neighbour) ordering:
ORDER BY a <-> b LIMIT klets PostGIS return rows nearest-first straight from the index.
Design decisions.
| Decision | Choice | Reason |
|---|---|---|
| Point representation | Stored generated column (a column PostgreSQL computes automatically from other columns via a formula and writes to disk once, at insert or update time, instead of recomputing it on every read) geog geography(Point,4326) | Built once at write time. The alternative is an expression index (an index built on the result of a computation over columns rather than on a stored column) on (ST_SetSRID(ST_MakePoint(lon, lat), 4326)::geography), which works if the table cannot change, but every query then has to repeat that exact expression or the index is ignored. |
| Index | USING gist (geog) WHERE is_active | A partial index holds only the rows the query can return, so it is smaller and every probe skips inactive listings for free. The query must repeat l.is_active for the planner (PostgreSQL's query optimizer: the component that decides which indexes and scan strategies to use for a given query) to use it. |
| Radius filter | ST_DWithin(geog, pt, 5000) | Index-aware: it expands the point by 5 km, uses the box for the index, then checks exact distance. ST_Distance(...) < 5000 in a WHERE clause cannot use the index. |
| Ordering | ORDER BY geog <-> pt LIMIT 5 | Stops after 5 rows per search. |
| Zero-result searches | LEFT JOIN LATERAL ... ON true | A plain lateral join would drop searches that found nothing, which would inflate the city percentage. |
| City | LEFT JOIN cities ON ST_Covers(c.geom, pt) | ST_Covers includes points exactly on the boundary (ST_Contains does not). Searches outside every polygon land in 'unassigned' instead of vanishing. |
Runnable solution (PostgreSQL 16, PostGIS 3.4)
The script creates the tables, generates deterministic synthetic data with a pinned seed, builds the indexes, runs both queries and checks the plan. Run it with psql -f in a database where PostGIS is installed.
-- Setup: tables as given, plus a surrogate search_id and a city boundary table.
CREATE EXTENSION IF NOT EXISTS postgis;
DROP TABLE IF EXISTS listings, searches, cities;
CREATE TABLE listings (
listing_id bigint PRIMARY KEY,
lat double precision NOT NULL,
lon double precision NOT NULL,
is_active boolean NOT NULL,
price_usd numeric(10,2) NOT NULL,
geog geography(Point,4326)
GENERATED ALWAYS AS (ST_SetSRID(ST_MakePoint(lon, lat), 4326)::geography) STORED
);
CREATE TABLE searches (
search_id bigserial PRIMARY KEY,
lat double precision NOT NULL,
lon double precision NOT NULL,
search_time timestamptz NOT NULL
);
CREATE TABLE cities (city text PRIMARY KEY, geom geometry(Polygon,4326) NOT NULL);
-- Deterministic synthetic data (PostgreSQL 16).
SELECT setseed(0.42);
INSERT INTO cities VALUES
('Austin', ST_MakeEnvelope(-97.90, 30.15, -97.60, 30.45, 4326)),
('Denver', ST_MakeEnvelope(-105.10, 39.60, -104.80, 39.90, 4326));
-- Austin: 3000 listings, Denver: 60 listings, ~80% active.
INSERT INTO listings (listing_id, lat, lon, is_active, price_usd)
SELECT g, 30.15 + random()*0.30, -97.90 + random()*0.30, random() < 0.8, round((50 + random()*450)::numeric, 2)
FROM generate_series(1, 3000) g;
INSERT INTO listings (listing_id, lat, lon, is_active, price_usd)
SELECT g, 39.60 + random()*0.30, -105.10 + random()*0.30, random() < 0.8, round((50 + random()*450)::numeric, 2)
FROM generate_series(3001, 3060) g;
-- 500 searches per city.
INSERT INTO searches (lat, lon, search_time)
SELECT 30.15 + random()*0.30, -97.90 + random()*0.30, timestamptz '2026-01-01' + g * interval '1 minute'
FROM generate_series(1, 500) g;
INSERT INTO searches (lat, lon, search_time)
SELECT 39.60 + random()*0.30, -105.10 + random()*0.30, timestamptz '2026-01-01' + g * interval '1 minute'
FROM generate_series(1, 500) g;
-- Indexes: partial GiST on active listings, GiST on city polygons.
CREATE INDEX listings_active_geog_gix ON listings USING gist (geog) WHERE is_active;
CREATE INDEX cities_geom_gix ON cities USING gist (geom);
ANALYZE listings; ANALYZE searches; ANALYZE cities;
-- Query 1: up to 5 nearest active listings within 5 km, per search.
CREATE TEMP TABLE search_results AS
SELECT s.search_id, s.search_time, COALESCE(c.city, 'unassigned') AS city,
nn.listing_id, nn.distance_km, nn.price_usd
FROM searches s
CROSS JOIN LATERAL (SELECT ST_SetSRID(ST_MakePoint(s.lon, s.lat), 4326) AS pt) p
LEFT JOIN cities c ON ST_Covers(c.geom, p.pt)
LEFT JOIN LATERAL (
SELECT l.listing_id, l.price_usd,
round((ST_Distance(l.geog, p.pt::geography) / 1000.0)::numeric, 3) AS distance_km
FROM listings l
WHERE l.is_active
AND ST_DWithin(l.geog, p.pt::geography, 5000)
ORDER BY l.geog <-> p.pt::geography
LIMIT 5
) nn ON true;
SELECT search_id, city, listing_id, distance_km, price_usd
FROM search_results WHERE search_id IN (1, 501)
ORDER BY search_id, distance_km;
-- Query 2: % of searches per city with at least 3 results.
SELECT city,
count(*) AS searches,
count(*) FILTER (WHERE n_results >= 3) AS with_3plus,
round(100.0 * count(*) FILTER (WHERE n_results >= 3) / count(*), 1) AS pct_3plus
FROM (SELECT search_id, city, count(listing_id) AS n_results
FROM search_results GROUP BY search_id, city) per_search
GROUP BY city ORDER BY city;
-- Plan check for the lateral probe.
EXPLAIN (COSTS OFF)
SELECT l.listing_id FROM listings l
WHERE l.is_active AND ST_DWithin(l.geog, ST_SetSRID(ST_MakePoint(-97.75, 30.30), 4326)::geography, 5000)
ORDER BY l.geog <-> ST_SetSRID(ST_MakePoint(-97.75, 30.30), 4326)::geography
LIMIT 5;
Output of that script (the empty setseed result and the NOTICE lines from the DROP ... IF EXISTS are omitted):
search_id | city | listing_id | distance_km | price_usd
-----------+--------+------------+-------------+-----------
1 | Austin | 1985 | 0.117 | 311.20
1 | Austin | 925 | 0.197 | 239.97
1 | Austin | 1743 | 0.398 | 223.44
1 | Austin | 2100 | 0.554 | 445.05
1 | Austin | 267 | 0.627 | 129.20
501 | Denver | 3034 | 0.776 | 347.95
501 | Denver | 3041 | 1.057 | 278.69
501 | Denver | 3010 | 2.690 | 306.64
501 | Denver | 3036 | 3.902 | 432.48
city | searches | with_3plus | pct_3plus
--------+----------+------------+-----------
Austin | 500 | 500 | 100.0
Denver | 500 | 373 | 74.6
Limit
-> Sort
Sort Key: ((geog <-> '...'::geography))
-> Bitmap Heap Scan on listings l
Recheck Cond: is_active
Filter: st_dwithin(geog, '...'::geography, '5000'::double precision, true)
-> Bitmap Index Scan on listings_active_geog_gix
Index Cond: (geog && _st_expand('...'::geography, '5000'::double precision))
(The long hex point literals in the plan are abbreviated to '...'.)
Reading the result
- Search 501 in Denver returned only 4 rows, which is correct: only 4 active listings sit within 5 km. The
LIMIT 5is a ceiling, not a guarantee. - The density difference drives the percentage. Austin has 3,000 listings in a 0.30 by 0.30 degree box, so every Austin search clears 3 results. Denver has only 60 listings in a box of the same size, 46 of them active in this run: spread over that box, which at Denver's latitude works out to about 25.6 km by 33.4 km, roughly 856 km², that is about 0.054 active listings per km². A 5 km search radius is a circle of about pi x 5^2 = 78.5 km², so a search is expected to find about 0.054 x 78.5, roughly 4.2 active listings, close to the 3-result bar the query checks against. With the expected count that close to the bar, whether any one search actually clears it varies a lot from search to search, and 373 of 500 Denver searches do: 373 / 500 = 74.6%.
- Reading the plan, line by line.
Bitmap Index Scan on listings_active_geog_gixwalks the partial GiST index and builds a bitmap (an in-memory list of candidate row locations) instead of visiting table rows directly;Index Cond: (geog && _st_expand(...))is the actual index lookup, where_st_expandgrows the search point into a bounding box covering the 5,000 m radius and&&(PostGIS's "bounding boxes overlap" operator) is the cheap, index-supported test the GiST tree can answer.Bitmap Heap Scan on listings lthen visits just those candidate rows in the table's own storage (the heap, as opposed to the index) to check the conditions the bitmap step could only approximate:Recheck Cond: is_activere-checks the real condition against each row once it gets there, because a bitmap only marks which pages might contain a match, andFilter: st_dwithin(...)re-checks the true distance the same way, since the bounding box only proves "close enough to be worth checking", not "actually within 5 km". TheBitmap Index Scanline is what proves the partial index was actually used; theRecheck/Filterlines are why the plan is still correct even though its first pass over the box is approximate. - The plan confirms the partial index is used:
Bitmap Index Scan on listings_active_geog_gixwith the&&bounding-box condition thatST_DWithingenerated. The planner chose a bitmap scan plus a small sort here rather than a pure KNN index walk, because a 5 km radius holds few enough rows that sorting them is cheap. Either plan touches only nearby rows, which is the point.
Complexity
Let S be the number of searches, L the number of active listings and m the listings inside one 5 km radius. Each probe costs roughly O(log L + m log m) (tree descent, then sort the candidates), so the whole query is about O(S (log L + m log m)); with m capped by the LIMIT 5 in practice this stays cheap, but the honest bound keeps the log m term rather than dropping it. The naive cross join is O(S * L): with 1,000 searches and 3,000 listings that is 3,000,000 distance computations, and it grows with both tables.
Edge cases and pitfalls
- Longitude first.
ST_MakePoint(x, y)takes(lon, lat). Swapping them puts Austin in Antarctica and the query silently returns nothing. - Distance units.
ST_Distanceongeographyreturns metres (divide by 1,000 for km). Ongeometryin SRID 4326 it returns degrees, which is meaningless as a distance. - Sphere vs spheroid.
<->on geography orders by sphere distance whileST_Distancedefaults to the more exact spheroid. The two can disagree by a fraction of a percent, which only matters for near-ties in ordering. - Ties. Add
, l.listing_idto theORDER BYif results must be stable across runs when two listings are equidistant. - Searches with no city. Counting only
INNER JOINmatches hides searches outside the polygon table; theCOALESCE(..., 'unassigned')bucket keeps them visible. search_timeis unused by the spec, but the same pattern takes a time filter trivially (for example restricting to the last 7 days before the lateral step), which is how you would run it incrementally over a largesearchestable.- At larger scale, run the lateral step in batches keyed by
search_idor by day, and consider physically reordering the table by location so nearby listings share heap pages (the fixed-size blocks of storage that hold a table's actual rows, the "heap"; reading fewer distinct pages means fewer disk or cache reads).CLUSTER(the command that physically rewrites a table's heap in index order) needs a non-partial index, so it would use a separate full GiST index ongeog.
Explain what downsampling and rollups are in TSDBs. Provide an example policy for converting 1-second resolution to 1-minute and 1-hour rollups and describe the query implications for dashboards and alerts.
Sample Answer
Direct answer
Downsampling means replacing many fine-grained samples with fewer coarse ones: sixty 1-second readings become one 1-minute summary. A rollup is the stored result of that summary, usually several statistics per bucket (minimum, maximum, sum, count, sometimes a last value), kept in its own table or series. A TSDB (time-series database, a store built for timestamped measurements) uses rollups so that long-range queries read thousands of rows instead of millions and so old data can be kept cheaply. A sensible policy: keep 1-second raw data for 7 days, 1-minute rollups for 90 days, 1-hour rollups for 2 years. Dashboards then pick the coarsest tier that still draws a smooth line; alerts evaluate on raw data, because a rollup can hide a short spike and always arrives late.
Key terms, in plain language
- Resolution: the time between stored points. 1-second resolution means 86,400 points per series per day.
- Series: one metric with one fixed set of labels, for example
cpu_usage{host="web-12"}. - Bucket: a fixed time window (one minute, one hour) that a rollup summarizes.
- Gauge vs counter: a gauge is a value that goes up and down (temperature, queue depth). A counter only increases (requests served since start) and is useful only as a rate of change.
- Retention: how long each tier is kept before it is deleted.
The example policy
| Tier | Resolution | Kept for | Stored per bucket | Built from |
|---|---|---|---|---|
| Raw | 1 s | 7 days | the value | ingestion |
| Minute rollup | 1 min | 90 days | min, max, sum, count, last | raw |
| Hour rollup | 1 h | 2 years | min, max, sum, count, last | minute rollup |
Why store sum and count instead of the average. Averages cannot be combined correctly. If minute A has 60 samples averaging 10 and minute B has 30 samples (a gap) averaging 40, the true combined average is (600 + 1,200) / 90 = 20, but averaging the two averages gives 25. With sum and count kept, the hour rollup is just sum of sums divided by sum of counts, which is exact.
Why build the hour tier from the minute tier. Sixty minute rows per hour are much cheaper to read than 3,600 raw rows, and min, max, sum and count all combine exactly (min of mins, max of maxes, sum of sums, sum of counts). This is called a hierarchical rollup.
Counters need a different rollup. For a counter, store the increase over the bucket (last value minus first value), corrected for resets when the process restarts and the counter drops back to zero, not the average of the raw counter value, which is meaningless. Concretely: if a counter reads 1,000,000 at the start of a minute and the process restarts partway through, ending the minute at 15,000 (climbing again from zero), a plain last - first gives 15,000 - 1,000,000 = -985,000, an impossible negative increase for a value that only ever goes up. Detecting the reset (the minute's last value is lower than its first) and instead recording that minute's increase as just the post-restart value, 15,000, gives a sensible positive number: everything before the reset was already counted in the rollups written before it happened, so only the climb from zero to 15,000 is new.
Schedule. A rollup job runs every minute over the minute that closed a little while ago (for example, the minute that ended 2 minutes ago, so late-arriving samples have landed), and every hour over the hour that closed. Deletion jobs drop raw data older than 7 days and minute data older than 90 days. The deletion must never run ahead of the rollup: if raw data for a minute is deleted before that minute's rollup is written, that minute is lost for good.
Worked example: what the policy buys per series
Counting stored values per series (each rollup row holds 4 numeric aggregates, ignoring last to keep the arithmetic simple):
Keeping 1-second raw data for the full 2 years instead would be 86,400 x 730 = 63,072,000 values, about 53 times more.
The query side is where the difference is felt. A 30-day dashboard panel for one series reads:
| Source | Points read |
|---|---|
| 1 s raw | 30 x 86,400 = 2,592,000 |
| 1 min rollup | 30 x 1,440 = 43,200 |
| 1 h rollup | 30 x 24 = 720 |
A panel is roughly 1,000 pixels wide, so 720 hourly points already fill it. Reading 2.6 million raw points to draw 1,000 pixels is pure waste.
Query implications for dashboards
-
Pick the tier by time range. Rule of thumb: choose the coarsest tier that still gives at least a few hundred points across the panel. Last 1 hour: raw. Last 7 days: minute (10,080 points). Last 90 days and beyond: hour. Most TSDB query layers can do this automatically or through a variable in the query: for example, Grafana paired with Thanos or Mimir (long-term storage layers that sit behind Prometheus, adding object-storage retention and a query API spanning wide time ranges), or TimescaleDB continuous aggregates (materialized rollup views that TimescaleDB keeps incrementally up to date as new raw rows arrive, so the coarse tier is only a few seconds behind instead of needing a separate batch job).
-
Show the right statistic. A line of hourly averages hides a 30-second outage. Plot max (or a max band) alongside the average for latency and saturation panels (dashboard panels tracking how full a limited resource is, such as queue depth or percent of disk used), so spikes stay visible at coarse resolution.
-
Percentiles do not roll up. You cannot compute the p99 (99th percentile) for an hour from sixty minute-level p99 values. If percentiles matter, store, per time bucket, a histogram (here "bucket" shifts meaning: not the time window above, but a range of VALUES, such as "100 to 150 ms", with a count of how many samples fell in that range) or a mergeable sketch (a small approximate summary built so that two of them, from different time windows or workers, can be combined into one and still estimate a percentile accurately, unlike a plain average), and compute the percentile at query time by merging the value-histograms (or sketches) from every time bucket in the query range.
A tiny worked example: merging two minutes' latency histograms (bin edges in ms) gives
[0,50): 80, [50,100): 60, [100,150): 45, [150,200): 15, 200 samples total. For p90, the target rank is 0.90 x 200 = 180. Walking the bins, the running count reaches 80 after the first bin and 140 after the second, still short of 180; the third bin,[100,150)with 45 samples, carries the count from 140 to 185, which is where the target rank of 180 falls. Estimate the value by interpolating inside that bin: 100 + (180 - 140) / 45 x 50, about 144 ms. That interpolated estimate, not an exact reconstruction of every raw sample, is what a merged histogram buys you. -
Tier boundaries. A query spanning "last 10 days" crosses the raw-to-minute boundary at day 7. Either query only the minute tier for the whole range (simplest, consistent), or stitch raw and minute data and accept a visible change in smoothness at the boundary.
Query implications for alerts
- Evaluate alerts on raw data. An alert like "CPU above 95% for 20 seconds" needs samples finer than the window. On raw data, a minute containing 20 seconds at 100% and 40 seconds at 50% fires it. On minute rollups, that minute's average is (20 x 100 + 40 x 50) / 60, about 67%, so an alert on the average never fires.
- Rollups are late by design. A minute rollup written 2 minutes after the minute closes adds that delay to any alert built on it.
- If an alert must run on a rollup (a slow trend over weeks, like disk filling), use max or last for threshold alerts and a rate over the counter increase for throughput alerts, never the average of a spiky gauge.
Trade-offs and pitfalls
- Rollups lose information permanently. After 7 days you cannot answer "what exactly happened at 14:03:17". Decide which questions need raw data and set raw retention from that, not from disk size alone.
- Late data. A sample that arrives after its minute was rolled up is missing from the rollup unless the job re-processes a trailing window (for example, always recompute the last 10 minutes).
- Averaging averages and averaging percentiles are the two classic errors. Keep sum, count and a histogram, and derive everything else.
- Retention ordering bug. A deletion job that runs before the rollup job has covered the same window silently erases history. Make the rollup lag strictly smaller than the raw retention.
Your table stores time-series events and needs to answer 'rolling metrics for this user over the last 7/30/90 days' at query time, without maintaining separate precomputed rollup tables. Design the schema, partitioning, clustering, and secondary indexes that would keep those queries fast as the table grows, and explain the query pattern you'd use to compute the rolling windows efficiently.
Sample Answer
Direct answer
Partition the events table by time (monthly), and give every partition a composite index on (user_id, event_ts) that includes the measured value. Then a 90-day lookup for one user is one contiguous index range per partition, answered without reading the table. Compute all three windows in one pass over the last 90 days with conditional aggregation (FILTER (WHERE event_ts >= as_of - interval '7 days')), because the 7- and 30-day windows are subsets of the 90-day one. For a rolling time series (a 7-day value for every day), bucket events into days first, then use a window function with a RANGE frame on the day, so the work grows with the number of days, not the number of events.
Schema, partitioning and indexes
CREATE TABLE events (
user_id bigint NOT NULL,
event_ts timestamptz NOT NULL,
amount numeric(12,2) NOT NULL
) PARTITION BY RANGE (event_ts);
CREATE TABLE events_2026_09 PARTITION OF events FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
-- one partition per month
CREATE INDEX ON events (user_id, event_ts) INCLUDE (amount);
- Partitioning by month: a 90-day query touches at most 4 partitions however long history grows, and retention drops old months.
- Clustering by user: the composite index gives one user's events in time order inside each partition. In a columnar warehouse the same idea is
PARTITION BY DATE(event_ts)plus clustering or sort key onuser_id. INCLUDE (amount)makes the scan index-only: Postgres answers from the index without fetching table rows, as long as autovacuum (Postgres's background process that runs VACUUM to reclaim space and keep this bookkeeping current) keeps the table's visibility map (a per-table bitmap marking which pages hold only rows every transaction can see) current.- No other secondary indexes on this path. Add one only for a different access pattern, such as
(event_type, event_ts)for per-type analytics.
The query patterns, runnable
This script builds five monthly partitions, gives user 42 exactly one 10.00 event per day from 1 May to 29 September 2026, adds noise rows for 1,999 other users, and computes the windows as of 30 September 00:00 UTC. The as-of time is a literal so the output is reproducible; in production it is now() or a parameter. Run it with psql -f on PostgreSQL 16:
SET TIME ZONE 'UTC';
DROP TABLE IF EXISTS events CASCADE;
CREATE TABLE events (
user_id bigint NOT NULL,
event_ts timestamptz NOT NULL,
amount numeric(12,2) NOT NULL
) PARTITION BY RANGE (event_ts);
CREATE TABLE events_2026_05 PARTITION OF events FOR VALUES FROM ('2026-05-01') TO ('2026-06-01');
CREATE TABLE events_2026_06 PARTITION OF events FOR VALUES FROM ('2026-06-01') TO ('2026-07-01');
CREATE TABLE events_2026_07 PARTITION OF events FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');
CREATE TABLE events_2026_08 PARTITION OF events FOR VALUES FROM ('2026-08-01') TO ('2026-09-01');
CREATE TABLE events_2026_09 PARTITION OF events FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE INDEX ON events (user_id, event_ts) INCLUDE (amount);
-- user 42: one 10.00 purchase per day from 2026-05-01 to 2026-09-29; other users as noise
INSERT INTO events
SELECT 42, timestamptz '2026-05-01 12:00' + d * interval '1 day', 10.00
FROM generate_series(0, 151) AS d;
INSERT INTO events
SELECT u, timestamptz '2026-05-01 00:00' + (u * 37 + d) % 150 * interval '1 day', 1.00
FROM generate_series(1, 2000) AS u, generate_series(0, 19) AS d WHERE u <> 42;
ANALYZE events;
\echo one pass, three windows, as of 2026-09-30 00:00 UTC
SELECT
sum(amount) FILTER (WHERE event_ts >= timestamptz '2026-09-30' - interval '7 days') AS sum_7d,
sum(amount) FILTER (WHERE event_ts >= timestamptz '2026-09-30' - interval '30 days') AS sum_30d,
sum(amount) AS sum_90d,
count(*) AS events_90d
FROM events
WHERE user_id = 42
AND event_ts >= timestamptz '2026-09-30' - interval '90 days'
AND event_ts < timestamptz '2026-09-30';
\echo rolling 7-day sum per day, last 3 days
SELECT day, rolling_7d FROM (
SELECT day,
sum(daily) OVER (ORDER BY day RANGE BETWEEN interval '6 days' PRECEDING AND CURRENT ROW) AS rolling_7d
FROM (
SELECT date_trunc('day', event_ts) AS day, sum(amount) AS daily
FROM events
WHERE user_id = 42
AND event_ts >= timestamptz '2026-09-30' - interval '96 days'
AND event_ts < timestamptz '2026-09-30'
GROUP BY 1
) d
) w ORDER BY day DESC LIMIT 3;
\echo plan
EXPLAIN (COSTS OFF)
SELECT sum(amount) FROM events
WHERE user_id = 42 AND event_ts >= timestamptz '2026-09-30' - interval '90 days' AND event_ts < timestamptz '2026-09-30';
Output:
one pass, three windows, as of 2026-09-30 00:00 UTC
sum_7d | sum_30d | sum_90d | events_90d
--------+---------+---------+------------
70.00 | 300.00 | 900.00 | 90
(1 row)
rolling 7-day sum per day, last 3 days
day | rolling_7d
------------------------+------------
2026-09-29 00:00:00+00 | 70.00
2026-09-28 00:00:00+00 | 70.00
2026-09-27 00:00:00+00 | 70.00
(3 rows)
plan
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate
-> Append
Subplans Removed: 2
-> Index Only Scan using events_2026_07_user_id_event_ts_amount_idx on events_2026_07 events_1
Index Cond: ((user_id = 42) AND (event_ts >= ('2026-09-30 00:00:00+00'::timestamp with time zone - '90 days'::interval)) AND (event_ts < '2026-09-30 00:00:00+00'::timestamp with time zone))
-> Index Only Scan using events_2026_08_user_id_event_ts_amount_idx on events_2026_08 events_2
Index Cond: ((user_id = 42) AND (event_ts >= ('2026-09-30 00:00:00+00'::timestamp with time zone - '90 days'::interval)) AND (event_ts < '2026-09-30 00:00:00+00'::timestamp with time zone))
-> Index Only Scan using events_2026_09_user_id_event_ts_amount_idx on events_2026_09 events_3
Index Cond: ((user_id = 42) AND (event_ts >= ('2026-09-30 00:00:00+00'::timestamp with time zone - '90 days'::interval)) AND (event_ts < '2026-09-30 00:00:00+00'::timestamp with time zone))
(9 rows)
Check the numbers by hand: 90 days before 30 September is 2 July, so the window holds 2 July to 29 September, which is 90 daily events of 10.00 = 900.00. The last 30 days hold 30 events (300.00) and the last 7 hold 7 (70.00). The plan shows the May and June partitions removed ("Subplans Removed: 2") and an index-only scan on the other three.
Why these patterns
- One pass, three windows: three separate queries would read the last 7 days three times and the last 30 twice.
FILTERevaluates all three sums while scanning the 90-day range once. - Bucket, then window:
RANGE BETWEEN interval '6 days' PRECEDING AND CURRENT ROWmeans "rows whose day is within 6 days before this one", which is correct even when a user has no events on some days. AROWS BETWEEN 6 PRECEDINGframe would silently span more than 7 calendar days whenever days are missing. - Scanning 96 days for a 90-day series: the first output day needs 6 days of history before it, otherwise its 7-day value is truncated.
Keeping it fast as the table grows
Cost per query is proportional to one user's events in the window, not to table size, because the index range and partition pruning bound it. Worked example: a heavy user with 200 events/day has 18,000 events in 90 days; that is about 18,000 index entries across 4 partitions, which is a small read. The worst case is a bot account with millions of events: add a per-user daily cap at ingest, or accept that such users need a pre-aggregated path.
Two things silently break it:
- Stable functions and pruning.
now()is a stable function (a Postgres function guaranteed to return the same value for every row within one query, but not a fixed literal Postgres already knows ahead of time), sonow() - interval '90 days'is not a constant at planning time (the step where Postgres works out, before running the query, which partitions could possibly match). Instead it prunes partitions (skips the ones that cannot match) at executor start (the moment it actually begins running the query, once "now" is known); that is still effective, as the plan shows, it just happens a beat later than pruning on a literal timestamp would. Wrapping the column, as inWHERE date(event_ts) >= ..., defeats both pruning and the index; always compare the raw column to a computed bound. - Index-only scans need a current visibility map. On a table with heavy updates or deletes, the scan falls back to fetching rows from the table itself (shown in
EXPLAINoutput as heap fetches) instead of answering from the index alone. Events are insert-only, so this holds if autovacuum keeps up.
Trade-offs and pitfalls
- Time zones:
date_trunc('day', event_ts)uses the session time zone. Set it explicitly (the script uses UTC), or compute days in the user's own zone withAT TIME ZONE. - Window boundaries: decide whether "last 7 days" means the last 168 hours (as here) or the last 7 calendar days including today; they differ by up to a day of events.
- Batch many users: for a feature pipeline scoring millions of users, the same
FILTERquery withGROUP BY user_idover one 90-day scan is better than millions of index lookups. - What would change the design: if the SLO (service-level objective) is single-digit milliseconds at thousands of requests per second, the "no rollups" constraint is worth challenging, because incrementally maintained daily per-user totals (a background job adds each day's events to a running per-user-per-day row as they arrive, instead of a request recomputing sums from scratch) turn each request into reading at most 90 small rows.
Unlock Full Question Bank
Get access to all 12 Geospatial and Time-Series Data interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.