Ride-Hailing Data Modeling & Analytics Requirements Questions
Data modeling and analytics requirements for ride-hailing and mobility-marketplace platforms, including ride event data, trip-level schemas, driver and rider dimensions, pricing and surge data, and geospatial/location data. Covers analytics needs such as reporting, dashboards, and real-time analytics: analytic schema design (star/snowflake), ETL/ELT patterns, data quality and governance at scale, data lineage, privacy considerations, and integration with the broader data stack (data lake/warehouse, streaming pipelines).
EasyTechnical
130 practiced
Explain H3 hexagonal geospatial indexing and how it can be used to aggregate rides by geographic area for Lyft. Describe how to choose resolution levels for city-level vs neighborhood-level analytics and the trade-offs between precision and storage/query performance.
Sample Answer
H3 is a hierarchical hexagonal geospatial indexing system (Uber open-source) that maps lat/lon to compact integer cell IDs at discrete resolutions. Hexes avoid edge/corner bias of squares and support fast aggregation via indexed cell IDs and parent/child relationships.How to use for Lyft ride aggregation:- During ETL, convert pickup/dropoff lat/lon to H3 cell IDs at a chosen resolution (h3.geo_to_h3).- Store cell_id as a column (indexed/partitioned) in your event table.- Aggregate rides by cell: GROUP BY cell_id, COUNT(*), SUM(fare), etc.- For multi-scale analytics, roll up by computing parent(cell_id, k) to group to coarser areas without re-geocoding.Choosing resolutions (examples):- City-level: use low-res hexes like res 5–7 (res5 ~ 3,000 km²? approximate; res7 ~ 70 km²) to get few cells covering a metro for dashboard overviews.- Neighborhood-level: use res 9–11 (res9 ~ 2.5 km²; res11 ~ 0.09 km²) for street/neighborhood granularity and hotspots.Trade-offs:- Precision vs storage: higher resolution → more distinct cell_ids → larger dimension cardinality, larger indexes, and bigger storage for materialized aggregates.- Query performance: coarse cells reduce GROUP BY cardinality and speed queries; fine cells increase compute and shuffle in distributed engines (Spark), but allow precise spatial joins.- Pre-aggregation: maintain both resolutions (e.g., nightly rollups to res7 and res10) to serve fast dashboards while keeping raw events for ad-hoc high-res analysis.- Joins and boundaries: hexes simplify joins but may split logical regions (municipal boundaries) — store mapping tables if strict administrative alignment is required.Best practice: pick one or two operational resolutions (city and neighborhood), store raw coords + base high-res H3 cell, and precompute rollups to lower resolutions to balance precision and performance.
HardTechnical
72 practiced
Design a geospatial data model to support multi-zoom-level analytics at Lyft: city-wide heatmaps, neighborhood aggregations, and route-level analysis. Include H3 indexing strategy, storage of zone geometries, pre-aggregations for common tiles, and approaches for spatial joins and indexing in warehouses that support GEOGRAPHY types.
Sample Answer
Requirements & constraints:- Support city heatmaps (many zoom levels), neighborhood aggregates, and route-level queries with low latency for exploration and batch analytics.- Scale: millions of trips/day, multi-tenant, hourly/daily rollups.H3 indexing strategy- Choose a small set of H3 resolutions that map to UI zoom levels (example mapping): - Zoom 6 → H3 res 5 (~4.3km), Zoom 10 → res 8 (~0.18km), Zoom 13 → res 9 (~0.07km).- Always compute and store multiple resolutions per geometry/event so queries can use nearest pre-aggregated resolution. For each event/trip geometry compute h3_cells = {h3_to_parent(h3(event), r) for r in prechosen_resolutions}.- Use h3-extensions in Spark (h3-java / h3-py) to compute cells during ingestion.Storage/data model (example tables)- raw_trips (parquet/Delta) - trip_id, start_ts, end_ts, driver_id, geom_line STRING/WKB, start_h3_res9, end_h3_res9, h3_sequence ARRAY<STRING>, attributes...- h3_cells_master - h3_index STRING PRIMARY KEY, res INT, geography GEOGRAPHY (cell polygon), centroid GEOGRAPHY, city_id- preagg_tile_metrics (base unit for fast heatmaps) - date_hour, h3_index (for chosen res set, e.g., res7,res8,res9), metric_counts JSON or columns (trips, pickups, avg_wait), shard_key - Partition by date_hour, cluster by h3_index (or Z-order / sort on h3_index)- neighborhoods table - neighborhood_id, name, geography GEOGRAPHY, outer_h3_indexes ARRAY<STRING> (precomputed at a chosen res)Pre-aggregations & pipelines- Real-time: streaming job (Spark Structured Streaming / Flink) consumes events, maps to h3 cells for the set of resolutions, increments streaming materialized aggregates in a fast store (Redis or Kafka + windowed aggregates), writes batched updates to preagg_tile_metrics hourly.- Batch: daily rollups from raw_trips to compute aggregated metrics at all resolutions; write compact parquet/Delta partitioned files and update preagg_tile_metrics via upsert.- Precompute common tiles (resolutions mapping to UI zooms) and store as materialized views for low-latency queries.Route-level model & analytics- Store route as polyline (WKB/GEOGRAPHY) and as sequence of h3 cells (h3_sequence). For route comparisons/heatmaps, use sequences or compute route tile counts per trip (explode h3_sequence).- Precompute route-level aggregates: route_id, frequent_h3_sequence_signature (MinHash or hashed n-grams), metrics (avg duration, congestion signals).Spatial joins & warehouse indexing- Use native GEOGRAPHY types in warehouse (BigQuery/Redshift/Snowflake/Snowflake GEOGRAPHY via GEOGRAPHY type/BigQuery GEOGRAPHY/Redshift with GEOMETRY extension).- For point-in-polygon / neighborhood joins: - Option A: Use h3 precomputed cell lists for polygons: join on h3_index IN neighborhood.outer_h3_indexes (fast equality join). - Option B: Use ST_CONTAINS(neighborhood.geography, point) using GEOGRAPHY functions—ensure spatial indexes where supported.- Indexing advice: - Postgres/PostGIS: GiST index on geography column (CREATE INDEX ON table USING GIST (geog)); - Snowflake: cluster by h3_index or use SEARCH optimization; store h3_index as varchar for clustering keys. - BigQuery: partition by date and clustering on h3_index and city_id to accelerate range scans.- For joins between large tables, prefer hash joins on h3_index (explode arrays), pushdown filters by date_hour and h3_index to prune data.Performance & storage best-practices- Store cell geometries once (h3_cells_master); reference by h3 index to avoid repeating polygons.- Use columnar storage (Parquet/ORC/Delta) for preagg_tile_metrics and raw_trips.- Materialize aggregated tiles for top N zoom levels; compute higher-res tiles on demand via rollup/downsampling using parent-child h3 relationships.- Use compaction jobs to merge small files and optimize layout; use Z-order or sort-by h3_index for locality.Edge cases & trade-offs- Choosing too many resolutions increases storage; pick small set matching UI needs and support on-the-fly aggregation for others.- Point-in-polygon with complex neighborhood boundaries: prefer h3-based cover + final ST_INTERSECTS filter to reduce expensive geometry ops.- Route sequence storage can be large: compress h3 sequences (delta-encode) or store only hotspots and frequent-pattern signatures.Example Spark pseudocode (mapping events to multi-res h3)This model gives fast, scalable heatmaps via pre-aggregated tiles, accurate neighborhood joins using both h3 covers and GEOGRAPHY functions, and route-level analytics by storing sequences + aggregated signatures.
python
from h3 import h3
def map_event_to_cells(lat, lng):
res_list = [7,8,9]
base = h3.geo_to_h3(lat,lng,9) # highest-res cell
return {r: h3.h3_to_parent(base, r) for r in res_list}MediumTechnical
131 practiced
Given schema:trips(trip_id STRING, rider_id STRING, occurred_at TIMESTAMP, fare_amount_cents INT, currency STRING, refund_cents INT)fx_rates(date DATE, currency STRING, usd_rate FLOAT)Write standard SQL (BigQuery/Snowflake) to compute each rider's 365-day lifetime value (LTV) in USD, handling refunds and missing conversion rates (use last known rate if available). Return: rider_id, ltv_usd_cents.
Sample Answer
Approach: filter trips to the past 365 days, compute net cents per trip (fare - refund), convert to USD using the fx rate for that currency on or before the trip date (if missing, use the last available rate for that currency), then sum per rider and return ltv in USD cents. I use a correlated subquery to find the most recent fx rate <= trip date and fallback to the latest overall rate if none exists.Key points / implementation notes:- Filter trips to the last 365 days with occurred_at >= CURRENT_TIMESTAMP() - INTERVAL '365' DAY.- Net cents = fare_amount_cents - COALESCE(refund_cents, 0).- Convert cents to USD cents by multiplying net_cents * usd_rate (assuming fx.usd_rate = 1 unit currency = usd_rate USD). No additional /100 needed because cents * rate yields USD-cents.- To get the rate on-or-before the trip date, pick MAX(date) <= trip date. If none exists, pick the latest available rate for that currency.- If no rate exists at all for a currency, decide whether to drop those trips or treat rate=0; above query uses a COALESCE fallback to 0 (you can change to exclude such trips by adding WHERE COALESCE(...) IS NOT NULL).- For performance at scale, precompute a rates-lookup table (currency, date, usd_rate) and join using a window (LAST_VALUE IGNORE NULLS OVER partitioned by currency ordered by date) or use a lateral join per trip partitioned by rider-batch to avoid correlated-subquery cost.Edge cases:- Refunds greater than fare (net negative) — included (reduces LTV).- Missing rates — handled via last-known fallback; adjust business rule if prefer excluding.- Timezone handling: ensure occurred_at -> DATE conversion uses the desired timezone.- Very large datasets: prefer a pre-joined daily snapshot or use analytic windows to avoid per-row correlated subqueries.
sql
-- Standard SQL (works in BigQuery / Snowflake)
SELECT
t.rider_id,
SUM(ROUND((t.fare_amount_cents - COALESCE(t.refund_cents, 0)) * usd_rate_for_trip)) AS ltv_usd_cents
FROM trips t
CROSS JOIN -- no-op; keeps query portable between engines
(
-- placeholder so correlated subqueries below can refer to this alias in some engines;
SELECT 1 AS one
) _dummy
LEFT JOIN LATERAL
(
-- determine usd_rate to use for this trip:
-- 1) most recent rate with date <= trip date
-- 2) if none, the most recent rate available for that currency
SELECT f1.usd_rate AS usd_rate_for_trip
FROM fx_rates f1
WHERE f1.currency = t.currency
AND f1.date = (
SELECT MAX(f2.date)
FROM fx_rates f2
WHERE f2.currency = t.currency
AND f2.date <= DATE(t.occurred_at)
)
LIMIT 1
) fx1 ON TRUE
LEFT JOIN LATERAL
(
-- fallback: if fx1 is null (no rate on-or-before trip), pick latest available
SELECT f3.usd_rate AS usd_rate_last_known
FROM fx_rates f3
WHERE f3.currency = t.currency
ORDER BY f3.date DESC
LIMIT 1
) fx2 ON fx1.usd_rate_for_trip IS NULL
CROSS JOIN
(
-- final USD rate, prefer fx1 then fx2; if still NULL, treat as 0 (or you may choose to exclude)
SELECT 1
) _join
-- compute usd_rate_for_trip using COALESCE of the two lateral results
-- note: use a sub-select in SELECT to coalesce the lateral joins' outputs
WHERE t.occurred_at >= CURRENT_TIMESTAMP() - INTERVAL '365' DAY
GROUP BY t.rider_id
HAVING SUM(ROUND((t.fare_amount_cents - COALESCE(t.refund_cents, 0)) *
COALESCE(
(SELECT fx1.usd_rate_for_trip FROM (SELECT 1) AS s WHERE TRUE), -- placeholder; see note
(SELECT fx2.usd_rate_last_known FROM (SELECT 1) AS s WHERE TRUE),
0
)
)) IS NOT NULL;MediumSystem Design
72 practiced
Design a streaming architecture for ingesting driver GPS pings, performing map-matching to road segments, enriching with H3 cell and zone id, and emitting enriched events to downstream Kafka topics for analytics. Discuss doing enrichment at the edge (mobile) vs centralized processing and trade-offs (latency, cost, correctness, SDK deployment).
Sample Answer
Requirements:- Ingest high-frequency GPS pings from mobile drivers (lat/lon, timestamp, heading, speed, driver_id)- Map-match each ping to a road segment, enrich with H3 cell and zone id, produce low-latency enriched events to Kafka for analytics and downstream consumers- Throughput: millions of pings/sec eventually. Latency SLO: <1s for real-time features. Reasonable correctness for analytics (tolerate occasional re-match).High-level architecture:Mobile SDK → Ingress Gateway (API Gateway / MQTT / HTTP) → Kafka (raw pings) → Stream Processing (Flink / Spark Structured Streaming / ksqlDB) → Map-matching & enrichment → Kafka (enriched topics) → Downstream consumers / OLAP / materialized viewsCore components:1. Mobile SDK: sends pings with batching, backoff, and sequence IDs.2. Ingress Gateway: handles auth, throttling, lightweight validation; writes to raw-kafka topic or cloud pub/sub.3. Stream Processor: stateful operator that performs map-matching (local road graph shard), computes H3 cell, joins zone lookup (in-memory store / RocksDB), emits enriched events to Kafka.4. Road Graph & Tiles: precomputed graph tiles (vector tiles) stored in object storage; stream processors load relevant tiles into local state.5. Lookup stores: zone id mapping by H3 cell, cached in state store with TTL and periodic refresh.6. Observability: metrics for latency, mismatch rate, backfills for reprocessing.Data flow & scaling:- Partition raw Kafka by driver_id or H3 cell to keep ordering and locality.- Stateful Flink operators use keyed state (driver_id) and RocksDB for map-matching history.- Scale by adding operator parallelism; road tile cache sharding by region.Edge (mobile) vs Centralized enrichment — trade-offsDo enrichment at the edge (mobile SDK):- Pros: Lowest tail latency; reduces upstream bandwidth by sending only enriched events; offline resilience if connectivity intermittent.- Cons: SDK complexity (map-matching library, H3), heavier CPU/battery cost, harder to update logic (SDK releases), correctness issues due to inconsistent tile data, variable device capabilities, security/trust concerns (clients can be tampered), telemetry fragmentation.Do enrichment centrally:- Pros: Single source of truth for map tiles and algorithms; easier to iterate, test, and roll out improvements; stronger correctness and reproducibility; simpler SDK (just raw pings), centralized observability.- Cons: Higher ingress bandwidth and higher compute cost in centralized cluster; slightly higher end-to-end latency (network + processing); needs careful partitioning and caching to meet SLOs.Hybrid approach (recommended):- Do lightweight enrichment at edge: compute H3 cell and basic filtering (dedupe, sampling) to reduce volume and provide coarse low-latency hints.- Do authoritative map-matching and zone resolution centrally: run robust, stateful map-matching in stream processors that can use full road graph and historical context.- Benefits: balance latency and correctness; reduce central compute and bandwidth while retaining authoritative processing and easier updates.Operational concerns and best practices:- Versioning: include schema and algorithm version in events so downstream can differentiate edge vs central enrichments.- Backfill & reprocessing: store raw-pings topic with long retention for re-processing when map-matching improves.- SDK deployment: prefer config-driven behavior (enable/disable local enrich) and remote feature flags to toggle functionality without app release.- Consistency: use monotonic sequence IDs and server-side reconciliation jobs to detect and fix mismatches.- Cost optimization: cache road tiles per operator, use region-based partitions, autoscale processing based on traffic patterns.This design prioritizes correctness and maintainability while allowing low-latency hints from the edge.
MediumTechnical
66 practiced
Describe best practices for managing schema evolution for Avro/Parquet events with a schema registry. Provide examples of backward-compatible and incompatible changes, and explain how to coordinate producer and consumer deployments, compatibility checks, and testing to avoid breaking downstream analytics.
Sample Answer
Best practices for managing Avro/Parquet schema evolution with a schema registry- Define compatibility policy up front: choose BACKWARD, FORWARD, or FULL depending on consumers. For analytics pipelines BACKWARD is common (new producers can add fields with defaults; old consumers still read).- Use the schema registry to enforce compatibility on schema registration (CI job that registers against a sandbox or latest subject).- Prefer additive changes: add optional fields with sensible defaults (or union with null) and document semantics.- Avoid renaming or changing field types without migration; those are typically incompatible.Examples:- Backward-compatible: add field "user_tier" with type ["null","string"] and default "free". Existing readers ignore it.- Incompatible: rename "userId"->"id"; change "timestamp" from long to string; remove a required field with no default.Coordination and deployments:- Use producer feature flags and dual-write or versioned topics during migration.- Deploy consumer changes first if they require new fields (if using FORWARD compatibility) or deploy producers first if adding optional fields under BACKWARD.- Maintain semantic versioning for schemas and announce breaking changes on a schedule.Compatibility checks & testing:- Automate schema validation in CI: run registry compatibility check, run serialization/deserialization roundtrips.- Create integration tests that run consumers against a small dataset produced with the new schema (use a sandbox Kafka or S3 with Parquet files).- Smoke-test downstream analytics: run representative DAGs/SQL queries on test data and compare row counts, null rates, and key KPIs.- Monitor production after rollout: schema drift alerts, consumer errors, increased parse failures, and data quality metrics.- Maintain a migration playbook: rollback steps, data reprocessing plan, and communication templates for stakeholders.Following these practices minimizes surprises and ensures analytics remain stable during schema evolution.
Unlock Full Question Bank
Get access to hundreds of Ride-Hailing Data Modeling & Analytics Requirements interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.