Airbnb-Specific Data Patterns Questions
Domain-specific data modeling and analytics patterns used in Airbnb-scale product analytics. Covers data schema design, event and transaction patterns, feature engineering templates for predictive models, cohort and lifecycle analytics, geospatial and temporal data patterns, price and demand forecasting signals, AB testing data patterns, and data quality, governance, and lineage considerations relevant to Airbnb data.
EasyTechnical
76 practiced
Write SQL (Postgres/BigQuery) to sessionize user events into sessions using a 30-minute inactivity gap. Given table events(user_id STRING, occurred_at TIMESTAMP, event_type STRING), produce: session_id, user_id, session_start, session_end, event_count. Use window functions and deterministic session_id creation.
Sample Answer
Approach: for each user order events by time, compute time difference to previous event; mark a new session when gap > 30 minutes (or first event). Cumulative sum of session boundaries yields a session ordinal per user. Group by user + session ordinal to get session_start/end/count and create a deterministic session_id (e.g., md5(user_id || session_start)).Key points:- Uses LAG to compute inactivity gaps and a cumulative SUM to assign sessions.- Deterministic session_id built from user_id + session_start then hashed.Edge cases:- Events with identical timestamps; first event per user handled by IS NULL check.- Timezone considerations: ensure occurred_at is in a consistent timezone or use TIMESTAMP WITH TIME ZONE.Alternative: BigQuery-specific functions (TIMESTAMP_DIFF / FORMAT_TIMESTAMP) can replace EXTRACT/TO_CHAR if preferred.
sql
WITH ordered AS (
SELECT
user_id,
occurred_at,
event_type,
-- time (in seconds) since previous event for same user
EXTRACT(EPOCH FROM (occurred_at - LAG(occurred_at) OVER (PARTITION BY user_id ORDER BY occurred_at))) AS seconds_since_prev
FROM events
),
boundaries AS (
SELECT
user_id,
occurred_at,
-- start a new session when no previous event OR gap > 30 minutes (1800s)
CASE WHEN seconds_since_prev IS NULL OR seconds_since_prev > 1800 THEN 1 ELSE 0 END AS is_new_session
FROM ordered
),
sessionized AS (
SELECT
user_id,
occurred_at,
-- cumulative session number per user
SUM(is_new_session) OVER (PARTITION BY user_id ORDER BY occurred_at ROWS UNBOUNDED PRECEDING) AS session_num
FROM boundaries
)
SELECT
-- deterministic session id: md5 of user_id + session_start (ISO)
md5(user_id || '|' || TO_CHAR(MIN(occurred_at) OVER (PARTITION BY user_id, session_num), 'YYYY-MM-DD"T"HH24:MI:SS')) AS session_id,
user_id,
MIN(occurred_at) AS session_start,
MAX(occurred_at) AS session_end,
COUNT(*) AS event_count
FROM sessionized
GROUP BY user_id, session_num
ORDER BY user_id, session_start;MediumSystem Design
62 practiced
Design an event ingestion pipeline that can handle up to 1M events/sec globally with end-to-end ingestion latency under 5 seconds for analytics. Specify components (client SDK, edge ingestion, message broker, stream processing, schema registry, storage), how you'd handle schema evolution, idempotency, backpressure, and region-aware routing. What trade-offs would you make to control cost and ensure reliability?
Sample Answer
Requirements & constraints clarified:- 1M events/sec global, <5s end-to-end for analytics.- Multi-region, cost-sensitive, reliable, supports schema evolution & idempotency.High-level architecture (flow):Clients (mobile/web/servers) → Edge ingestion (regional collectors + CDN) → Regional message broker (Kafka cluster / MSK/Confluent Cloud) → Stream processing (Flink/Spark Structured Streaming) → Serving/analytics storage (hot: ClickHouse/Druid/Kafka Topics→materialized views; cold: S3 data lake/Parquet + Hive metastore) → Downstream consumers/warehouse (BigQuery/Redshift/Snowflake) Supporting: Schema Registry (Confluent/Apicurio, Avro/Protobuf/JSON Schema), Observability (Prometheus, Grafana, OpenTelemetry), IAM/ACLs.Components & responsibilities:- Client SDK: small batch + local buffering, exponential backoff, event ID generation (UUID + clientId + seq), optional compression, auth tokens. Send to nearest edge endpoint (geo-DNS/anycast).- Edge Ingestion: lightweight autoscaling collectors (Kubernetes + HPA) or CDN-backed HTTP ingestion (CloudFront + Lambda@Edge) to absorb spikes, perform auth, lightweight validation, attach ingestion metadata, forward to regional Kafka.- Message Broker: Regional Kafka clusters for low latency. Partitioning by customer/tenant to balance load. Use topic per logical stream, retention short for hot processing, tiered storage to S3 for cost.- Stream Processing: Flink for low-latency exactly-once processing (Kafka transactions/stateful checkpoints). Enrich, filter, deduplicate, write to hot stores and batch sinks.- Schema Registry: Central registry replicated across regions; enforce forward/backward compatibility; use semantic versioning and compatibility checks on deployment.- Storage: Hot OLAP (Druid/ClickHouse) for sub-second queries; long-term Parquet on S3 for analytics and reprocessing.Schema evolution:- Use Avro/Protobuf with Schema Registry. Enforce compatibility rules (default: backward compatible for producers; producers can add optional fields). For breaking changes, use versioned topics or transformation layer in stream processing. Provide SDK that can handle missing/new fields gracefully (defaults).Idempotency & deduplication:- Clients attach deterministic event_id and ingestion_time. At edge, lightweight dedupe cache (TTL 1–5 min) to reduce obvious duplicates. At stream processor, use Flink's stateful dedupe keyed by event_id with TTL to guarantee one-time downstream side-effects. For sinks supporting Kafka transactions or idempotent writes (e.g., transactional writes to Kafka / upserts to DB), leverage exactly-once semantics.Backpressure & throttling:- Server-side: per-tenant rate limits at edge collectors; return 429 with Retry-After. Edge collectors buffer to Kafka with local backpressure when brokers are slow. Use circuit breakers and secondary spillover (S3 upload of raw events for later replay) when sustained overload.- Client-side SDK: configurable max buffer, flush interval, exponential backoff on 429s or connection issues.Region-aware routing & replication:- Geo-aware DNS/anycast sends clients to nearest edge. In-region Kafka clusters handle regional traffic; critical data replicated asynchronously via MirrorMaker 2 or Kafka-tiered-storage-based cross-region replication to a global Kafka or central lake. For low-latency analytics keep regional processing local; run global aggregation jobs that consume mirrored topics for global views. For GDPR/data residency, route and retain data per region.Trade-offs (cost vs reliability & latency):- Keep hot storage (fast OLAP) sized for recent window (hours/days) and offload older data to cheap S3 Parquet to reduce cost. This adds reprocessing cost when querying old data but saves run cost.- Use managed Kafka (MSK/Confluent) to reduce ops cost vs self-managed clusters (higher control). Managed reduces engineering overhead but increases recurring cost.- Exactly-once processing (Flink + Kafka transactions) adds compute and complexity; choose idempotent-at-least-once with dedupe for lower cost if acceptable.- Replication: synchronous cross-region replication guarantees consistency but multiplies cost and latency. Prefer asynchronous replication + regional processing for latency and cost.Operational considerations:- Autoscaling, IaC, chaos testing, SLOs (ingestion success, tail latency), alerting for broker lag and processing backpressure.- Runbooks for broker outages: failover to spill-to-S3 + replay.- Data governance: auditing, lineage in schema registry, retention policies.Why this design:- Regional Kafka + edge ingestion minimizes client-to-broker latency and absorbs spikes; Flink provides low-latency stateful processing and strong semantics; tiered storage keeps costs controlled while enabling reprocessability. The pattern balances <5s ingestion for analytics, high throughput, and operational cost/control.
EasyTechnical
75 practiced
What is a star schema and why is it commonly used for product analytics at Airbnb? Describe the fact table(s) and at least three dimension tables you would create for bookings analytics. For each table, give sample columns and typical join keys and explain how this model supports analytic queries.
Sample Answer
A star schema is a dimensional model with a central fact table (measures/events) joined to denormalized dimension tables (context). It’s widely used for product analytics because it simplifies SQL, speeds OLAP queries (wide, predictable joins), and maps directly to business metrics — ideal for Airbnb’s event- and booking-focused analyses.Core fact table(s)- bookings_fact - Sample columns: booking_id (PK), user_id, listing_id, host_id, checkin_date, checkout_date, booking_timestamp, nights, guests, price_total, commission, cancel_flag - Join keys: user_id → dim_user.user_id, listing_id → dim_listing.listing_id, host_id → dim_host.host_id - Purpose: stores one row per confirmed booking with numeric measures for aggregation.Optional supporting fact tables: booking_events_fact (search → click → booking funnel), payouts_fact (host payout flows) — same pattern, keyed to booking_id or event_id.Dimension tables (examples)1. dim_user - Columns: user_id (PK), signup_date, country, city, user_tier, signup_source - Join key: user_id - Use: segment bookings by geography, cohort, acquisition channel.2. dim_listing - Columns: listing_id (PK), host_id, room_type, city, neighborhood, capacity, created_at, amenity_flags - Join key: listing_id - Use: filter/aggregate by property attributes, city, neighborhood.3. dim_host - Columns: host_id (PK), host_since, total_listings, response_rate, is_superhost, region - Join key: host_id - Use: analyze host-level economics, compare superhosts vs others.4. dim_date - Columns: date (PK), year, month, week, day_of_week, is_holiday - Join key: checkin_date or booking_timestamp (date_key) - Use: time-series, seasonality, rolling windows.How it supports analytics- Fast aggregations: pre-joined small number of dimension tables reduces complex normalization joins.- Flexible segmentation: dimensions provide attributes for GROUP BY, WHERE, funnel breakdowns.- Easy cohort/time analysis: dim_date enables calendar-based windowing; dim_user supports retention/cohorts.- Maintainability: adding attributes (e.g., new listing features) is a schema change on dimensions without rewriting facts.This design balances query performance, clarity for analysts, and ETL simplicity for data engineering at scale.
MediumTechnical
86 practiced
Design a feature store architecture for Airbnb demand forecasting that must serve both low-latency online features and batch training features. Describe the schema for features, feature granularity (listing-day, listing-hour), freshness SLAs, batch vs streaming feature computes, TTL/retention, and how to guarantee consistency between offline training features and online serving features.
Sample Answer
Requirements & constraints (clarify):- Serve low-latency online lookups (<10–100ms) for inference, and large-scale batch feature reads for nightly retraining.- Support two granularities: listing-hour (hourly features for near-term demand) and listing-day (daily horizon features for long-term forecasts).- Maintain reproducible offline training datasets that match online-serving values.High-level architecture:- Ingestion: events (views, searches, bookings) → streaming pipeline (Kafka) and persistent event lake (S3/GCS).- Feature compute: - Streaming processors (Flink/Beam) for near-real-time aggregations / counters. - Batch jobs (Spark) for heavy aggregations, windowed historical joins, and expensive offline features.- Stores: - Offline feature store: columnar tables in data lake / warehouse (Parquet on S3, exposed via BigQuery/Snowflake). - Online feature store: low-latency key-value DB (DynamoDB / CockroachDB / Redis + persistent DB) for inference.- Metadata & governance: feature registry + schema registry, lineage, ownership, tests.- Serving API: read API backed by online store, fallback to lightweight compute or cold store for missing features.Feature schema (canonical per feature):- feature_id: string- entity_key: {listing_id} (compound if needed: listing_id, country)- granularity: enum {listing-hour, listing-day}- event_time: timestamp (event time)- created_at / ingestion_time: timestamp- value: typed (float/int/boolean/categorical id)- value_type: metadata- version: semantic version for transform code- ttl_seconds: int (metadata)- provenance: job_id / commit / checksumFeature granularity:- listing-hour: one row per (listing_id, hour). Use for short-horizon demand (next 24–72h).- listing-day: one row per (listing_id, date). Use for longer-horizon forecasting.Design rules: compute and materialize features at their natural granularity. If model needs both, join at training time with consistent event_time alignment.Freshness SLAs:- Online hot features (e.g., last-24h view count): update latency < 1 minute; read latency < 10–50 ms.- Near-online features (session-level counters): update latency < 5 minutes.- Batch/slow features (30/90-day rolling averages, price elasticities): refreshed daily (batch window finishes by 02:00).- SLA examples: streaming features 99th percentile update latency <60s; online read 99th percentile <50ms.Batch vs streaming computes:- Streaming: - Use streaming pipelines for incremental counters, session metrics, real-time rates (e.g., views_per_hour, active_listings_in_area). - Materialize incremental updates to online store using upserts or KV increments. - Use event-time processing + watermarking to handle late data; emit finalizations for windows.- Batch: - Use Spark for windowed historical aggregates, complex joins (user/listing history), expensive modeling features (embedding generation). - Produce columnar offline tables (feature views) partitioned by date and granularity; export snapshots for training.TTL / retention:- Raw event logs: retain 90–365 days depending on compliance.- Offline feature tables: retain 2+ years (partitioned by date); compact older partitions.- Online store TTL: per-feature configurable. Example: short-lived dynamic features TTL = 24–48 hours; stable features (static property attributes) TTL = 365 days or no TTL.- Materialized streaming checkpoints / changelog retention: match streaming platform requirements (e.g., Kafka retention >= 7 days).Guaranteeing offline-online consistency:- Single-source-of-truth transforms: keep feature transformation code in a shared library (feature SDK) used by both batch jobs and streaming jobs. Versioned and tested.- Feature versions: store a version id in both online and offline records. Training pipeline pins to feature versions used in serving.- Materialized feature views + snapshot export: - Offline training data is produced by joining raw events to the offline feature view materialized from the same transformation code and inputs as online. - At training time, generate a training dataset by exporting point-in-time feature snapshots (time-travel or event-time joins) from the offline store. Use event_time alignment to ensure you only use feature values available at prediction time.- Export pipeline & upsert parity: - Streaming compute writes the same transformation results to the online store and appends to an append-only changelog in the data lake. Batch recomputations also write to the offline store and can reconcile with the changelog.- Point-in-time correctness: - For training, use point-in-time (PIT) join logic: for each training example timestamp, join features as-of that timestamp using the offline feature tables containing event_time and created_at.- Testing and CI: - Unit + integration tests verifying sample inputs produce identical outputs in batch vs streaming jobs. - Periodic shadow replay: replay historical data through streaming pipeline and compare outputs to batch results; flag drift.- Monitoring: - Data quality checks: drift detectors, cardinality checks, null rates. - Liveness/latency metrics for feature writes to online store and staleness monitors (how old is the last update per entity).- Fallback and determinism: - When online feature missing, model uses default or fallbacks computed from cached offline aggregates; log and alert mismatches.Example flow for a streaming feature "views_last_24h_listing-hour":- Events -> Kafka -> Flink event-time tumbling windows per listing -> compute 24h sliding count -> write to DynamoDB with key (listing_id, granularity=hour, event_time=hour_start), include version and created_at -> append same record to S3 changelog partitioned by date -> Spark nightly job compacts changelog into offline Parquet feature table.Training snapshot:- Trainer selects model time range, pulls feature snapshots from offline Parquet using point-in-time join on event_time to prevent leakage. It records the feature version and the exact materialized dataset used; ML experiments reference that snapshot id.Trade-offs:- Strong consistency (synchronous writes) increases latency; prefer at-least-once streaming writes with idempotent upserts and reconciliation jobs.- Keeping one codebase increases development velocity but requires strict CI and versioning.- Online store choice balances cost vs latency vs durability (DynamoDB/Cockroach for durability, Redis for extreme low-latency with Redis+persistent backing).Wrap-up (operational practices):- Enforce schema registry and feature ownership.- Automatic CI tests for batch vs stream parity.- SLAs, monitoring, and daily reconciliation jobs ensure consistency and reliability for demand forecasting at Airbnb scale.
MediumTechnical
79 practiced
Given listings(listing_id, lat FLOAT, lon FLOAT, city STRING) and bookings(booking_id, listing_id, occurred_at TIMESTAMP), write SQL (Postgres/PostGIS or BigQuery) to compute conversion rate per 1km grid cell within a specified city and return the top-3 hotspot cells by conversion rate, requiring at least 50 listings per cell. Describe how you generate the grid and join listings to cells.
Sample Answer
Approach: project listings into a metric CRS (EPSG:3857), build a 1km x 1km square grid over the city's bounding box, assign each listing to a cell (ST_Intersects), aggregate listings and bookings per cell, filter cells with >=50 listings, compute conversion = bookings / listings, and return top-3 by conversion rate.PostGIS SQL (Postgres):Key points / reasoning:- Projecting to EPSG:3857 gives meter units so a 1000x1000 grid equals ~1km.- Grid generated with generate_series over bbox extents, making non-overlapping square cells.- Use ST_Contains with transformed points to assign listings deterministically.- Aggregation counts distinct listings and joins bookings to avoid double counting.- Filter >=50 listings to ensure statistical reliability.Edge cases & notes:- For better accuracy use local UTM zone CRS instead of 3857 if city spans large lat range.- If listings sit on cell boundaries, choose ST_Intersects vs ST_Contains consistently.- For BigQuery GIS, use ST_MAKEPOINT/ ST_TRANSFORM/GENERATE_ARRAY similarly (BigQuery has ST_GenerateGrid in some versions) and adapt syntax.- Consider temporal window for bookings and rate smoothing (Laplace) if needed.
sql
WITH city_bbox AS (
SELECT
ST_Transform(ST_Extent(ST_SetSRID(ST_MakePoint(lon, lat), 4326)), 3857) AS geom_3857
FROM listings
WHERE city = 'TargetCity'
),
grid AS (
SELECT
ST_SetSRID(
ST_MakeBox2D(
ST_Point(x, y),
ST_Point(x + 1000, y + 1000)
), 3857
) AS cell_geom_3857,
x AS x0, y AS y0
FROM city_bbox,
generate_series(
floor(ST_XMin(city_bbox.geom_3857)/1000)*1000,
ceil(ST_XMax(city_bbox.geom_3857)/1000)*1000 - 1000,
1000
) AS gs_x(x),
generate_series(
floor(ST_YMin(city_bbox.geom_3857)/1000)*1000,
ceil(ST_YMax(city_bbox.geom_3857)/1000)*1000 - 1000,
1000
) AS gs_y(y)
),
listings_in_cells AS (
SELECT g.x0, g.y0, g.cell_geom_3857,
l.listing_id,
ST_Transform(ST_SetSRID(ST_MakePoint(l.lon, l.lat),4326),3857) AS pt
FROM grid g
JOIN listings l
ON l.city = 'TargetCity'
AND ST_Contains(g.cell_geom_3857, ST_Transform(ST_SetSRID(ST_MakePoint(l.lon, l.lat),4326),3857))
),
agg AS (
SELECT
x0, y0,
COUNT(DISTINCT listing_id) AS listings_count,
COUNT(b.booking_id) FILTER (WHERE b.booking_id IS NOT NULL) AS bookings_count
FROM listings_in_cells lic
LEFT JOIN bookings b ON b.listing_id = lic.listing_id
GROUP BY x0, y0
)
SELECT
ST_Transform(ST_Centroid(cell_geom_3857), 4326) AS cell_centroid_wgs84,
listings_count,
bookings_count,
(bookings_count::float / NULLIF(listings_count,0)) AS conversion_rate
FROM agg
JOIN grid USING (x0,y0)
WHERE listings_count >= 50
ORDER BY conversion_rate DESC
LIMIT 3;Unlock Full Question Bank
Get access to hundreds of Airbnb-Specific Data Patterns interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.