Cloud Data Warehouse Architecture Questions
Understand modern cloud data platforms: Snowflake, BigQuery, Redshift, Azure Synapse. Know their architecture, scalability models, performance characteristics, and cost optimization strategies. Discuss separation of compute and storage, time travel, and zero-copy cloning.
EasyTechnical
27 practiced
Describe zero-copy cloning in Snowflake. As a data engineer, explain how it works (metadata pointers vs physical copies), common uses (dev/test, branching), and potential pitfalls related to long-lived clones and clone churn on storage costs.
Sample Answer
Situation: As a data engineer I often use Snowflake cloning to create fast, low-cost copies of datasets for dev, QA, and experiments. Here’s how to explain it technically, when to use it, and what to watch for.How it works:- Zero-copy cloning creates a new logical object (database/schema/table) that initially points to the same underlying micro-partitions and metadata as the source — no immediate physical data copy. Snowflake records metadata pointers to existing storage and maintains reference counts.- When you modify the clone or the source (DML, ALTER), Snowflake uses copy-on-write: only the changed micro-partitions are written as new physical storage; unchanged data remains shared.Common uses:- Dev/test environments: spin up isolated copies instantly without copying TBs of data.- Branching for experiments or ETL development: try schema changes or transformations safely.- Data snapshots for analysis or reproducible experimentation; combine with Time Travel to restore states.Pitfalls & cost considerations:- Clones are cheap/instant initially, but storage persists while any clone or source references micro-partitions. Long-lived clones prevent Snowflake from reclaiming older micro-partitions, increasing storage usage and costs.- Frequent clone/delete churn can create many small deltas (copy-on-write writes) and spike storage usage.- Time Travel and Fail-safe retention windows extend how long data is retained (and billed).Best practices:- Use transient or temporary tables/databases where appropriate.- Keep clones short-lived; delete when done.- Monitor DATABASE_STORAGE_USAGE and account usage views; track retention settings.- Consolidate tests or use smaller subsets for dev to minimize storage growth.
HardTechnical
48 practiced
Explain the internal storage/execution differences between Snowflake micro-partitions, Parquet/ORC columnar files, and Dremel's execution model (BigQuery). As a data engineer, say how these affect predicate pruning, columnar reads, and per-query bytes scanned.
Sample Answer
High-level summary: all three systems are columnar-oriented to reduce IO, but they differ in physical layout, metadata granularity, and execution that determine how well predicate pruning, column reads, and “bytes scanned” behave.Snowflake micro-partitions- Storage: immutable compressed columnar micro-partitions (~16–128MB logical), each stores per-column columnar data plus rich metadata: min/max, null count, distinct count, clustering keys, and bloom-filter-like stats.- Execution & pruning: query engine evaluates predicates against micro-partition metadata to prune entire micro-partitions cheaply. This gives very effective partition elimination even without explicit partition keys.- Column reads & bytes: only needed columns from surviving micro-partitions are read; because pruning happens at micro-partition granularity, per-query bytes scanned can be much smaller than raw table size when data is well-clustered.Parquet / ORC files- Storage: file-based columnar formats; within each file there are row groups/stripes (e.g., Parquet row groups) storing column chunks and statistics (min/max, nulls, sometimes bloom filters).- Execution & pruning: pruning happens at file and row-group level; effectiveness depends on how data was written (file size, sort/clustering). If files are large and unsorted, pruning is coarse and less effective.- Column reads & bytes: readers can skip non-requested columns and skip whole row groups, but may still scan larger files if row-group statistics are weak; per-query bytes scanned vary with layout and writer choices (small sorted files → better pruning but more files to manage).Dremel / BigQuery execution model- Storage & layout: BigQuery uses a columnar storage with a hierarchical column chunk layout and per-column encodings plus extensive metadata; execution uses a tree-structured, dispatch-parallel model inspired by Dremel.- Execution & pruning: predicate pushdown and column pruning occur; BigQuery also tracks per-column zone maps and uses sharding/clustering metadata for pruning. The distributed execution (multi-level aggregation/shuffle) enables fast projection of small subsets of columns across massive datasets.- Column reads & bytes: because of aggressive column projection and storage-level pruning, BigQuery reports bytes scanned per query based on what physical column chunks were accessed; clustering improves pruning and reduces bytes scanned.Practical implications for a data engineer- Clustering/sorting matters: all systems are far more efficient if data is clustered on predicate columns. Snowflake’s micro-partitioning gives automatic lightweight pruning even without explicit partition columns; Parquet/ORC rely on writer-side choices (row-group size, sort, partitioning); BigQuery benefits from clustering and partitioning.- Granularity trade-offs: smaller partitions/row-groups improve pruning but increase metadata and potential read overhead; aim for balanced file/row-group sizes (e.g., 64–256MB) and sort on high-selectivity columns.- Cost visibility: bytes-scanned metrics depend on reported accessed column chunks/micro-partitions; engineer choices (clustering, compaction, file sizing) directly affect downstream query cost and latency.- Operational tips: compact small files, choose appropriate row-group sizes, implement clustering keys where supported, and test predicate-selectivity to measure real bytes scanned on representative queries.
EasyTechnical
22 practiced
Write a SQL MERGE statement in BigQuery (Standard SQL) to perform an upsert from a staging table staging.events into analytics.events, deduplicating on event_id and choosing latest event_ts if duplicates exist. Provide assumptions about partitions and constraints.
Sample Answer
Approach: Use BigQuery MERGE to upsert staging.events into analytics.events. Pre-aggregate staging to deduplicate by event_id keeping the row with the latest event_ts (and tie-breaker if needed). Then MERGE: WHEN MATCHED -> UPDATE if staging newer; WHEN NOT MATCHED -> INSERT.Key points and assumptions:- analytics.events primary key is event_id (enforced by uniqueness via query patterns; BigQuery has no PK constraints natively).- Tables partitioned by DATE(event_ts) for analytics.events; clustering on event_id improves merge performance.- staging.events may contain duplicates; deduping CTE uses latest event_ts and ingestion_id as tie-breaker.- Ensure permissions and reservation slots for MERGE; large merges may be expensive—consider batching by partition (e.g., date) for scale.- Edge cases: null event_ts, timezone consistency—normalize timestamps before merge.
sql
-- Deduplicate staging by event_id keeping latest event_ts
WITH staged_dedup AS (
SELECT
ARRAY_AGG(s ORDER BY event_ts DESC, ingestion_id DESC LIMIT 1)[OFFSET(0)] AS row
FROM staging.events s
GROUP BY s.event_id
),
src AS (
SELECT
row.event_id,
row.event_ts,
row.user_id,
row.event_type,
row.payload
FROM staged_dedup
)
MERGE INTO `analytics.events` T
USING src S
ON T.event_id = S.event_id
WHEN MATCHED AND S.event_ts > T.event_ts THEN
UPDATE SET
event_ts = S.event_ts,
user_id = S.user_id,
event_type = S.event_type,
payload = S.payload,
updated_at = CURRENT_TIMESTAMP()
WHEN NOT MATCHED BY TARGET THEN
INSERT(event_id, event_ts, user_id, event_type, payload, created_at, updated_at)
VALUES(S.event_id, S.event_ts, S.user_id, S.event_type, S.payload, CURRENT_TIMESTAMP(), CURRENT_TIMESTAMP());HardSystem Design
22 practiced
Design a unified lakehouse architecture that integrates object storage (S3/GCS) with a cloud warehouse (e.g., Snowflake external tables or BigQuery federated sources). Address metadata management, format selection, performance for interactive queries at 100 TB, and governance.
Sample Answer
Requirements & constraints:- Store 100+ TB on object storage (S3/GCS), support interactive SQL (sub-second–few seconds / tens of seconds) for analysts, native integration with cloud warehouse (Snowflake/BigQuery) via external tables/federated sources, ACID-ish semantics for ETL, strong metadata, access control, lineage and cost control.High-level architecture:Object store (S3/GCS) ← raw + curated zones (partitioned, versioned) Metadata + table format layer: Apache Iceberg / Delta Lake (on object store) Compute: Batch ETL (Spark/Dataproc/EMR) writes Iceberg/Delta files; Query engines: Snowflake external tables or BigQuery federated + Presto/Trino or warehouse compute for heavy joins. Data catalog & governance: central metastore + Cloud Data Catalog (or AWS Glue) + IAM + Ranger-like policies. Monitoring & lineage: OpenLineage/Marquez.Key components & responsibilities:1. Table format (Iceberg preferred): native metadata manifests, snapshot isolation, partition evolution, efficient metadata pruning, time travel. Supports both object store and external table connectors.2. Metadata service: store table metadata in Iceberg manifests on S3 and register in Glue/Data Catalog; push lightweight catalog entries to Snowflake/BigQuery external-table definitions (automated).3. Partitioning & clustering: partition by date + high-cardinality bucketing/zone maps (Iceberg’s data files + min/max) to reduce IO.4. File format: Parquet (columnar, predicate pushdown), ORC as alternative. Use Parquet with dictionary encoding, snappy or zstd.5. Compaction & small-file handling: background compaction jobs to merge small files; use size targets (256MB–1GB) for optimal read throughput.6. Query performance at 100 TB: - Predicate pushdown and column projection via Parquet + Iceberg metadata reduces scanned bytes. - Use file-level statistics, partition pruning, and manifest lists to limit metadata scanning. - Maintain materialized views / pre-aggregations for interactive dashboards; refresh incrementally. - Caching: warehouse external table caching (Snowflake external table local cache) or a read-cache layer (Alluxio) for hot data. - Use adaptive join strategies and pushdown filters to compute engines; for repeated heavy queries, replicate curated tables inside Snowflake / BigQuery as native tables.7. Integration patterns: - Preferred: register Iceberg tables and let compute engines read directly; synchronize external-table definitions automatically. - Federated fallback: BigQuery federation reads Parquet/ORC directly but expect higher latency; use for ad-hoc exploration.8. Governance: - Access control: IAM at bucket + table-level policies via catalog (Glue/Dataplex) and in-warehouse RBAC. - Data lineage: capture lineage on ETL jobs and query runs with OpenLineage. - Data quality: enforce schema evolution rules, use pre-commit checks, unit/integration tests, and CI. - Encryption & retention: server-side encryption, lifecycle policies, and GDPR masking through tokenization or column-level encryption.9. Observability & cost control: - Monitor bytes scanned, query latency, and compute cost; autoscale compute; alert on runaway queries; enforce query limits.Trade-offs:- Iceberg/Delta adds metadata complexity but gives performance and safe evolution; replicating data into warehouse yields best latency but increases storage/cost. Use hybrid: external for most, native copies for hot, SLA-driven workloads.This design balances low-cost scalable object storage with warehouse capabilities, offering fast interactive queries through metadata-aware formats, caching/materialized views, and strict governance.
HardTechnical
24 practiced
Design an analytics schema for high-cardinality user event data to support fast sessionization queries and GDPR right-to-be-forgotten deletions. Discuss how you would structure facts and dimensions, indexing/clustering, and approaches to delete or mask user data efficiently.
Sample Answer
Situation & goal: design a schema for very-high-cardinality user events (millions+ distinct users/day) that supports fast sessionization queries (group events into sessions) and efficient GDPR right-to-be-forgotten deletions.Schema overview- Events fact (append-only, time-ordered): event_id (uuid), event_ts (timestamp), user_key (sharded surrogate or hashed_user_id), event_type, event_props (json), device_id, anon_user_id, session_id (nullable), ingestion_batch_id- User dimension (PII separated): user_id (PK), user_hash (one-way), pii_encrypted/json, consent_flags, deleted_at (nullable)- Lookup dimension(s): device, geo, app_version (low-cardinality)Why separate PII: keep event facts mostly non-PII so deletes/masking operate on a small table; joining at query time when allowed.Physical layout & indexing- Partition events by event_date (day) or ingestion_date for pruning.- Cluster/Sort by (user_key, event_ts) within partitions to speed sessionization windowing. On BigQuery: partition by date + clustered by user_hash,event_ts. On Snowflake: rely on micro-partitioning + clustering key. On Delta/Hudi/Iceberg: Z-order or sort by user_key,event_ts.- Secondary indexes: if OLAP store supports it, create index on ingestion_batch_id and session_id for quick re-computation.Sessionization approach- Use per-user ordered processing: window functions over partition by user_key order by event_ts to compute gaps > X minutes => session_id. Implement in Spark/Beam for batch or streaming (stateful keyed processing by user_key).- For fast ad-hoc: maintain a materialized sessions table (session facts) produced by streaming/batch job that groups events into sessions and stores session_id, user_key, start_ts, end_ts, event_count, derived metrics. Query session-level table instead of recomputing.GDPR deletions / masking strategies1) Best practice: separate PII store + reference keys. To forget a user: - Mark user dimension deleted_at and wipe pii_encrypted, rotate user_hash or replace with tombstone value. - For events: two options depending on SLA/legal: a) Soft-delete/mask: update events to null user_key/anon_user_id and remove event_props that contain PII. Implement via update queries (supported in Delta/Hudi/Iceberg, Snowflake). Use CDC to track changes. b) Hard delete: physically delete rows matching user_key across partitions; on large datasets prefer storage formats that support row-level deletes (Delta/Hudi/Iceberg) and then compact/optimize to reclaim space.2) Efficient delete pipeline: - Maintain a deletions table (tombstones) with user_hash, delete_ts, request_id. - A periodic compaction job: join events to deletions and produce masked/cleaned partitions; write new optimized files and drop old ones (avoids per-row shuffle). - For streaming stores, emit a delete event that propagates through stream and stateful processors to evict user state and write masking records.3) Minimize cost: - Keep PII out of high-volume facts; store sensitive props in separate encrypted table referenced by event_id — delete by removing from small table. - Use hashed/salted user identifiers (one-way) in facts; store mapping in a secure db. For deletion, rotate or remove mapping so linking is impossible.4) Auditing & SLAs: - Log deletion requests, actions, and verification snapshots. Provide verification query that no PII exists.Trade-offs- Updates/deletes are expensive in immutable file stores (Parquet on S3). Use Delta/Hudi/Iceberg to support deletes/updates efficiently.- Materialized sessions speed queries but need recomputation on deletes if sessions include user-level aggregates.- Soft-mask keeps analytic continuity but may not satisfy strict deletion rules.Example technologies- Batch: Spark + Delta Lake (AWS/GCP), use data compaction and Z-order by user_key.- Streaming: Kafka + Flink/Beam for per-user state sessionization and propagation of deletes.- Warehouse: BigQuery (partition+clustering) for fast scans; use DML or maintain masked copies; Snowflake for cloning and secure views.Key operational controls- Automated test for deletion completeness- Retention / TTL policies for raw events- Encryption-at-rest and RBAC for PII tables- Approval workflow for deletion requests and SLA monitoringThis design gives fast sessionization by clustering/sorting on user+time and precomputed session facts, and enables scalable GDPR deletion via PII separation, tombstone-driven compaction, and using storage formats that support row-level deletes.
Unlock Full Question Bank
Get access to hundreds of Cloud Data Warehouse Architecture interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.