Data Lake Architecture and Governance Questions

Design and governance of data lake systems to support diverse analytics and machine learning workloads. Topics include organizing data zones such as raw and processed layers, schema on read versus schema on write considerations, metadata and cataloging, data discovery, lineage tracking, data versioning, handling schema evolution, access controls, and policies to ensure data quality and discoverability. Candidates should explain how governance and architecture together enable reliable analytics and reuse.

HardTechnical

41 practiced

Discuss the trade-offs between a centralized, single-source-of-truth metadata catalog and a federated catalog architecture where teams manage their own metadata. Consider discoverability, ownership, latency, scalability, and governance implications and recommend when to use each approach.

Sample Answer

Start by clarifying terms: a centralized single-source-of-truth (SSOT) metadata catalog is one authoritative service that stores metadata, schemas, lineage, quality metrics and access policies. A federated catalog lets teams own and operate local catalogs (or metadata stores) and surface metadata via a common search/index or API layer.

Trade-offs

- Discoverability - Centralized: excellent — one place to search, consistent indexing and global search UX. - Federated: harder — users must hit a unified index or federation gateway; inconsistent tagging hurts search recall unless enforced.

- Ownership & Agility - Centralized: ownership is shared or owned by central data platform; can bottleneck updates and slow domain teams. - Federated: strong domain ownership; teams can evolve schemas and metadata quickly and embed domain knowledge.

- Latency & Freshness - Centralized: depends on ingestion cadence; near-real-time requires robust pipelines and can be heavy. - Federated: lower local-latency for team-owned changes; federation can expose near-real-time if implemented with event-driven sync.

- Scalability - Centralized: simpler conceptually but must scale to store and serve all metadata; single system scaling and operational burden. - Federated: scales horizontally by team; less central operational load but more heterogeneous systems to maintain.

- Governance, Compliance & Consistency - Centralized: easier to enforce global policies, access controls, lineage standards and audits. - Federated: harder to guarantee consistent policies; requires governance contracts, automated policy propagation, or a policy-as-code layer.

Recommendations- Use Centralized SSOT when: organization requires tight governance/compliance (finance, healthcare), strong cross-team discoverability is critical, or you have a mature platform team able to operate and scale a central system.- Use Federated when: you have many autonomous domains needing rapid iteration, domain-specific metadata models, or you want to reduce central operational bottlenecks. Mitigate governance via a federated contract: common metadata schema core, shared APIs, event-driven sync, and a central index for discovery plus policy-as-code enforcement.

Hybrid approach (often best): implement a federated ingestion/ownership model with a centralized index and governance layer — teams own metadata lifecycle; the platform provides standards, validation, policy enforcement and a global search UI. This balances discoverability, domain agility, scalability and governance while keeping operational costs manageable.

MediumTechnical

35 practiced

Compare lakehouse technologies (Delta Lake, Apache Hudi, Apache Iceberg) versus a classic object-store data lake. Focus on features: ACID transactions, time travel, upserts/merges, compaction, metadata management, and operational complexity for ML pipelines.

Sample Answer

Situation: As a data engineer choosing storage for ML pipelines, I compare lakehouse engines (Delta Lake, Apache Hudi, Apache Iceberg) to a classic object-store data lake (e.g., raw Parquet on S3/GCS).

ACID transactions- Lakehouse: Provide ACID semantics via transactional layers (Delta uses transaction log, Hudi/ICEBERG have metadata layers + catalog). Reliable concurrent writes and reads; simplifies streaming + batch consistency.- Classic object store: No native ACID — concurrent writes can cause partial files or read inconsistencies; you must implement locking or overwrite strategies.

Time travel- Lakehouse: Built-in time travel via commit history / snapshots (query previous versions easily).- Classic object store: Requires manual snapshotting/versioned prefixes; harder to query historical state.

Upserts/Merges- Lakehouse: Support upserts/merge semantics (MERGE INTO in Delta, Hudi’s MOR/Copy-On-Write, Iceberg’s MERGE INTO via engines). Enables idempotent CDC and GDPR deletes.- Classic object store: Upserts require rewriting partitions/files or complex ETL; inefficient.

Compaction- Lakehouse: Provide compaction/cleaning features (Hudi compaction for MOR, Delta OPTIMIZE, Iceberg rewrite/compaction planning). Reduces small-file problem and improves read perf.- Classic object store: You must run custom compaction jobs.

Metadata management- Lakehouse: Centralized transaction log or table metadata (faster listings, partition evolution). Integrates with Hive/Glue catalogs.- Classic: Rely on object listing — slow at scale and brittle for schema evolution.

Operational complexity for ML pipelines- Lakehouse: Lowers complexity — reliable incremental ETL, reproducible feature engineering (time travel), easier model retraining and rollback. But you must manage table services (compaction, retention policies), choose correct write mode (COW vs MOR), and monitor transaction logs.- Classic: Simpler to start but higher operational burden long-term: bespoke upsert logic, slower joins, fragile pipelines, and difficult reproducibility.

Recommendation: For production ML at scale prefer a lakehouse engine (choose based on ecosystem: Delta for Spark + Databricks, Hudi for near-real-time CDC, Iceberg for open-table format and multi-engine reads). Use object-store only for cheap archival or simple append-only ingestion.

HardSystem Design

38 practiced

Design a policy enforcement architecture for data lake governance using a policy engine like Open Policy Agent (OPA). Explain how policies are authored (sensitivity, retention, access), where they are evaluated (ingest, catalog, query gateway), how to handle policy updates, and how to audit enforcement events.

Sample Answer

Requirements (clarify):- Enforce sensitivity, retention, access policies across ingestion, cataloging, and query.- Low-latency checks on query path; near-real-time checks on ingestion.- Centralized authoring, safe rollout, full audit trail.

High-level architecture:- Policy Authoring & CI: Git repo of Rego policies + JSON/YAML data (sensitivity labels, retention schedules, role mappings). PRs trigger CI: unit tests (conftest/opa test), static checks, canary promotion.- Policy Distribution: OPA Bundles served from central HTTP bundle server (or OPA Gatekeeper + admission controllers in K8s). Use versioned bundles and hash-based caching.- Enforcement Points: 1. Ingest layer (stream processors / Airflow tasks / Kafka Connect): call local OPA sidecar/SDK to validate metadata (sensitivity tags, retention TTL) before write; block or tag pipelines. 2. Catalog (Metastore like Hive/Glue): evaluate on register/update table API via OPA; enforce required tags and retention policies. 3. Query Gateway (Presto/Trino, Spark SQL gateway): enforce row/column-level access and masking — evaluate OPA per-request to return policy decision + obligations (e.g., mask columns).- Policy Data: store policy data (labels, DLP regexes, role bindings) in a config DB (encrypted) and fetch into OPA as data.

Example Rego (simple sensitivity check):

rego

package datalake.access

default allow = false

allow {
  input.user_role == data.roles[input.resource.owner]
  not sensitive_blocked
}

sensitive_blocked {
  data.sensitivity[input.resource.path] == "PII"
  input.user_role != "data_privileged"
}

Policy updates & rollout:- GitOps flow: changes → CI tests → bundle build → promote to staging bundle endpoint.- Gate deployment: canary with a percentage of requests routed to new bundle; collect metrics.- Atomic rollout via bundle version/header; clients fetch on change or subscribe using SSE/webhooks.- Schema/version compatibility tests to avoid breaking evaluations.

Auditing enforcement events:- Every evaluation returns decision + metadata; log structured events (timestamp, decision, policy_id, input_hash, user, resource, obligation).- Ship logs to centralized event bus (Kafka) and immutable storage (S3 with WORM or append-only DB).- Downstream consumers: SIEM, data-governance UI, replay capability for forensics.- Maintain policy change audit: Git history + CI artifacts + bundle versions correlated with enforcement events.

Scalability & reliability:- Deploy OPA as local sidecars for low latency; use memory-cached bundles.- For high-throughput query gateways, push some decisions to compiled wasm plugins or pre-computed ACL caches.- Trade-offs: More runtime checks add latency—mitigate with caching, obligations, and async enforcement for non-blocking controls (e.g., tagging + remediation).

This design provides centralized, testable policy authoring, low-latency enforcement at three critical points, safe rollouts, and complete auditable trails for compliance.

MediumTechnical

36 practiced

Describe what a data contract between a producer team and a consumer team should contain for datasets landing in a data lake. Explain enforcement mechanisms (schema registry, CI checks, semantic tests), evolution policy, and how you’d handle breaking changes requested by producers.

Sample Answer

A strong data contract between a producer and consumer for datasets in a data lake should be a concise, versioned agreement that covers schema, semantics, SLAs, and operational expectations.

Key contents:- Schema (field names, types, nullability, units, allowed values) with an example Avro/Parquet/JSON schema and a version identifier.- Semantic definitions (business meaning for each field, canonical enumerations, owner/contact).- Quality & validation rules (null thresholds, distribution bounds, uniqueness constraints).- SLA & delivery (frequency, latency, expected partitioning, retention).- Access & security (ACLs, PII flags, masking rules).- Change policy and compatibility rules (what’s allowed as non-breaking vs breaking).- Monitoring & observability requirements (metrics to emit, error handling).

Enforcement mechanisms:- Schema registry (Avro/Protobuf/JSON Schema) stores versions and enforces compatibility (backward/forward/none). Producers must register changes; consumers fetch schemas programmatically.- CI checks in producer pipelines: validate output against registered schema, run unit tests, enforce linting and contract tests before deployment.- Semantic tests (data tests) run in CI: e.g., Great Expectations or custom PySpark checks for distributions, null rates, referential integrity. These run daily in orchestration (Airflow) and on commits.- Runtime guards: ingestion jobs reject or quarantine non-conforming files; automated alerts to owners.

Evolution policy:- Default mode: backward-compatible changes allowed (adding nullable fields, relaxed enums). Use semantic versioning (MAJOR.MINOR.PATCH) where MAJOR indicates breaking changes.- Deprecation window: announce breaking fields 2+ release cycles or 90 days, mark deprecated fields, and maintain dual-write or adapter logic during transition.- Contract change process: change request ticket, impact analysis (list of consumers & downstream pipelines), automated compatibility report, scheduled rollout.

Handling breaking changes requested by producers (how I’d run it):1. Assess impact: identify consumers via registry metadata and run a dry-run CI to surface failures.2. Propose alternatives: prefer non-breaking approaches (add new field, feature flag, versioned dataset).3. If breaking is unavoidable: require a migration plan—dual-write period (producer writes v1 and v2), adapters in consumers, or provide a transformation layer in the lake (e.g., a compatibility view/ETL that presents old schema).4. Coordinate timeline: agree SLAs, testing plan, and rollback path; enforce via gated deployments (CI/CD) and monitor key metrics post-change.5. Post-migration: retire old version after consumers confirm migration and an agreed deprecation period.

This approach combines clear, machine-readable contracts with automated enforcement and a collaborative, risk-managed evolution process so producers can innovate without breaking consumers.

HardSystem Design

43 practiced

Design a dataset versioning model to support reproducible ML experiments that require 1) snapshotting datasets, 2) storing diffs efficiently, and 3) enabling quick checkout of a historic version. Explain storage layout (objects vs deltas), indexing, and integration with model training pipelines.

Sample Answer

Requirements & constraints:- Functional: immutable dataset snapshots, efficient diffs between versions, fast checkout for training.- Non-functional: petabyte-scale, many small files and large binary blobs, low-latency checkouts, cheap storage for historical versions.

High-level architecture:- Object storage (S3/GCS) for raw data blobs + a metadata service (SQL/NoSQL) holding version graph and indices.- Compute layer (Spark/Kube) to apply deltas and materialize snapshots.- API/CLI for snapshot, diff, checkout; integration hooks for training pipelines (KFP/Airflow).

Storage layout:- Content-addressed objects: store every file/chunk as immutable object keyed by hash. Large files chunked (e.g., fixed-size or rabin).- Manifests: a snapshot manifest lists object hashes + file paths + metadata (schema, timestamp, provenance).- Deltas: store diffs as lists of object additions, deletions, and rename metadata. For file-level small diffs, store binary deltas (bsdiff) when beneficial.- Tiering: recent snapshots materialized (full manifests); older snapshots only store manifests + deltas to a nearby full snapshot to save space.

Indexing & version graph:- Global object index (hash -> storage location, size, refs, creation time) in a scalable key-value store (Dynamo/Bigtable/Cassandra).- Snapshot index: manifest id -> parent id(s), provenance tags, and precomputed diff summaries (added/removed counts) for quick query.- Secondary indices: path -> list of manifests containing that path; tag -> manifest; quality metrics.

Checkout and performance:- Fast checkout: for a requested manifest, if materialized full bundle exists, stream objects directly to training nodes; else reconstruct by applying deltas from closest full snapshot using parallel fetch of object hashes and assemble in parallel (use checksums for integrity).- Use local cache on training infra (node-level cache or shared fast cache) to avoid repeated downloads. Serve small files via index + sharded fetch; for large files use ranged GETs to parallelize chunk downloads.

Integration with training pipelines:- Training job requests manifest id (or tag like "experiment-123"). The pipeline pulls manifest, validates object availability, mounts/streams data (FUSE or direct S3 reads), and records manifest id used into experiment metadata for reproducibility.- Provide lightweight SDK to expose dataset APIs: get_manifest(), diff(a,b), checkout(manifest, path), and checkpoint(current_dataset_state) which creates manifests and triggers ingestion of new objects.

Consistency, GC & retention:- Reference counting in object index for GC; enforce retention policies per tag (retain "production" snapshots longer).- Reclaims unreferenced objects via batched GC jobs; retain deltas until next compaction.

Scalability & trade-offs:- Storing objects deduplicates and speeds up diffs; binary deltas save space but increase reconstruction CPU/time.- Materialize recent checkpoints to optimize checkout latency; compact periodically to limit delta chain length.- Use eventual consistency for object store but strong consistency for manifest metadata to guarantee reproducibility.

Monitoring & governance:- Audit logs linking manifests to producers, checksums, and experiments.- Metrics: checkout latency, reconstruction cost, storage saved via deduplication.

This design balances storage efficiency (objects + deltas), fast checkout (materialized snapshots + caching), and traceable reproducibility via immutable manifests and tight integration with training pipelines.

Unlock Full Question Bank

Get access to hundreds of Data Lake Architecture and Governance interview questions and detailed answers.

Join thousands of developers preparing for their dream job.