Data Platform Architecture and Technology Selection Questions

The core design choice is a federated operating model with central guardrails, not full centralization: domain teams own their data and remain accountable for its quality, while a small central governance function owns the shared standards, tooling, and the small set of policies that must be consistent everywhere (naming conventions for canonical entities, classification tiers, the schema-contract format, the escalation path for a cross-domain incident). This mirrors how a data-mesh migration frames domain-level responsibility with central guardrails: the mesh doesn't eliminate governance, it moves enforcement out of a central team's backlog and into each domain's own pipeline, backed by shared infrastructure the central team maintains.

Operating model

• Domain teams: own their source tables, their producer contracts, and first-line triage of quality incidents in their own data.
• Central governance function: owns the policy-as-code framework, the catalog, the classification standard, and cross-domain incident coordination. It sets the bar; it does not review every table.
• Federation boundary: a domain team can ship changes freely as long as they pass the shared automated gates; anything that would break a documented cross-domain contract requires the same CI-enforced compatibility check as any producer contract, not a manual sign-off meeting.

Technical controls and organizational changes

1. Ownership assignment. Every dataset with downstream consumers gets a named owning team in the catalog, not a person (people leave, teams persist). This is the single highest-leverage change: an unowned incident cannot be triaged.
2. Runbooks. Each owning team maintains a runbook for their domain's common failure modes (a stalled ingestion job, a schema-drift alert, a null-rate spike) so on-call response does not depend on institutional memory. Runbooks are checked into the same repo as the pipeline and reviewed on the same cadence as the code.
3. Policy-as-code. Classification rules, retention windows, and compatibility rules are expressed as machine-checkable policy (for example, Open Policy Agent rules or a custom linter run in CI) rather than a wiki page, so a violation fails a build instead of being caught in a quarterly audit.
4. Quality gates. Automated checks (freshness, null rate, referential integrity, schema compatibility) run on every load and block publication of a failing batch. This is what actually prevents incidents rather than just detecting them faster.

Concrete rollout: ownership plus runbooks as the first program elements, over 12 months, by vertical

Rather than rolling out to "dozens of teams" simultaneously, sequence by business vertical so the program proves itself before it scales. A realistic first-year plan:

• Months 1-3: pick one vertical, for example the CRM (customer relationship management) domain, as the pilot. Assign explicit ownership for every CRM-sourced table with downstream consumers, write the first runbooks for its two or three most common incident types, and stand up the quality gates on ingestion. This is deliberately narrow so the pattern can be validated before it is asked of teams who did not choose it.
• Months 4-6: extend the same ownership-assignment-plus-runbook pattern to a second vertical with materially different risk profile, for example finance data (where compatibility breaks have compliance consequences, not just dashboard noise). This tests whether the guardrails generalize or were accidentally tuned to CRM's shape of problem.
• Months 7-9: roll the same pattern to the remaining domains in waves of a handful of teams at a time, reusing the policy-as-code rules and runbook templates built in the first two verticals rather than writing new tooling per team.
• Months 10-12: the central function shifts from active rollout to maintenance: quarterly policy review, an incident retrospective process, and a self-service onboarding path so a new team can adopt the standard without a central-team engagement.

What to measure to know it's working, not just running

Trade-offs and pitfalls

The biggest risk in this design is the central function drifting into a review bottleneck by accident: every "just this once" manual sign-off makes the next domain team expect the same, and the whole point of policy-as-code is to make the guardrail a build check, not a meeting. The other common failure is measuring activity instead of outcomes (number of runbooks written, number of tables catalogued) without ever checking whether incident counts actually fell; a program can look busy and still not be reducing the thing it was funded to reduce.

Design least-privilege roles around business function rather than data source, and enforce the finance-versus-product separation independently at all three layers (storage, the semantic/metric layer, and the BI tool), since relying on any single layer to be the sole enforcement point means a gap anywhere else in the stack becomes a full bypass.

Least-privilege role design

Role-based access control (RBAC) here means granting each role only the permissions its job function actually needs. Define roles by what a person's job requires, not by what's convenient to grant:

• finance-analyst: read access to finance-tagged datasets only, no access to the broader product-analytics namespace by default.
• product-analyst: read access to self-service product-analytics datasets only, no access to the finance namespace.
• cross-functional-lead: a narrow, explicitly-granted role for the small number of people (for example a finance business partner embedded with a product team) who genuinely need both, rather than widening either base role to cover this case.
• platform-admin: manages the roles and policies themselves, distinct from having standing analytical access to either namespace.

Keeping finance-analyst and product-analyst disjoint by default, with cross-functional-lead as a deliberate, rare exception, avoids the common failure of one broad analyst role slowly accumulating both kinds of access because it was easier to grant everyone the same thing.

Enforcement at the storage layer

Finance data lives under a distinct namespace (a separate schema, bucket, or prefix) with its own identity and access management (IAM) policy boundary, so the default permission set for product-analyst simply doesn't include the finance namespace; new datasets default to whichever namespace they're created in, not to a shared "everything" bucket that then needs per-dataset exceptions carved out. This is the deepest and most reliable enforcement point: even a query or tool that somehow bypasses every layer above still can't read data it has no storage-level grant for.

Enforcement at the semantic or metric layer

A semantic layer (a governed model that defines how raw tables map to business-facing metrics and dimensions) is where row- and column-level policy actually gets attached to a queryable object: finance metrics and dimensions are defined in models tagged and access-controlled the same way as the storage namespace, so a product-analyst role querying through the semantic layer gets an empty or denied result for finance metrics regardless of whether the underlying storage-layer grant would technically allow a raw query to slip through. This layer is also where the metric definitions themselves are governed (a single definition of "revenue" reused everywhere), which matters here because finance and product teams computing the "same" metric differently is a data-integrity risk on top of the access-control risk.

Enforcement at the BI-tool layer

The BI tool connects only through the semantic layer's governed models, never directly to raw storage, and its own user-group-to-role mapping (via single sign-on) mirrors the same finance-analyst/product-analyst split, so a dashboard built against finance data simply has no data source to bind to for a user in the product-analyst group. This layer is the weakest of the three on its own (it's the easiest to misconfigure or bypass by connecting a different client directly to the warehouse) but is still worth enforcing explicitly, since most day-to-day access happens through it and a correctly-configured BI layer keeps the common case fast and correctly scoped without needing every user to understand the layers beneath it.

flowchart TB
    subgraph Storage[Storage layer]
        FINSTOR[(finance namespace)]
        PRODSTOR[(product-analytics namespace)]
    end
    subgraph Semantic[Semantic / metric layer]
        FINMODEL[finance models + metrics]
        PRODMODEL[product-analytics models + metrics]
    end
    subgraph BITool[BI tool]
        FINDASH[finance dashboards]
        PRODDASH[product dashboards]
    end
    FINSTOR --> FINMODEL --> FINDASH
    PRODSTOR --> PRODMODEL --> PRODDASH
    RBAC[RBAC: finance-analyst / product-analyst roles] --> FINSTOR
    RBAC --> PRODSTOR
    RBAC --> FINMODEL
    RBAC --> PRODMODEL
    RBAC --> FINDASH
    RBAC --> PRODDASH

PII masking

Regardless of the finance/product split, any personally identifiable information (PII) present in either namespace (customer names or emails inside finance records, user identifiers inside product-analytics events) is masked by default at the semantic layer, so the finance-versus-product separation and the PII-masking policy are two independent, stacked controls rather than one being assumed to cover the other; a finance-analyst having legitimate access to the finance namespace doesn't automatically mean they should see unmasked customer PII within it.

Audit logging

Every query, at every layer, is logged with the requester's role, the namespace or model accessed, and the columns touched; because enforcement happens at three layers, correlating logs across all three (a query that was denied at the semantic layer but attempted at the storage layer directly, for example) is what surfaces an actual bypass attempt rather than a single layer's log looking clean in isolation.

Trade-offs & pitfalls

Enforcing the same boundary at three layers is more implementation and maintenance work than picking one and trusting it, and policy drift between layers (the semantic layer's finance tag not matching the storage namespace's actual contents after a schema change) is the main ongoing risk; a periodic reconciliation check comparing what's tagged finance at each layer catches this before it becomes a real gap rather than after.

Batch processing runs on a schedule, reads a bounded chunk of data (an hour's worth, a day's worth), and produces a result once the job finishes. Streaming processing reads an unbounded, continuous flow of events and produces results incrementally as data arrives. The practical differences follow directly from that distinction.

Core differences

Latency: batch is naturally minutes-to-hours behind real time (however often the job runs); streaming can be seconds behind, sometimes sub-second.
Throughput: batch systems can be tuned for very high aggregate throughput because they process large chunks efficiently in one pass; streaming systems trade some raw throughput for the ability to process continuously.
Complexity: streaming introduces problems batch mostly avoids, out-of-order arrivals, the need to define when a window of time is "done" (watermarking), and the need for durable state that survives a restart. Batch just reruns on the next scheduled window.
State management: batch jobs are largely stateless between runs (each run reads from source, computes, writes out). Streaming jobs often hold running state (a count-so-far, a session-in-progress) that must be checkpointed so a crash doesn't lose it.
Fault tolerance: a failed batch job simply reruns from the same bounded input. A failed streaming job must resume from a checkpoint without either losing events or double-counting them, which is a materially harder engineering problem.

When to combine both

A product that needs nightly retraining or historical reporting alongside near-real-time personalization is describing exactly the situation where a hybrid makes sense: use batch for the large, cost-efficient jobs where a few hours of staleness is fine (full model retraining, historical dashboards, data-quality backfills), and use a streaming path only for the narrow slice of the product that genuinely needs sub-minute freshness (live personalization signals, a real-time counter). Building everything as streaming when only one feature needs it adds operational cost and failure surface for no product benefit; building everything as batch when one feature needs real-time responsiveness means that feature simply won't work.

Worked example

Say the near-real-time need is "show a user's updated recommendation score within 30 seconds of a click." The streaming path only needs to update a small, targeted piece of state (this user's feature vector, or a lightweight score) rather than recomputing the whole model. The nightly batch path retrains the underlying model on the full historical dataset, which the streaming path then just applies. This is a genuinely common hybrid: batch does the expensive, infrequent heavy lifting; streaming does the cheap, frequent update on top of it.

Trade-offs and pitfalls

The most common mistake is choosing streaming everywhere because it sounds more modern, without asking whether the product actually needs sub-minute freshness anywhere. Streaming systems are harder to operate, harder to test (an out-of-order event or a late arrival is a real production case, not an edge case), and more expensive to run continuously than a batch job that only spins up compute once a day. The reverse mistake, building a hybrid where the "streaming" half quietly polls every few minutes instead of processing continuously, is a legitimate design if the freshness requirement tolerates it, and it's simpler to build and operate; call it what it is (frequent micro-batch) rather than "streaming."

A data warehouse stores structured, cleaned data in a schema defined before you write it (schema-on-write), optimized for fast, repeatable SQL queries. A data lake stores data in its raw, often semi-structured or unstructured form in cheap object storage, with the schema applied when you read it (schema-on-read). A lakehouse adds warehouse-like guarantees, ACID (atomicity, consistency, isolation, durability) transactions, schema enforcement, and fast query performance, directly on top of lake storage, so you get one copy of the data serving both BI and machine learning workloads instead of two.

The three side by side

Where to put what, for a mid-size SaaS company at tens of millions of events/day

Raw events: land in object storage (S3 or equivalent) as the system of record, append-only, partitioned by date. This is the lake layer, and it stays cheap even as retention grows to years of history.
Curated BI tables: build these as warehouse tables (or lakehouse tables if you've adopted one) on top of the raw layer, modeled for the specific questions the business asks repeatedly (funnels, revenue, retention). This is where schema-on-write pays off: BI tools expect stable, typed schemas.
ML feature sets: these usually want the row-level, less-aggregated data that lives closer to the raw layer, but with reproducibility and point-in-time correctness that a bare lake doesn't guarantee. This is the strongest argument for adopting a lakehouse table format even before you need full BI-and-ML unification: it gives the ML team versioned, ACID-safe access to data that would otherwise require a second copy of the pipeline.

Trade-offs and pitfalls

The lakehouse's ACID guarantees are not free: someone still has to run compaction and manage table maintenance (small-file cleanup, log/manifest pruning), or the table format's benefits erode over months of high-frequency writes. A pure warehouse is simplest to operate at a small scale but tends to become the wrong long-term choice once machine learning or genuinely unstructured data (logs, images, free text) enters the picture, because forcing that data through schema-on-write either loses information or requires an awkward second raw-storage layer anyway. A common mistake is treating "lakehouse" as a single product decision: it is a pattern implemented via a table format on top of storage you already have, not something you buy instead of a lake.

A modern end-to-end analytics platform has roughly eight logical components, and naming a real tool at each layer is what separates a memorized list from a working mental model.

The components

Why these eight and not fewer

A platform that skips the raw layer and transforms directly on ingestion loses the ability to replay history when a transform bug is found. A platform that skips the catalog works fine at 20 tables and becomes undiscoverable at 500. The orchestration and monitoring layers are often the two people forget to name first, but they are what make the difference between "a pipeline" and "a platform": without them you have a collection of scripts that happen to run, with no visibility into whether they ran correctly.

Worked example

For a 15-person startup with one data engineer: a Fivetran-style connector or a simple Python script for ingestion, S3 for raw storage, dbt running inside a serverless warehouse (BigQuery on-demand) for transformation and the curated layer, the warehouse plus a light BI tool for serving, a managed orchestrator (Cloud Composer) or even just dbt Cloud's own scheduler if the DAG is simple, warehouse query-history alerts plus a handful of dbt tests for monitoring, and the warehouse's native catalog rather than a standalone tool. Every layer is present, but three of the eight are riding on a single managed product rather than a dedicated tool, which is the correct trade-off at this scale.

Trade-offs and pitfalls

The temptation at small scale is to skip orchestration and monitoring because "the pipeline is simple enough to just run on a cron job." That works until a job silently fails at 3am and nobody notices for a week, which is exactly the failure mode a thin orchestrator plus alerting is built to catch. The opposite failure, at larger scale, is standing up a dedicated best-of-breed tool for every layer before there is a team big enough to operate eight different systems; a platform team of two people cannot realistically run Kafka, Airflow, a standalone catalog, and a standalone observability tool all at once, so consolidating onto the warehouse's native features for two or three of these layers is often the more honest answer than naming eight separate best-of-breed products.