Data Platform Architecture and Technology Selection Questions

Designing a data mesh for 100+ domains has to solve four things at once: who owns what, how a domain publishes something others can safely depend on, how anyone finds it, and what stops the whole thing collapsing into chaos as more domains join.

Architecture

flowchart TB
    subgraph Domain A
    A1[Domain data + logic] --> A2[Data product: contract + schema]
    end
    subgraph Domain B
    B1[Domain data + logic] --> B2[Data product: contract + schema]
    end
    subgraph Domain N
    N1[Domain data + logic] --> N2[Data product: contract + schema]
    end
    A2 --> C[Federated catalog / discovery layer]
    B2 --> C
    N2 --> C
    C --> D[Consumers: BI, ML, other domains]
    E[Self-serve platform: storage, compute, observability, access control] -.-> A2
    E -.-> B2
    E -.-> N2
    F[Federated governance: schema standards, SLAs as code, quality gates] -.-> C

Domain ownership: each domain team owns the data it produces end to end, including its correctness and its documented contract, not just the raw pipeline. Ownership without a real contract just relocates the old central-team-as-bottleneck problem to a hundred smaller bottlenecks.
Data-product contracts: every domain publishes its data through a defined contract: a stable schema, an SLA (service-level agreement covering freshness and uptime), semantic documentation of what each field means, and a versioning policy for changes. This is what lets a consumer depend on domain B's data without needing to understand domain B's internal pipeline.
Discovery/catalog layer: a federated catalog indexes every domain's published data products with consistent metadata (owner, schema, SLA, quality score), so a consumer in domain N can find and evaluate a data product from domain A without asking around. Without this, a mesh with 100+ domains is strictly worse for discoverability than a single centralized platform.
Governance guardrails: federated computational governance means the STANDARDS (schema conventions, required SLA fields, quality thresholds) are centrally defined and automatically enforced (as code, via CI checks on every published data product), while the actual DATA remains domain-owned. This is the mechanism that prevents a hundred domains from each inventing incompatible conventions.
Platform services: a shared self-serve platform (storage, compute, observability, access control, the catalog itself) that every domain builds on, so domains aren't each reinventing infrastructure; this is what makes domain ownership operationally feasible instead of just organizationally declared.

Preventing sprawl

Every published data product must pass automated contract checks (schema validity, required metadata fields, an SLA declaration) before it's discoverable in the catalog; a domain that publishes something without meeting the baseline contract simply doesn't show up as a trustworthy option for consumers. Periodically audit for genuinely duplicate data products across domains (the same underlying concept published inconsistently by two teams) and resolve ownership, this is a governance process, not a purely technical one, but the catalog's metadata is what surfaces the duplication for a human to act on.

Trade-offs and pitfalls

The most common failure is standing up domain ownership and a catalog without genuine, automatically-enforced governance standards; the result looks like a mesh but behaves like a hundred independent, inconsistent data silos, which is worse than the centralized platform it replaced. The second is under-investing in the shared self-serve platform, which pushes every domain to build its own infrastructure from scratch, defeating the efficiency argument for a mesh in the first place.

When compute and storage costs on a data platform have grown too high, the fix has to separate three distinct cost drivers, wasteful storage tiering, wasteful compute usage patterns, and wasteful query design, because each has a different remedy.

Storage tiering

Move data that is rarely queried (older than, say, 90 days) from hot, expensive storage to a cheaper warm or cold tier, and delete or archive data past its actual retention requirement rather than keeping everything "just in case." This is usually the single largest and least risky lever, because it doesn't touch how anyone queries the data, only where it physically sits.

Materialized views and caching

If ad-hoc queries or dashboards repeatedly recompute the same aggregation from raw data, materializing that aggregation once (refreshed on a schedule matching how fresh the answer actually needs to be) turns many expensive scans into one cheap scan plus many cheap reads of the precomputed result. A result cache in front of the BI layer catches the common case of the exact same query being run many times a day by different dashboard viewers.

Compute autoscaling

For scheduled ETL (extract, transform, load) jobs on transient clusters, right-size the cluster to the actual job (a job that historically finishes with cluster utilization at 30 percent is over-provisioned) and ensure clusters actually shut down between runs rather than idling. For interactive query compute, autoscale down aggressively outside business hours rather than running a fixed-size warehouse around the clock.

Architectural changes

If ad-hoc analysts are scanning raw, unpartitioned files directly, partitioning and clustering the underlying tables on the columns those queries actually filter on (date, region, customer segment) is often the highest-leverage architectural fix, because it reduces bytes scanned per query regardless of who runs it or how often.

Worked example

Say ad-hoc queries against a raw events table currently scan the full table (say, 10 TB) on every query because it isn't partitioned by date, and analysts typically only care about the last 30 days. Partitioning by date so a typical query only scans the relevant 30-day slice (say, roughly 300 GB, if daily volume is even) cuts bytes-scanned-per-query by roughly 97 percent for that common access pattern, which for a byte-scanned pricing model translates almost directly into a proportional cost reduction on that workload, without changing a single line of the analyst-facing query.

Preserving query performance while cutting cost

The order of operations matters: apply storage tiering and partitioning first (these reduce cost without touching latency, and often improve it), then materialize the specific hot aggregations dashboards depend on (this improves latency while cutting cost), and only then look at autoscaling policy (which affects cost directly but needs headroom tuning to avoid introducing cold-start latency during traffic spikes). Cutting compute capacity before fixing the underlying query patterns just makes the same wasteful queries slower without actually reducing waste.

Trade-offs and pitfalls

The most common mistake is reaching for aggressive autoscaling or smaller compute allocations as the first lever, because it's the fastest change to make, without first fixing the partitioning and materialization issues that are the actual source of wasted spend; this trades a cost problem for a latency problem instead of solving either. The second is archiving or deleting data based on age alone without checking whether a specific, infrequently-run but business-critical report (an annual compliance report, say) still depends on it.

Batch processing runs on a schedule, reads a bounded chunk of data (an hour's worth, a day's worth), and produces a result once the job finishes. Streaming processing reads an unbounded, continuous flow of events and produces results incrementally as data arrives. The practical differences follow directly from that distinction.

Core differences

Latency: batch is naturally minutes-to-hours behind real time (however often the job runs); streaming can be seconds behind, sometimes sub-second.
Throughput: batch systems can be tuned for very high aggregate throughput because they process large chunks efficiently in one pass; streaming systems trade some raw throughput for the ability to process continuously.
Complexity: streaming introduces problems batch mostly avoids, out-of-order arrivals, the need to define when a window of time is "done" (watermarking), and the need for durable state that survives a restart. Batch just reruns on the next scheduled window.
State management: batch jobs are largely stateless between runs (each run reads from source, computes, writes out). Streaming jobs often hold running state (a count-so-far, a session-in-progress) that must be checkpointed so a crash doesn't lose it.
Fault tolerance: a failed batch job simply reruns from the same bounded input. A failed streaming job must resume from a checkpoint without either losing events or double-counting them, which is a materially harder engineering problem.

When to combine both

A product that needs nightly retraining or historical reporting alongside near-real-time personalization is describing exactly the situation where a hybrid makes sense: use batch for the large, cost-efficient jobs where a few hours of staleness is fine (full model retraining, historical dashboards, data-quality backfills), and use a streaming path only for the narrow slice of the product that genuinely needs sub-minute freshness (live personalization signals, a real-time counter). Building everything as streaming when only one feature needs it adds operational cost and failure surface for no product benefit; building everything as batch when one feature needs real-time responsiveness means that feature simply won't work.

Worked example

Say the near-real-time need is "show a user's updated recommendation score within 30 seconds of a click." The streaming path only needs to update a small, targeted piece of state (this user's feature vector, or a lightweight score) rather than recomputing the whole model. The nightly batch path retrains the underlying model on the full historical dataset, which the streaming path then just applies. This is a genuinely common hybrid: batch does the expensive, infrequent heavy lifting; streaming does the cheap, frequent update on top of it.

Trade-offs and pitfalls

The most common mistake is choosing streaming everywhere because it sounds more modern, without asking whether the product actually needs sub-minute freshness anywhere. Streaming systems are harder to operate, harder to test (an out-of-order event or a late arrival is a real production case, not an edge case), and more expensive to run continuously than a batch job that only spins up compute once a day. The reverse mistake, building a hybrid where the "streaming" half quietly polls every few minutes instead of processing continuously, is a legitimate design if the freshness requirement tolerates it, and it's simpler to build and operate; call it what it is (frequent micro-batch) rather than "streaming."

A data mesh decentralizes data ownership to the domain teams that generate it (marketing, orders, payments), each publishing "data products" with clear contracts, while a centralized platform keeps one team owning ingestion, transformation, and the warehouse for the whole company.

The comparison

Ownership: mesh puts the people closest to the data (who understand it best) in charge of its quality and its contract; centralized puts one platform team in charge of everyone's data, whether or not they understand every domain equally well.
Discoverability: mesh requires a strong shared catalog and standardized metadata across domains, or discovery becomes worse than centralized, since data is now scattered across many owners. Centralized discovery is simpler by construction because everything sits in one place.
Governance: mesh uses federated computational governance, shared standards enforced by tooling (schema validation, SLAs as code) rather than a single team manually reviewing everything; centralized governance is easier to enforce consistently because one team controls the whole pipeline, but that team becomes a bottleneck as the company grows.
Latency (time-to-new-data-product): mesh domain teams can ship a new data product without waiting on a central team's backlog; centralized platforms often become the bottleneck once a company has more than a handful of domains competing for the same platform team's attention.
Cost: mesh usually costs more in tooling and duplicated infrastructure across domains; centralized concentrates cost (and the ability to optimize it) in one place.
Developer velocity: mesh scales velocity horizontally (more domains means more parallel capacity) once the platform is mature; centralized velocity is capped by the size of the platform team, and that cap gets worse as the company scales.

When to favor each

Favor a centralized platform when the organization has fewer than roughly a dozen distinct data domains, when the central platform team is not yet a bottleneck, and when the cost and complexity of federated governance tooling would outweigh its benefit. A 50-person company with three product lines rarely needs a mesh; it needs a competent, well-staffed central team.

Favor a data mesh once a large organization has enough independent domains (order of dozens or more) that a central team has become the bottleneck for every new dataset, and once there is executive appetite to invest in the self-serve platform and governance tooling a mesh requires to avoid becoming an ungoverned mess of inconsistent, undiscoverable domain data. A mesh adopted before that tooling investment is made tends to produce exactly the failure mode critics warn about: each domain reinventing its own formats and quality bar, with no way to reliably query across domains.

Trade-offs and pitfalls

The most common mistake is adopting "data mesh" as an organizational restructuring (just move ownership to domain teams) without investing in the platform and governance tooling that makes federation work; the result is worse than a centralized platform, because now nobody is accountable for cross-domain consistency at all. The second is applying mesh principles to an organization too small to need them, which adds coordination overhead (data contracts, domain-team accountability meetings) that a single central team would have handled faster.

The choice between managed streaming services and self-hosted Kafka comes down to three things: how much operational burden the team can absorb, how deep the existing Kafka-specific expertise already runs, and how tightly the workload needs multi-region, high-throughput guarantees that differ meaningfully between the options.

The comparison

Recommendation by scenario

For a high-growth startup with limited SRE (site reliability engineering) staff, a managed service (Kinesis or Pub/Sub, whichever matches the existing cloud provider) is the clear default: the operational cost of running Kafka well, partition rebalancing, broker upgrades, capacity planning, is exactly the kind of ongoing specialist burden a small team can't easily absorb, and the managed alternatives handle multi-region delivery and at-least-once guarantees adequately for most workloads at this stage.

For an enterprise with existing, deep Kafka operational expertise and an established ecosystem of Kafka Connect integrations and Kafka Streams applications, self-hosted (or a managed Kafka flavor like Confluent Cloud or Amazon MSK, which preserves the Kafka ecosystem while offloading some operations) is usually the better choice, because throwing away years of accumulated tooling and expertise to move to a different messaging paradigm is a real, often underestimated migration cost that a pure feature comparison misses.

Trade-offs and pitfalls

The most common mistake is choosing self-hosted Kafka for its ecosystem richness without honestly weighing whether the team has the SRE capacity to operate it reliably at 3am; a beautifully-designed Kafka Streams application that goes down for hours during an unattended broker issue is a worse outcome than a less flexible managed service that simply stays up. The second is assuming a managed service's multi-region story is a solved problem out of the box; verify the specific consistency and latency guarantees for cross-region delivery on your chosen managed offering, since "multi-region" isn't a single guarantee across providers.