Data Platform Architecture and Technology Selection Questions

Vendor lock-in risk assessment starts with a concrete question: if we had to leave this vendor in six months, what specifically would break, and how long would it take to rebuild.

Assessing the risk

Audit which proprietary features are actually load-bearing versus merely convenient: a proprietary SQL extension used in every transformation query is load-bearing (rewriting it is real work); a UI feature used for occasional debugging is not. Check whether the data itself is stored in an open, portable format (an open table format, standard file formats) or a proprietary storage layer that only that vendor's engine can read; the latter means even "exporting your data" doesn't actually free you, because the export format may not preserve what made the data useful. Check whether your team's operational knowledge (tuning, troubleshooting, cost optimization) has become vendor-specific tribal knowledge that wouldn't transfer.

The playbook

Short-term mitigations: stop adding NEW dependencies on proprietary features going forward, even while continuing to use existing ones, so the lock-in stops growing while you plan the response. Where feasible, start writing new transformation logic in a portable form (standard SQL, or a transformation tool that can target multiple backends) rather than the vendor's proprietary dialect.

Long-term migration strategy: prioritize migrating away from the SPECIFIC features that are hardest to replace, not the easiest ones, since the hard ones are exactly what makes leaving expensive later; tackle them while the total surface area is still smaller than it will be in another year of continued dependency. Where full migration isn't justified yet, invest in an abstraction layer (a transformation framework, a query interface) that could be repointed at a different backend with less rewrite effort than a direct migration would need.

Rough cost estimate for regaining independence

Estimate cost as the sum of: engineering time to rewrite proprietary-feature-dependent logic in a portable form, the cost of running two systems in parallel during validation (a real, often underestimated cost), and the opportunity cost of the engineering time not spent on other roadmap work during the migration window. For an organization with, say, a few dozen pipelines with moderate proprietary-feature dependency, this commonly lands in the range of several engineer-months to a few engineer-years, heavily dependent on how deep the dependency actually runs, which is exactly why the audit step above has to come first: without it, this estimate is a guess rather than a plan.

Trade-offs and pitfalls

The most common mistake is treating "no vendor lock-in" as an absolute goal worth paying for regardless of actual risk; a small team may rationally accept meaningful lock-in in exchange for never having to operate a component themselves, if the switching cost, honestly estimated, is genuinely low relative to the ongoing operational savings. The second mistake is assessing lock-in risk only at adoption time and never revisiting it; dependency accumulates gradually as teams add "just one more" proprietary feature over years, so the actual risk at year three is rarely what it looked like at year one.

Designing a data mesh for 100+ domains has to solve four things at once: who owns what, how a domain publishes something others can safely depend on, how anyone finds it, and what stops the whole thing collapsing into chaos as more domains join.

Architecture

flowchart TB
    subgraph Domain A
    A1[Domain data + logic] --> A2[Data product: contract + schema]
    end
    subgraph Domain B
    B1[Domain data + logic] --> B2[Data product: contract + schema]
    end
    subgraph Domain N
    N1[Domain data + logic] --> N2[Data product: contract + schema]
    end
    A2 --> C[Federated catalog / discovery layer]
    B2 --> C
    N2 --> C
    C --> D[Consumers: BI, ML, other domains]
    E[Self-serve platform: storage, compute, observability, access control] -.-> A2
    E -.-> B2
    E -.-> N2
    F[Federated governance: schema standards, SLAs as code, quality gates] -.-> C

Domain ownership: each domain team owns the data it produces end to end, including its correctness and its documented contract, not just the raw pipeline. Ownership without a real contract just relocates the old central-team-as-bottleneck problem to a hundred smaller bottlenecks.
Data-product contracts: every domain publishes its data through a defined contract: a stable schema, an SLA (service-level agreement covering freshness and uptime), semantic documentation of what each field means, and a versioning policy for changes. This is what lets a consumer depend on domain B's data without needing to understand domain B's internal pipeline.
Discovery/catalog layer: a federated catalog indexes every domain's published data products with consistent metadata (owner, schema, SLA, quality score), so a consumer in domain N can find and evaluate a data product from domain A without asking around. Without this, a mesh with 100+ domains is strictly worse for discoverability than a single centralized platform.
Governance guardrails: federated computational governance means the STANDARDS (schema conventions, required SLA fields, quality thresholds) are centrally defined and automatically enforced (as code, via CI checks on every published data product), while the actual DATA remains domain-owned. This is the mechanism that prevents a hundred domains from each inventing incompatible conventions.
Platform services: a shared self-serve platform (storage, compute, observability, access control, the catalog itself) that every domain builds on, so domains aren't each reinventing infrastructure; this is what makes domain ownership operationally feasible instead of just organizationally declared.

Preventing sprawl

Every published data product must pass automated contract checks (schema validity, required metadata fields, an SLA declaration) before it's discoverable in the catalog; a domain that publishes something without meeting the baseline contract simply doesn't show up as a trustworthy option for consumers. Periodically audit for genuinely duplicate data products across domains (the same underlying concept published inconsistently by two teams) and resolve ownership, this is a governance process, not a purely technical one, but the catalog's metadata is what surfaces the duplication for a human to act on.

Trade-offs and pitfalls

The most common failure is standing up domain ownership and a catalog without genuine, automatically-enforced governance standards; the result looks like a mesh but behaves like a hundred independent, inconsistent data silos, which is worse than the centralized platform it replaced. The second is under-investing in the shared self-serve platform, which pushes every domain to build its own infrastructure from scratch, defeating the efficiency argument for a mesh in the first place.

The decision matrix has to separate what each engine is actually good at from what it merely can do, because Spark, Presto/Trino, and a serverless warehouse can all technically run most SQL-shaped workloads, whether that workload is scheduled ETL (extract, transform, load) or interactive analysis.

The matrix

Where each wins

Spark is the right default when the workload is genuinely a large, complex transformation, joins across huge datasets, custom business logic that doesn't fit cleanly in SQL, or machine learning feature pipelines, run on a schedule rather than interactively. Presto/Trino earns its keep when analysts need to query across multiple data sources (a lake, a warehouse, an operational database) without first copying everything into one place, and when query latency for interactive exploration matters more than raw batch throughput. A serverless warehouse is the right default for the bulk of day-to-day BI: dashboards, scheduled reports, ad-hoc analyst queries against curated tables, where you want to hand end users a SQL box and not think about cluster sizing at all.

How to benchmark before committing

Pick three or four representative real queries from the actual workload, not a generic benchmark suite; run each on all three finalists against the same dataset; and measure wall-clock latency at a fixed concurrency level (say, 20 simultaneous queries) rather than a single query in isolation, since concurrency behavior is where these engines actually diverge. Also price out a realistic month of usage under each engine's billing model using the query patterns you just measured, not the vendor's example workload. A benchmark that only tests one query at a time will make every engine look adequate and hide the concurrency differences that actually matter in production.

Trade-offs and pitfalls

The most common mistake is standardizing on one engine for everything because "we don't want three systems to operate." That instinct is reasonable at a two-person data team, but at any real scale, forcing 500 interactive BI queries a day through a Spark cluster built for nightly ETL, or forcing a multi-terabyte join through a serverless warehouse's on-demand pricing, produces both worse performance and a worse bill than matching the engine to the workload shape. The second mistake is benchmarking on synthetic data that doesn't reflect your actual join cardinalities or partition layout, which routinely produces a benchmark winner that behaves differently once real data and real concurrency show up.

A multi-tenant analytics platform serving customers of varying size has to answer one question first, before any tenancy-model choice: how much isolation does each tenant actually need, because that answer, not raw scale, drives the right model.

Tenancy models compared

Recommendation for varying tenant sizes

A single tenancy model rarely fits customers of genuinely varying size well; a tiered approach is usually the honest answer. Small, cost-sensitive tenants share tables with strict, tested row-level security. Mid-size tenants that need stronger guarantees (perhaps for their own compliance requirements) get separate schemas. A small number of the largest or most security-sensitive tenants get dedicated infrastructure, justified because their revenue or regulatory requirements can absorb the added cost.

Isolation, cost tracking, and retention

Row-level security policies (enforced at the query engine, not just in application code) are the baseline correctness mechanism for the shared-table tier; a bug here is a real cross-tenant data leak, so this deserves the same rigor as an authentication system. Per-tenant cost tracking requires tagging usage (storage and compute) at the tenant level from day one; retrofitting tenant-level cost attribution onto a platform that wasn't designed to track it is materially harder than building it in from the start. Customizable retention per tenant (some tenants may contractually require longer or shorter retention) needs to be a first-class, per-tenant policy in the storage layer, not a single global setting, or you end up either over-retaining data some tenants require you to delete, or under-retaining data others are contractually owed.

Enforcing quotas

Set per-tenant compute and storage quotas enforced by the platform itself (not by convention or manual monitoring), with alerting before a tenant approaches its limit, so a single noisy tenant's usage spike can't degrade performance for everyone sharing that tier's infrastructure. This "noisy neighbor" problem is the most common operational failure mode of the shared-table tier specifically, and it's the strongest argument for moving a consistently-noisy tenant to a more isolated tier rather than trying to throttle them indefinitely within the shared tier.

Trade-offs and pitfalls

The most common mistake is picking one tenancy model for the whole platform because it's simpler to build and operate, when tenants genuinely differ enough in size and isolation needs that a single model either overspends on small tenants (dedicated infrastructure for everyone) or under-isolates large ones (shared tables for everyone). The second is treating row-level security as "done" once written, without ongoing automated testing that verifies a query genuinely cannot return another tenant's rows, since this is exactly the kind of bug that can regress silently after a seemingly unrelated schema change.

Batch processing runs on a schedule, reads a bounded chunk of data (an hour's worth, a day's worth), and produces a result once the job finishes. Streaming processing reads an unbounded, continuous flow of events and produces results incrementally as data arrives. The practical differences follow directly from that distinction.

Core differences

Latency: batch is naturally minutes-to-hours behind real time (however often the job runs); streaming can be seconds behind, sometimes sub-second.
Throughput: batch systems can be tuned for very high aggregate throughput because they process large chunks efficiently in one pass; streaming systems trade some raw throughput for the ability to process continuously.
Complexity: streaming introduces problems batch mostly avoids, out-of-order arrivals, the need to define when a window of time is "done" (watermarking), and the need for durable state that survives a restart. Batch just reruns on the next scheduled window.
State management: batch jobs are largely stateless between runs (each run reads from source, computes, writes out). Streaming jobs often hold running state (a count-so-far, a session-in-progress) that must be checkpointed so a crash doesn't lose it.
Fault tolerance: a failed batch job simply reruns from the same bounded input. A failed streaming job must resume from a checkpoint without either losing events or double-counting them, which is a materially harder engineering problem.

When to combine both

A product that needs nightly retraining or historical reporting alongside near-real-time personalization is describing exactly the situation where a hybrid makes sense: use batch for the large, cost-efficient jobs where a few hours of staleness is fine (full model retraining, historical dashboards, data-quality backfills), and use a streaming path only for the narrow slice of the product that genuinely needs sub-minute freshness (live personalization signals, a real-time counter). Building everything as streaming when only one feature needs it adds operational cost and failure surface for no product benefit; building everything as batch when one feature needs real-time responsiveness means that feature simply won't work.

Worked example

Say the near-real-time need is "show a user's updated recommendation score within 30 seconds of a click." The streaming path only needs to update a small, targeted piece of state (this user's feature vector, or a lightweight score) rather than recomputing the whole model. The nightly batch path retrains the underlying model on the full historical dataset, which the streaming path then just applies. This is a genuinely common hybrid: batch does the expensive, infrequent heavy lifting; streaming does the cheap, frequent update on top of it.

Trade-offs and pitfalls

The most common mistake is choosing streaming everywhere because it sounds more modern, without asking whether the product actually needs sub-minute freshness anywhere. Streaming systems are harder to operate, harder to test (an out-of-order event or a late arrival is a real production case, not an edge case), and more expensive to run continuously than a batch job that only spins up compute once a day. The reverse mistake, building a hybrid where the "streaming" half quietly polls every few minutes instead of processing continuously, is a legitimate design if the freshness requirement tolerates it, and it's simpler to build and operate; call it what it is (frequent micro-batch) rather than "streaming."