Data Platform Architecture and Technology Selection Questions

The decision matrix has to separate what each engine is actually good at from what it merely can do, because Spark, Presto/Trino, and a serverless warehouse can all technically run most SQL-shaped workloads, whether that workload is scheduled ETL (extract, transform, load) or interactive analysis.

The matrix

Where each wins

Spark is the right default when the workload is genuinely a large, complex transformation, joins across huge datasets, custom business logic that doesn't fit cleanly in SQL, or machine learning feature pipelines, run on a schedule rather than interactively. Presto/Trino earns its keep when analysts need to query across multiple data sources (a lake, a warehouse, an operational database) without first copying everything into one place, and when query latency for interactive exploration matters more than raw batch throughput. A serverless warehouse is the right default for the bulk of day-to-day BI: dashboards, scheduled reports, ad-hoc analyst queries against curated tables, where you want to hand end users a SQL box and not think about cluster sizing at all.

How to benchmark before committing

Pick three or four representative real queries from the actual workload, not a generic benchmark suite; run each on all three finalists against the same dataset; and measure wall-clock latency at a fixed concurrency level (say, 20 simultaneous queries) rather than a single query in isolation, since concurrency behavior is where these engines actually diverge. Also price out a realistic month of usage under each engine's billing model using the query patterns you just measured, not the vendor's example workload. A benchmark that only tests one query at a time will make every engine look adequate and hide the concurrency differences that actually matter in production.

Trade-offs and pitfalls

The most common mistake is standardizing on one engine for everything because "we don't want three systems to operate." That instinct is reasonable at a two-person data team, but at any real scale, forcing 500 interactive BI queries a day through a Spark cluster built for nightly ETL, or forcing a multi-terabyte join through a serverless warehouse's on-demand pricing, produces both worse performance and a worse bill than matching the engine to the workload shape. The second mistake is benchmarking on synthetic data that doesn't reflect your actual join cardinalities or partition layout, which routinely produces a benchmark winner that behaves differently once real data and real concurrency show up.

A data mesh decentralizes data ownership to the domain teams that generate it (marketing, orders, payments), each publishing "data products" with clear contracts, while a centralized platform keeps one team owning ingestion, transformation, and the warehouse for the whole company.

The comparison

Ownership: mesh puts the people closest to the data (who understand it best) in charge of its quality and its contract; centralized puts one platform team in charge of everyone's data, whether or not they understand every domain equally well.
Discoverability: mesh requires a strong shared catalog and standardized metadata across domains, or discovery becomes worse than centralized, since data is now scattered across many owners. Centralized discovery is simpler by construction because everything sits in one place.
Governance: mesh uses federated computational governance, shared standards enforced by tooling (schema validation, SLAs as code) rather than a single team manually reviewing everything; centralized governance is easier to enforce consistently because one team controls the whole pipeline, but that team becomes a bottleneck as the company grows.
Latency (time-to-new-data-product): mesh domain teams can ship a new data product without waiting on a central team's backlog; centralized platforms often become the bottleneck once a company has more than a handful of domains competing for the same platform team's attention.
Cost: mesh usually costs more in tooling and duplicated infrastructure across domains; centralized concentrates cost (and the ability to optimize it) in one place.
Developer velocity: mesh scales velocity horizontally (more domains means more parallel capacity) once the platform is mature; centralized velocity is capped by the size of the platform team, and that cap gets worse as the company scales.

When to favor each

Favor a centralized platform when the organization has fewer than roughly a dozen distinct data domains, when the central platform team is not yet a bottleneck, and when the cost and complexity of federated governance tooling would outweigh its benefit. A 50-person company with three product lines rarely needs a mesh; it needs a competent, well-staffed central team.

Favor a data mesh once a large organization has enough independent domains (order of dozens or more) that a central team has become the bottleneck for every new dataset, and once there is executive appetite to invest in the self-serve platform and governance tooling a mesh requires to avoid becoming an ungoverned mess of inconsistent, undiscoverable domain data. A mesh adopted before that tooling investment is made tends to produce exactly the failure mode critics warn about: each domain reinventing its own formats and quality bar, with no way to reliably query across domains.

Trade-offs and pitfalls

The most common mistake is adopting "data mesh" as an organizational restructuring (just move ownership to domain teams) without investing in the platform and governance tooling that makes federation work; the result is worse than a centralized platform, because now nobody is accountable for cross-domain consistency at all. The second is applying mesh principles to an organization too small to need them, which adds coordination overhead (data contracts, domain-team accountability meetings) that a single central team would have handled faster.

When an organization genuinely needs heavy BI, ad-hoc data science, and near-real-time features all on the same data, the honest answer is usually neither a pure warehouse nor a pure lake alone, but a lakehouse serving BI and data science together, plus a narrow, separate low-latency path for the specific real-time feature needs.

Why not standardize on one alone

A pure warehouse struggles with the unstructured or semi-structured data data science often wants, and it usually isn't built for the sub-second, high-throughput reads a real-time feature-serving layer needs. A pure lake without transactional guarantees creates correctness headaches for BI (dashboards need trustworthy, consistent numbers, not "eventually consistent" files) and forces data science to build its own reproducibility discipline from scratch. A lakehouse (an open table format like Iceberg or Delta Lake over lake storage) closes most of that gap for BI and data science on one copy of the data, but it is not, by itself, a low-latency online feature store.

The architecture

Curated lakehouse tables serve both BI (via the warehouse/query-engine layer on top) and data science (direct table access, versioned, ACID-safe [atomicity, consistency, isolation, durability], so an experiment run last month is reproducible today). For the near-real-time feature need specifically, add a narrow streaming path: a stream processor computes the handful of features that genuinely need sub-second freshness and writes them to a low-latency key-value store (the actual online feature-serving layer), separate from the lakehouse. The lakehouse remains the batch/offline source of truth that periodically backfills and reconciles against that online store, so the two never permanently diverge.

Data flow and governance

Raw events land once, in the lakehouse's raw layer. Batch transforms build the curated tables BI and data science both read. A stream processor reads the same raw event stream in parallel (not a copy pulled from the curated layer) to avoid adding batch latency to the real-time path, computes the specific low-latency features, and writes them to the online store. Governance-wise, both paths should derive from the same event schema and the same feature definitions, checked in code, so a BI number and a real-time feature score never quietly diverge on what "active user" or "recent purchase" means.

Trade-offs and pitfalls

The most common mistake is trying to make one system do all three jobs by choosing whichever technology sounds most feature-complete on paper. A lakehouse alone will disappoint the real-time-feature team on latency; a dedicated feature-store product alone will disappoint BI and data science on the breadth of historical, ad-hoc querying they need. The second mistake is letting the batch and streaming paths compute the same business logic independently in two codebases, which reliably drifts apart over time; sharing the feature/metric definitions between the two paths, even if the execution engines differ, is what prevents that.

Direct answer

Evaluating an ML platform like Kubeflow, SageMaker, or Vertex AI for org-wide adoption needs a structured framework spanning technical fit (does it actually support the org's training/serving workloads), operational impact (what changes for the teams that adopt it, including migration cost), cost (both direct platform cost and the vendor lock-in risk of committing to it), data governance (does it meet the org's data-handling and compliance requirements), and developer experience (will teams actually want to use it), evaluated through a bounded pilot before committing to organization-wide rollout.

Structured elaboration

• Technical criteria: does the platform support the org's actual training workload shapes (distributed training at the needed scale, the specific frameworks/libraries in use, any custom infrastructure needs like specialized hardware); a platform that fits a typical use case well but poorly supports the org's specific heaviest workloads is a real risk that generic feature-comparison checklists can miss.
• Operational impact: what existing tooling/workflows change for teams (does it replace or need to integrate with existing CI/CD, experiment tracking, and monitoring the org already has), and what the migration effort looks like for teams currently on other infrastructure.
• Cost and vendor lock-in risk: direct platform/usage cost at projected scale, plus the harder-to-quantify cost of being locked into a specific vendor's ecosystem (how hard would it be to migrate away later if needed, and does the platform use genuinely portable underlying technology like Kubernetes/open standards, or proprietary abstractions that make later migration expensive).
• Data governance: does the platform meet the org's data residency, access-control, and compliance requirements (relevant especially for regulated industries or sensitive data), and how mature is its audit-logging and permission model.
• Developer experience: would the teams who'd actually use this daily find it genuinely better than their current setup, evaluated through direct hands-on trial by representative users, not just a feature checklist reviewed by the evaluating team alone.
• Pilot plan with success metrics: run a bounded pilot with one or two representative teams and workloads, defining success metrics up front (e.g. reduction in time-to-first-training-run, reduction in infrastructure-debugging time, developer satisfaction survey results) and a clear go/no-go decision point based on the pilot's actual results, before committing to the cost and disruption of an organization-wide rollout.

Worked example

A pilot selecting one team with a representative (not the org's most unusual/edge-case) training workload, running that team's real training pipeline on the candidate platform for a defined 6-week period, tracking time-to-first-successful-training-run, any workload-specific technical gaps discovered, direct cost incurred, and a structured developer-experience survey from the pilot team at the end; a go decision requires the pilot to meet pre-defined thresholds on these metrics (e.g. time-to-first-run improved or at least not regressed, no blocking technical gaps found, developer satisfaction above a defined bar), not just "the pilot team didn't complain."

Trade-offs & pitfalls

The most common evaluation-framework failure is skipping the hands-on pilot and deciding based on a feature-comparison document alone; vendor marketing materials and documentation reliably understate real integration friction and workload-specific gaps that only surface once real engineers try to run real workloads on the platform, which is exactly what a bounded pilot with pre-defined success metrics is designed to surface before the org commits at scale.

The core design choice is a federated operating model with central guardrails, not full centralization: domain teams own their data and remain accountable for its quality, while a small central governance function owns the shared standards, tooling, and the small set of policies that must be consistent everywhere (naming conventions for canonical entities, classification tiers, the schema-contract format, the escalation path for a cross-domain incident). This mirrors how a data-mesh migration frames domain-level responsibility with central guardrails: the mesh doesn't eliminate governance, it moves enforcement out of a central team's backlog and into each domain's own pipeline, backed by shared infrastructure the central team maintains.

Operating model

• Domain teams: own their source tables, their producer contracts, and first-line triage of quality incidents in their own data.
• Central governance function: owns the policy-as-code framework, the catalog, the classification standard, and cross-domain incident coordination. It sets the bar; it does not review every table.
• Federation boundary: a domain team can ship changes freely as long as they pass the shared automated gates; anything that would break a documented cross-domain contract requires the same CI-enforced compatibility check as any producer contract, not a manual sign-off meeting.

Technical controls and organizational changes

1. Ownership assignment. Every dataset with downstream consumers gets a named owning team in the catalog, not a person (people leave, teams persist). This is the single highest-leverage change: an unowned incident cannot be triaged.
2. Runbooks. Each owning team maintains a runbook for their domain's common failure modes (a stalled ingestion job, a schema-drift alert, a null-rate spike) so on-call response does not depend on institutional memory. Runbooks are checked into the same repo as the pipeline and reviewed on the same cadence as the code.
3. Policy-as-code. Classification rules, retention windows, and compatibility rules are expressed as machine-checkable policy (for example, Open Policy Agent rules or a custom linter run in CI) rather than a wiki page, so a violation fails a build instead of being caught in a quarterly audit.
4. Quality gates. Automated checks (freshness, null rate, referential integrity, schema compatibility) run on every load and block publication of a failing batch. This is what actually prevents incidents rather than just detecting them faster.

Concrete rollout: ownership plus runbooks as the first program elements, over 12 months, by vertical

Rather than rolling out to "dozens of teams" simultaneously, sequence by business vertical so the program proves itself before it scales. A realistic first-year plan:

• Months 1-3: pick one vertical, for example the CRM (customer relationship management) domain, as the pilot. Assign explicit ownership for every CRM-sourced table with downstream consumers, write the first runbooks for its two or three most common incident types, and stand up the quality gates on ingestion. This is deliberately narrow so the pattern can be validated before it is asked of teams who did not choose it.
• Months 4-6: extend the same ownership-assignment-plus-runbook pattern to a second vertical with materially different risk profile, for example finance data (where compatibility breaks have compliance consequences, not just dashboard noise). This tests whether the guardrails generalize or were accidentally tuned to CRM's shape of problem.
• Months 7-9: roll the same pattern to the remaining domains in waves of a handful of teams at a time, reusing the policy-as-code rules and runbook templates built in the first two verticals rather than writing new tooling per team.
• Months 10-12: the central function shifts from active rollout to maintenance: quarterly policy review, an incident retrospective process, and a self-service onboarding path so a new team can adopt the standard without a central-team engagement.

What to measure to know it's working, not just running

Trade-offs and pitfalls

The biggest risk in this design is the central function drifting into a review bottleneck by accident: every "just this once" manual sign-off makes the next domain team expect the same, and the whole point of policy-as-code is to make the guardrail a build check, not a meeting. The other common failure is measuring activity instead of outcomes (number of runbooks written, number of tables catalogued) without ever checking whether incident counts actually fell; a program can look busy and still not be reducing the thing it was funded to reduce.