Data Platform Architecture and Technology Selection Questions

A workable catalog for this scale needs three things working together: a required metadata contract every dataset must carry, lineage captured by instrumenting the pipelines and BI tools themselves (not by asking people to fill in a diagram), and an ownership model that survives reorgs because it is tied to a role, not a person.

Metadata fields required

Split the schema into technical and operational fields; both are required, not optional extras:

At 500 datasets and 50 dashboards, the field that gets skipped first under time pressure is the SLA field, and it's the one that matters most once something breaks: without it, "is this dashboard's data late" has no defined answer.

Automated lineage capture

Manual lineage documentation rots within a quarter at this scale (50 dashboards means a busy schedule of independent changes). Capture it from the systems that already know the truth:

1. Pipeline instrumentation. If pipelines run on an orchestrator (for example Airflow) and a transformation tool (for example dbt), both can emit lineage events in the OpenLineage format on every run: which tables were read, which were written. Aggregate these events into the catalog automatically.
2. SQL parsing for ad-hoc jobs. For scripts that bypass dbt, parse the SQL text with a SQL lineage library (a static analyzer that extracts FROM/JOIN/INSERT INTO targets) to build the same upstream/downstream edges without a human annotating anything.
3. BI-to-dataset mapping. Most BI tools expose an API or a query log; poll it to extract which underlying tables each of the 50 dashboards actually queries, so the catalog knows table-to-dashboard edges, not just table-to-table edges.
4. Reconciliation, not replacement, for manual input. Let owners add business-glossary terms and descriptions by hand (that's genuinely undocumentable automatically), but never let a human hand-maintain the graph edges themselves; automation is the source of truth for structure, humans annotate meaning on top of it.

Assigning and keeping ownership current

• Assign at creation, not after the fact. Whoever's pipeline job or dbt model first writes a dataset is proposed as the default owning team; require them to confirm (not silently accept) during onboarding so ownership isn't a guess.
• Tie ownership to a team/role, not a person. "The Payments Analytics team" survives a reorg; "Priya" does not. The catalog stores a team identifier plus a rotating on-call contact resolved from that team's paging system, so ownership metadata never manually needs updating when an individual leaves.
• Detect drift automatically. Run a quarterly job that checks: does the recorded owning team still exist in the org directory, and has anyone from that team actually queried or modified the dataset in the last 90 days? A dataset whose "owner" team no longer touches it gets flagged as orphaned rather than silently trusted.
• Escalation path for orphans. Orphaned or contested datasets go into a review queue owned by a data governance function (even a lightweight one person or committee), who either reassigns ownership based on who's actually consuming the data downstream, or schedules the dataset for deprecation if nobody claims it.

flowchart LR
    subgraph Sources
        AF[Orchestrator jobs]
        DBT[Transformation runs]
        BI[BI tool query logs]
    end
    AF -->|OpenLineage events| CAT[(Catalog metadata store)]
    DBT -->|OpenLineage events| CAT
    BI -->|dashboard-to-table mapping| CAT
    OWN[Ownership drift job] -->|quarterly check| CAT
    CAT --> UI[Catalog search / lineage graph UI]
    CAT --> IMPACT[Impact analysis: dataset change to affected dashboards]

Worked example

Say the average dataset in this catalog feeds 3 downstream tables before reaching a dashboard (a 3-hop lineage chain), and the catalog runs its ownership-drift check weekly. Across 500 datasets that's roughly 1,500 lineage edges to maintain (500 datasets times an average of 3 hops), which is exactly the scale where hand-maintained lineage diagrams stop being kept honest and automated capture stops being optional.

Concretely: if a source table's schema changes, the catalog's lineage graph lets you query "which of the 50 dashboards depend on this table, directly or transitively" and get a definite answer in one lookup, instead of grepping dashboard definitions or asking around.

A capacity-planning model has to connect three growing inputs (data volume, retention, and query concurrency) to two cost drivers (storage and compute) with enough transparency that someone can challenge your assumptions later.

The model

Storage cost scales with retained data volume, which is a function of daily ingestion volume and retention period:

Compute cost has two components: scheduled ETL/ELT (extract, transform, load / extract, load, transform) jobs, roughly proportional to data volume processed, and interactive query compute (roughly proportional to concurrent users and query complexity, largely independent of total data volume if partitioning is effective):

Worked example with pinned inputs

Assume: daily ingestion of 500 GB per day, a 2-year retention policy, 50 concurrent analysts averaging 20 queries per day each, and illustrative, platform-specific unit rates of 0.02 cost-units per GB-month for storage, 0.01 cost-units per GB processed for ETL compute, and 0.05 cost-units per query for interactive compute (substitute your own platform's actual billed rates here).

Storage: 500 GB/day times 365 days times 2 years equals 365,000 GB, or 365 TB, retained after two years. At 0.02 cost-units per GB-month, that is roughly 7,300 cost-units per month once fully ramped (this grows steadily over the two years as retained volume accumulates; it is not the month-1 cost).

ETL compute: 500 GB/day times 0.01 cost-units per GB times 30 days equals 150 cost-units per month.

Interactive compute: 50 users times 20 queries/day times 30 days times 0.05 cost-units per query equals 1,500 cost-units per month.

Total steady-state monthly cost at the 2-year retention horizon: roughly 7,300 plus 150 plus 1,500, or about 8,950 cost-units per month, with storage dominating because it compounds over the retention window while compute scales with current usage, not accumulated history.

Forecasting 12-24 months out

Apply a growth rate to daily ingestion and concurrent users (say, a business-projected 20 percent year-over-year data growth and 15 percent analyst-headcount growth) and recompute the same formulas at each future point; the storage term compounds fastest because it's a function of accumulated volume over the growing retention window, not just the current day's ingestion, which is precisely why storage tends to dominate total cost at scale even though its per-unit price is far cheaper than compute.

Validating the model

Compare the model's projected cost for the CURRENT month (using today's actual ingestion volume and concurrency) against the actual current bill; if they diverge by more than a small margin, the unit-cost assumptions are wrong and need recalibrating against real invoices before trusting the 12-24 month projection. A model that isn't first validated against a month you can already observe is not a forecast, it's a guess with a formula attached.

Trade-offs and pitfalls

The most common mistake is projecting compute cost as though it scales with total data volume the way storage does; well-partitioned interactive queries scan a bounded slice of data regardless of how much history is retained, so compute growth is really driven by user count and query complexity, not by retention. Ignoring that distinction produces a model that wildly overestimates future compute cost and underinvests in the partitioning discipline that would keep it flat.

A data mesh decentralizes data ownership to the domain teams that generate it (marketing, orders, payments), each publishing "data products" with clear contracts, while a centralized platform keeps one team owning ingestion, transformation, and the warehouse for the whole company.

The comparison

Ownership: mesh puts the people closest to the data (who understand it best) in charge of its quality and its contract; centralized puts one platform team in charge of everyone's data, whether or not they understand every domain equally well.
Discoverability: mesh requires a strong shared catalog and standardized metadata across domains, or discovery becomes worse than centralized, since data is now scattered across many owners. Centralized discovery is simpler by construction because everything sits in one place.
Governance: mesh uses federated computational governance, shared standards enforced by tooling (schema validation, SLAs as code) rather than a single team manually reviewing everything; centralized governance is easier to enforce consistently because one team controls the whole pipeline, but that team becomes a bottleneck as the company grows.
Latency (time-to-new-data-product): mesh domain teams can ship a new data product without waiting on a central team's backlog; centralized platforms often become the bottleneck once a company has more than a handful of domains competing for the same platform team's attention.
Cost: mesh usually costs more in tooling and duplicated infrastructure across domains; centralized concentrates cost (and the ability to optimize it) in one place.
Developer velocity: mesh scales velocity horizontally (more domains means more parallel capacity) once the platform is mature; centralized velocity is capped by the size of the platform team, and that cap gets worse as the company scales.

When to favor each

Favor a centralized platform when the organization has fewer than roughly a dozen distinct data domains, when the central platform team is not yet a bottleneck, and when the cost and complexity of federated governance tooling would outweigh its benefit. A 50-person company with three product lines rarely needs a mesh; it needs a competent, well-staffed central team.

Favor a data mesh once a large organization has enough independent domains (order of dozens or more) that a central team has become the bottleneck for every new dataset, and once there is executive appetite to invest in the self-serve platform and governance tooling a mesh requires to avoid becoming an ungoverned mess of inconsistent, undiscoverable domain data. A mesh adopted before that tooling investment is made tends to produce exactly the failure mode critics warn about: each domain reinventing its own formats and quality bar, with no way to reliably query across domains.

Trade-offs and pitfalls

The most common mistake is adopting "data mesh" as an organizational restructuring (just move ownership to domain teams) without investing in the platform and governance tooling that makes federation work; the result is worse than a centralized platform, because now nobody is accountable for cross-domain consistency at all. The second is applying mesh principles to an organization too small to need them, which adds coordination overhead (data contracts, domain-team accountability meetings) that a single central team would have handled faster.

A data warehouse stores structured, cleaned data in a schema defined before you write it (schema-on-write), optimized for fast, repeatable SQL queries. A data lake stores data in its raw, often semi-structured or unstructured form in cheap object storage, with the schema applied when you read it (schema-on-read). A lakehouse adds warehouse-like guarantees, ACID (atomicity, consistency, isolation, durability) transactions, schema enforcement, and fast query performance, directly on top of lake storage, so you get one copy of the data serving both BI and machine learning workloads instead of two.

The three side by side

Where to put what, for a mid-size SaaS company at tens of millions of events/day

Raw events: land in object storage (S3 or equivalent) as the system of record, append-only, partitioned by date. This is the lake layer, and it stays cheap even as retention grows to years of history.
Curated BI tables: build these as warehouse tables (or lakehouse tables if you've adopted one) on top of the raw layer, modeled for the specific questions the business asks repeatedly (funnels, revenue, retention). This is where schema-on-write pays off: BI tools expect stable, typed schemas.
ML feature sets: these usually want the row-level, less-aggregated data that lives closer to the raw layer, but with reproducibility and point-in-time correctness that a bare lake doesn't guarantee. This is the strongest argument for adopting a lakehouse table format even before you need full BI-and-ML unification: it gives the ML team versioned, ACID-safe access to data that would otherwise require a second copy of the pipeline.

Trade-offs and pitfalls

The lakehouse's ACID guarantees are not free: someone still has to run compaction and manage table maintenance (small-file cleanup, log/manifest pruning), or the table format's benefits erode over months of high-frequency writes. A pure warehouse is simplest to operate at a small scale but tends to become the wrong long-term choice once machine learning or genuinely unstructured data (logs, images, free text) enters the picture, because forcing that data through schema-on-write either loses information or requires an awkward second raw-storage layer anyway. A common mistake is treating "lakehouse" as a single product decision: it is a pattern implemented via a table format on top of storage you already have, not something you buy instead of a lake.

With a limited budget, the decision between investing in a new analytics platform versus fixing data quality on the current one is really a question of which problem is actually blocking value today, not which one sounds more foundational.

Criteria for the decision

Where is the actual pain currently concentrated? If analysts and stakeholders routinely distrust the numbers they already have (data quality issues), a shinier platform won't fix that; it will just produce untrustworthy numbers faster and with a nicer UI. If the current platform's limitations (poor query performance, no support for a needed data type, an unmaintainable legacy system) are what's actually blocking new use cases, fixing data quality on an outdated platform is treating a symptom while ignoring the ceiling.
What's the cost and time-to-value of each option? Data-quality fixes (adding validation checks, fixing known upstream issues, building monitoring) are typically incremental and can show value within weeks. A platform investment is typically a larger, longer commitment with value realized over months, and during the transition period data quality issues on the OLD platform don't go away, they often get worse as attention shifts.
What's the trend, not just the current snapshot? If data quality issues are getting worse over time despite ad hoc fixes, that's a signal the current platform or process has a structural problem a point fix won't solve, which shifts the calculus toward platform investment even if it costs more up front.

Decision framework over an 18-month horizon

Spend the first quarter on the cheaper, faster option, targeted data quality fixes on the highest-impact known issues, while simultaneously measuring whether those fixes actually reduce the trust and rework problems stakeholders are reporting. If quality measurably improves and stabilizes, defer the platform investment; the current platform, with its issues fixed, may be perfectly adequate for another year or two. If quality issues persist despite genuine fix effort, that is real evidence (not a guess) that the underlying platform has a structural limitation, and the platform investment case is now much stronger and easier to justify to whoever controls the budget.

What would change the decision

A concrete new business requirement the current platform genuinely cannot support (a real-time use case on a batch-only platform, a data volume the current system can't scale to) shifts the decision toward platform investment regardless of data quality's current state, since no amount of quality fixing solves a capability gap. Conversely, discovering that most "quality issues" stakeholders complain about actually trace back to one or two specific, fixable upstream sources (a single unreliable data feed, a single undocumented transformation) shifts the decision toward the cheaper fix, since the problem turns out to be narrower than it first appeared.

Trade-offs and pitfalls

The most common mistake is defaulting to the platform investment because it feels more strategic and impressive to propose, when the actual evidence points to a handful of fixable upstream quality issues. The opposite mistake, endlessly patching data quality on a platform that has a genuine structural ceiling, burns budget on fixes that will need to be redone once the platform is eventually replaced anyway.