Data Platform Architecture and Technology Selection Questions

A modern end-to-end analytics platform has roughly eight logical components, and naming a real tool at each layer is what separates a memorized list from a working mental model.

The components

Why these eight and not fewer

A platform that skips the raw layer and transforms directly on ingestion loses the ability to replay history when a transform bug is found. A platform that skips the catalog works fine at 20 tables and becomes undiscoverable at 500. The orchestration and monitoring layers are often the two people forget to name first, but they are what make the difference between "a pipeline" and "a platform": without them you have a collection of scripts that happen to run, with no visibility into whether they ran correctly.

Worked example

For a 15-person startup with one data engineer: a Fivetran-style connector or a simple Python script for ingestion, S3 for raw storage, dbt running inside a serverless warehouse (BigQuery on-demand) for transformation and the curated layer, the warehouse plus a light BI tool for serving, a managed orchestrator (Cloud Composer) or even just dbt Cloud's own scheduler if the DAG is simple, warehouse query-history alerts plus a handful of dbt tests for monitoring, and the warehouse's native catalog rather than a standalone tool. Every layer is present, but three of the eight are riding on a single managed product rather than a dedicated tool, which is the correct trade-off at this scale.

Trade-offs and pitfalls

The temptation at small scale is to skip orchestration and monitoring because "the pipeline is simple enough to just run on a cron job." That works until a job silently fails at 3am and nobody notices for a week, which is exactly the failure mode a thin orchestrator plus alerting is built to catch. The opposite failure, at larger scale, is standing up a dedicated best-of-breed tool for every layer before there is a team big enough to operate eight different systems; a platform team of two people cannot realistically run Kafka, Airflow, a standalone catalog, and a standalone observability tool all at once, so consolidating onto the warehouse's native features for two or three of these layers is often the more honest answer than naming eight separate best-of-breed products.

Moving off an on-prem Hadoop ecosystem (HDFS, Hive, Impala) to a cloud-native lakehouse is a bigger structural change than a warehouse-to-warehouse migration, because you're changing the storage substrate itself, not just the query layer on top of it.

The migration plan

Data replication: copy HDFS data to object storage (S3 or equivalent) in bulk first, then set up ongoing incremental replication (or dual-write from ingestion) to keep the object-storage copy current while the old system stays live. Adopt a table format, Apache Iceberg or Delta Lake, on top of that object storage as you land the data, rather than migrating raw files first and adding transactional guarantees later as a second project.
Reconciliation: for each migrated table, validate row counts and key aggregate sums against the Hive source, on a defined cadence, for the full parallel-run period, since silent data-type or encoding mismatches between Hive's format and the new table format are a common source of subtle corruption that a one-time count check would miss.
Cutover strategy: migrate by workload, not by a single flag day. Move less business-critical batch jobs first to validate the new stack's Spark/Presto access patterns under real load, then move BI-facing tables, and migrate the most business-critical (or highest-volume) tables last, once the process is proven.
Rollback: keep the Hadoop cluster running in read-only mode as a fallback source of truth through at least one full business cycle (commonly a full month-end close) after the primary workload has moved, since month-end reporting is often where subtle migration bugs first surface.

Keeping teams productive during the transition

For a reporting team, this means giving them working access to the SAME curated tables in both systems during the transition, with an agreed single source of truth (usually the old system, until validation is complete) so daily reporting never actually breaks, even while the migration is in flight. For a model-training team, this means specifically prioritizing the migration of TRAINING datasets early and validating that model outputs trained on the new platform's data match outputs trained on the old platform's data within an acceptable tolerance, since silently reproducing slightly different training data would corrupt every downstream model without an obvious symptom.

Worked example

Say the Hadoop cluster stores a customer-transactions Hive table partitioned by date. The migration writes that table into an Iceberg table on object storage, partitioned the same way. During the parallel-run period, a daily job compares the sum of transaction amounts and the row count, by partition, between the Hive table and the Iceberg table; any partition where the two disagree by more than a defined small tolerance gets flagged and investigated before that partition is trusted in the new system. Only once 30 consecutive days pass with zero discrepancies does that table's consumers get switched over.

Trade-offs and pitfalls

The most common mistake is treating this purely as an infrastructure lift-and-shift and underestimating how much Hive-specific tooling (custom UDFs, Oozie workflow definitions, Impala-specific SQL extensions) has to be rewritten, not just pointed at a new location. The second is skipping a genuine parallel-run reconciliation period to hit a migration deadline, which trades a visible, bounded near-term cost (a longer migration timeline) for an invisible, unbounded long-term cost (silently wrong numbers discovered months later).

Apache Iceberg, Delta Lake, and Apache Hudi all solve the same core problem, adding ACID (atomicity, consistency, isolation, durability) transactions and schema management to files sitting in object storage, but they were built with different primary use cases in mind, and that history still shows in where each is strongest today.

The three compared

Recommendation for this scenario

For a lakehouse needing multi-engine reads (Spark AND Presto/Trino specifically) alongside consistent upserts from concurrent writers at petabyte scale, Apache Iceberg is the strongest default, because multi-engine neutrality is explicitly one of the requirements here, and Iceberg has the broadest, most mature support across exactly the two engines named. If the stack were exclusively Spark-based with no requirement for Presto/Trino access, Delta Lake would be an equally reasonable choice given its deep native Spark integration. If the dominant workload were high-frequency CDC-style upserts (replicating a constantly-changing transactional source) rather than periodic batch writes, Apache Hudi's purpose-built upsert design would be worth weighing against Iceberg's more general-purpose approach, even at some cost to multi-engine breadth.

Trade-offs and pitfalls

None of these formats eliminates the operational work a lakehouse still requires: compaction (merging small files produced by frequent writes) and metadata/snapshot pruning still need to run regularly, or query performance degrades over months of high-frequency writes regardless of which format you picked. A common mistake is choosing a table format based on which one a specific team already knows, rather than which one matches the actual read-pattern requirements (multi-engine breadth versus deep single-engine integration versus upsert frequency); that choice is usually reversible early on, but becomes expensive to change once petabytes of data and dozens of downstream consumers depend on it.

A data warehouse stores structured, cleaned data in a schema defined before you write it (schema-on-write), optimized for fast, repeatable SQL queries. A data lake stores data in its raw, often semi-structured or unstructured form in cheap object storage, with the schema applied when you read it (schema-on-read). A lakehouse adds warehouse-like guarantees, ACID (atomicity, consistency, isolation, durability) transactions, schema enforcement, and fast query performance, directly on top of lake storage, so you get one copy of the data serving both BI and machine learning workloads instead of two.

The three side by side

Where to put what, for a mid-size SaaS company at tens of millions of events/day

Raw events: land in object storage (S3 or equivalent) as the system of record, append-only, partitioned by date. This is the lake layer, and it stays cheap even as retention grows to years of history.
Curated BI tables: build these as warehouse tables (or lakehouse tables if you've adopted one) on top of the raw layer, modeled for the specific questions the business asks repeatedly (funnels, revenue, retention). This is where schema-on-write pays off: BI tools expect stable, typed schemas.
ML feature sets: these usually want the row-level, less-aggregated data that lives closer to the raw layer, but with reproducibility and point-in-time correctness that a bare lake doesn't guarantee. This is the strongest argument for adopting a lakehouse table format even before you need full BI-and-ML unification: it gives the ML team versioned, ACID-safe access to data that would otherwise require a second copy of the pipeline.

Trade-offs and pitfalls

The lakehouse's ACID guarantees are not free: someone still has to run compaction and manage table maintenance (small-file cleanup, log/manifest pruning), or the table format's benefits erode over months of high-frequency writes. A pure warehouse is simplest to operate at a small scale but tends to become the wrong long-term choice once machine learning or genuinely unstructured data (logs, images, free text) enters the picture, because forcing that data through schema-on-write either loses information or requires an awkward second raw-storage layer anyway. A common mistake is treating "lakehouse" as a single product decision: it is a pattern implemented via a table format on top of storage you already have, not something you buy instead of a lake.

The build-versus-buy question is really an evaluation-framework question: what does it cost to build and maintain this ourselves, what does it cost to adopt a managed solution, and which risk profile fits the team's actual skills and time horizon.

The evaluation framework

Total cost of ownership over 3+ years, not just sticker price: for building, this means engineer salary time for initial build plus ongoing maintenance, infrastructure costs, and the opportunity cost of that engineering time not going toward the product. For buying, this means license or usage-based fees plus the (often underestimated) integration, migration, and customization work.
Vendor lock-in risk: how hard would it be to leave this vendor later. Standard, open interfaces (SQL, open table formats, exportable metadata) are low risk; a proprietary API that "traps" your metadata or code is high risk.
Time-to-value: building takes months before the first real user benefits; a managed solution can often deliver value in weeks, which matters when the business need is urgent.
Team skills and headcount: building and then operating a component like a data catalog or an orchestrator requires ongoing engineering attention, not just an initial build. A team without a dedicated platform engineer is signing up for perpetual part-time maintenance if it builds.

Worked example: build or buy a data catalog

Say a 500-person company with mixed data maturity is deciding whether to build a lightweight internal catalog (a service plus a UI reading table metadata from the warehouse) or adopt a managed catalog product. Building might cost an estimated 2 engineers for 3 months to reach a usable first version (roughly half an engineer-year), then an ongoing 20 to 30 percent of one engineer's time to maintain and extend it as new data sources are added. Buying costs a recurring subscription fee, but reaches a usable state in weeks rather than months, and the vendor absorbs the maintenance burden of keeping up with new source-system integrations. If the company has no dedicated platform team and catalog needs are fairly standard (ownership, lineage, search), buying wins on time-to-value and total cost once you count engineer time honestly. If the company's catalog needs are unusual (deep integration with a proprietary internal system no vendor supports out of the box), building may be the only realistic option regardless of cost.

How to pilot before committing

Run a bounded pilot, a single team or a handful of high-traffic datasets, on the leading option (build a minimal version, or trial the vendor) for a fixed period, with a small number of concrete success criteria defined up front: does search actually return the right table in under a specified number of clicks, does the team's satisfaction improve, does the pilot surface integration gaps the evaluation missed. A pilot without predefined success criteria tends to be judged only by whether it "felt fine," which doesn't generalize to the full rollout decision.

Trade-offs and pitfalls

The single biggest analytical mistake in these evaluations is comparing the vendor's subscription price directly against a build estimate that only counts initial development time, ignoring the ongoing maintenance a build requires. The second is treating "no lock-in" as an absolute good: a small team may genuinely benefit from accepting some vendor dependency in exchange for never having to think about that component again, if the vendor uses reasonably open interfaces underneath.