Data Platform Architecture and Technology Selection Questions

When compute and storage costs on a data platform have grown too high, the fix has to separate three distinct cost drivers, wasteful storage tiering, wasteful compute usage patterns, and wasteful query design, because each has a different remedy.

Storage tiering

Move data that is rarely queried (older than, say, 90 days) from hot, expensive storage to a cheaper warm or cold tier, and delete or archive data past its actual retention requirement rather than keeping everything "just in case." This is usually the single largest and least risky lever, because it doesn't touch how anyone queries the data, only where it physically sits.

Materialized views and caching

If ad-hoc queries or dashboards repeatedly recompute the same aggregation from raw data, materializing that aggregation once (refreshed on a schedule matching how fresh the answer actually needs to be) turns many expensive scans into one cheap scan plus many cheap reads of the precomputed result. A result cache in front of the BI layer catches the common case of the exact same query being run many times a day by different dashboard viewers.

Compute autoscaling

For scheduled ETL (extract, transform, load) jobs on transient clusters, right-size the cluster to the actual job (a job that historically finishes with cluster utilization at 30 percent is over-provisioned) and ensure clusters actually shut down between runs rather than idling. For interactive query compute, autoscale down aggressively outside business hours rather than running a fixed-size warehouse around the clock.

Architectural changes

If ad-hoc analysts are scanning raw, unpartitioned files directly, partitioning and clustering the underlying tables on the columns those queries actually filter on (date, region, customer segment) is often the highest-leverage architectural fix, because it reduces bytes scanned per query regardless of who runs it or how often.

Worked example

Say ad-hoc queries against a raw events table currently scan the full table (say, 10 TB) on every query because it isn't partitioned by date, and analysts typically only care about the last 30 days. Partitioning by date so a typical query only scans the relevant 30-day slice (say, roughly 300 GB, if daily volume is even) cuts bytes-scanned-per-query by roughly 97 percent for that common access pattern, which for a byte-scanned pricing model translates almost directly into a proportional cost reduction on that workload, without changing a single line of the analyst-facing query.

Preserving query performance while cutting cost

The order of operations matters: apply storage tiering and partitioning first (these reduce cost without touching latency, and often improve it), then materialize the specific hot aggregations dashboards depend on (this improves latency while cutting cost), and only then look at autoscaling policy (which affects cost directly but needs headroom tuning to avoid introducing cold-start latency during traffic spikes). Cutting compute capacity before fixing the underlying query patterns just makes the same wasteful queries slower without actually reducing waste.

Trade-offs and pitfalls

The most common mistake is reaching for aggressive autoscaling or smaller compute allocations as the first lever, because it's the fastest change to make, without first fixing the partitioning and materialization issues that are the actual source of wasted spend; this trades a cost problem for a latency problem instead of solving either. The second is archiving or deleting data based on age alone without checking whether a specific, infrequently-run but business-critical report (an annual compliance report, say) still depends on it.

Direct answer

Evaluating an ML platform like Kubeflow, SageMaker, or Vertex AI for org-wide adoption needs a structured framework spanning technical fit (does it actually support the org's training/serving workloads), operational impact (what changes for the teams that adopt it, including migration cost), cost (both direct platform cost and the vendor lock-in risk of committing to it), data governance (does it meet the org's data-handling and compliance requirements), and developer experience (will teams actually want to use it), evaluated through a bounded pilot before committing to organization-wide rollout.

Structured elaboration

• Technical criteria: does the platform support the org's actual training workload shapes (distributed training at the needed scale, the specific frameworks/libraries in use, any custom infrastructure needs like specialized hardware); a platform that fits a typical use case well but poorly supports the org's specific heaviest workloads is a real risk that generic feature-comparison checklists can miss.
• Operational impact: what existing tooling/workflows change for teams (does it replace or need to integrate with existing CI/CD, experiment tracking, and monitoring the org already has), and what the migration effort looks like for teams currently on other infrastructure.
• Cost and vendor lock-in risk: direct platform/usage cost at projected scale, plus the harder-to-quantify cost of being locked into a specific vendor's ecosystem (how hard would it be to migrate away later if needed, and does the platform use genuinely portable underlying technology like Kubernetes/open standards, or proprietary abstractions that make later migration expensive).
• Data governance: does the platform meet the org's data residency, access-control, and compliance requirements (relevant especially for regulated industries or sensitive data), and how mature is its audit-logging and permission model.
• Developer experience: would the teams who'd actually use this daily find it genuinely better than their current setup, evaluated through direct hands-on trial by representative users, not just a feature checklist reviewed by the evaluating team alone.
• Pilot plan with success metrics: run a bounded pilot with one or two representative teams and workloads, defining success metrics up front (e.g. reduction in time-to-first-training-run, reduction in infrastructure-debugging time, developer satisfaction survey results) and a clear go/no-go decision point based on the pilot's actual results, before committing to the cost and disruption of an organization-wide rollout.

Worked example

A pilot selecting one team with a representative (not the org's most unusual/edge-case) training workload, running that team's real training pipeline on the candidate platform for a defined 6-week period, tracking time-to-first-successful-training-run, any workload-specific technical gaps discovered, direct cost incurred, and a structured developer-experience survey from the pilot team at the end; a go decision requires the pilot to meet pre-defined thresholds on these metrics (e.g. time-to-first-run improved or at least not regressed, no blocking technical gaps found, developer satisfaction above a defined bar), not just "the pilot team didn't complain."

Trade-offs & pitfalls

The most common evaluation-framework failure is skipping the hands-on pilot and deciding based on a feature-comparison document alone; vendor marketing materials and documentation reliably understate real integration friction and workload-specific gaps that only surface once real engineers try to run real workloads on the platform, which is exactly what a bounded pilot with pre-defined success metrics is designed to surface before the org commits at scale.

Moving off an on-prem Hadoop ecosystem (HDFS, Hive, Impala) to a cloud-native lakehouse is a bigger structural change than a warehouse-to-warehouse migration, because you're changing the storage substrate itself, not just the query layer on top of it.

The migration plan

Data replication: copy HDFS data to object storage (S3 or equivalent) in bulk first, then set up ongoing incremental replication (or dual-write from ingestion) to keep the object-storage copy current while the old system stays live. Adopt a table format, Apache Iceberg or Delta Lake, on top of that object storage as you land the data, rather than migrating raw files first and adding transactional guarantees later as a second project.
Reconciliation: for each migrated table, validate row counts and key aggregate sums against the Hive source, on a defined cadence, for the full parallel-run period, since silent data-type or encoding mismatches between Hive's format and the new table format are a common source of subtle corruption that a one-time count check would miss.
Cutover strategy: migrate by workload, not by a single flag day. Move less business-critical batch jobs first to validate the new stack's Spark/Presto access patterns under real load, then move BI-facing tables, and migrate the most business-critical (or highest-volume) tables last, once the process is proven.
Rollback: keep the Hadoop cluster running in read-only mode as a fallback source of truth through at least one full business cycle (commonly a full month-end close) after the primary workload has moved, since month-end reporting is often where subtle migration bugs first surface.

Keeping teams productive during the transition

For a reporting team, this means giving them working access to the SAME curated tables in both systems during the transition, with an agreed single source of truth (usually the old system, until validation is complete) so daily reporting never actually breaks, even while the migration is in flight. For a model-training team, this means specifically prioritizing the migration of TRAINING datasets early and validating that model outputs trained on the new platform's data match outputs trained on the old platform's data within an acceptable tolerance, since silently reproducing slightly different training data would corrupt every downstream model without an obvious symptom.

Worked example

Say the Hadoop cluster stores a customer-transactions Hive table partitioned by date. The migration writes that table into an Iceberg table on object storage, partitioned the same way. During the parallel-run period, a daily job compares the sum of transaction amounts and the row count, by partition, between the Hive table and the Iceberg table; any partition where the two disagree by more than a defined small tolerance gets flagged and investigated before that partition is trusted in the new system. Only once 30 consecutive days pass with zero discrepancies does that table's consumers get switched over.

Trade-offs and pitfalls

The most common mistake is treating this purely as an infrastructure lift-and-shift and underestimating how much Hive-specific tooling (custom UDFs, Oozie workflow definitions, Impala-specific SQL extensions) has to be rewritten, not just pointed at a new location. The second is skipping a genuine parallel-run reconciliation period to hit a migration deadline, which trades a visible, bounded near-term cost (a longer migration timeline) for an invisible, unbounded long-term cost (silently wrong numbers discovered months later).

The core design choice is a federated operating model with central guardrails, not full centralization: domain teams own their data and remain accountable for its quality, while a small central governance function owns the shared standards, tooling, and the small set of policies that must be consistent everywhere (naming conventions for canonical entities, classification tiers, the schema-contract format, the escalation path for a cross-domain incident). This mirrors how a data-mesh migration frames domain-level responsibility with central guardrails: the mesh doesn't eliminate governance, it moves enforcement out of a central team's backlog and into each domain's own pipeline, backed by shared infrastructure the central team maintains.

Operating model

• Domain teams: own their source tables, their producer contracts, and first-line triage of quality incidents in their own data.
• Central governance function: owns the policy-as-code framework, the catalog, the classification standard, and cross-domain incident coordination. It sets the bar; it does not review every table.
• Federation boundary: a domain team can ship changes freely as long as they pass the shared automated gates; anything that would break a documented cross-domain contract requires the same CI-enforced compatibility check as any producer contract, not a manual sign-off meeting.

Technical controls and organizational changes

1. Ownership assignment. Every dataset with downstream consumers gets a named owning team in the catalog, not a person (people leave, teams persist). This is the single highest-leverage change: an unowned incident cannot be triaged.
2. Runbooks. Each owning team maintains a runbook for their domain's common failure modes (a stalled ingestion job, a schema-drift alert, a null-rate spike) so on-call response does not depend on institutional memory. Runbooks are checked into the same repo as the pipeline and reviewed on the same cadence as the code.
3. Policy-as-code. Classification rules, retention windows, and compatibility rules are expressed as machine-checkable policy (for example, Open Policy Agent rules or a custom linter run in CI) rather than a wiki page, so a violation fails a build instead of being caught in a quarterly audit.
4. Quality gates. Automated checks (freshness, null rate, referential integrity, schema compatibility) run on every load and block publication of a failing batch. This is what actually prevents incidents rather than just detecting them faster.

Concrete rollout: ownership plus runbooks as the first program elements, over 12 months, by vertical

Rather than rolling out to "dozens of teams" simultaneously, sequence by business vertical so the program proves itself before it scales. A realistic first-year plan:

• Months 1-3: pick one vertical, for example the CRM (customer relationship management) domain, as the pilot. Assign explicit ownership for every CRM-sourced table with downstream consumers, write the first runbooks for its two or three most common incident types, and stand up the quality gates on ingestion. This is deliberately narrow so the pattern can be validated before it is asked of teams who did not choose it.
• Months 4-6: extend the same ownership-assignment-plus-runbook pattern to a second vertical with materially different risk profile, for example finance data (where compatibility breaks have compliance consequences, not just dashboard noise). This tests whether the guardrails generalize or were accidentally tuned to CRM's shape of problem.
• Months 7-9: roll the same pattern to the remaining domains in waves of a handful of teams at a time, reusing the policy-as-code rules and runbook templates built in the first two verticals rather than writing new tooling per team.
• Months 10-12: the central function shifts from active rollout to maintenance: quarterly policy review, an incident retrospective process, and a self-service onboarding path so a new team can adopt the standard without a central-team engagement.

What to measure to know it's working, not just running

Trade-offs and pitfalls

The biggest risk in this design is the central function drifting into a review bottleneck by accident: every "just this once" manual sign-off makes the next domain team expect the same, and the whole point of policy-as-code is to make the guardrail a build check, not a meeting. The other common failure is measuring activity instead of outcomes (number of runbooks written, number of tables catalogued) without ever checking whether incident counts actually fell; a program can look busy and still not be reducing the thing it was funded to reduce.

Lambda, Kappa, and pure-batch are three different answers to the same tension: how do you get correct results, quickly, without building the same logic twice.

The three architectures

Which fits a fintech workload needing strict correctness and sub-minute updates

Kappa is usually the better starting point here, not Lambda, despite Lambda's reputation as the "correctness" architecture. The reason is specific to this requirement: a fintech workload needs strict correctness AND sub-minute freshness, and Lambda's whole premise is that the fast path is allowed to be approximate until the batch path catches up, which is the opposite of what "strict correctness" demands. Kappa avoids maintaining two divergent codebases by making the single streaming pipeline capable of both live processing and full reprocessing (replaying the event log from the beginning when a bug fix or a definition change requires it), which keeps live and historical numbers consistent because they run through identical logic. The trade-off is that Kappa demands real engineering discipline: exactly-once processing guarantees, careful state management, and an event log retained long enough to support full reprocessing.

Pure batch is only viable here if "sub-minute updates" turns out, on closer questioning, to not actually be a hard requirement, since pure batch simply cannot deliver it by construction.

Trade-offs and pitfalls

Lambda's biggest real-world failure mode is exactly what it's often praised for avoiding: the two paths (streaming approximation, batch ground truth) computing the same business logic independently, in two different codebases, with two different engineers maintaining them, and slowly disagreeing on edge cases (how to handle a late-arriving refund, for instance) until nobody trusts either number without manually reconciling. Kappa avoids that specific failure mode by construction but shifts the burden onto the maturity of the streaming engine and the team's comfort with exactly-once semantics; adopting Kappa without that operational maturity in place just relocates the correctness risk rather than removing it.