Workflow Orchestration and Scheduling Questions

Direct answer

Retraining 50 models daily at roughly 100 million rows each is really a question of what work is genuinely per-model versus shared across models. Treating everything as per-model wastes enormous redundant compute on the shared parts, especially feature generation from largely overlapping source data, while one decomposed pipeline shape, feature generation, validation, train, evaluate, deploy, applied per product line, with shared feature computation memoized and reused, keeps both compute cost and scheduler load proportional to genuinely distinct work rather than to the raw count of 50.

Structured elaboration

DAG decomposition. One dynamically mapped family of per-product-line sub-pipelines, each internally sequenced feature generation, then validation, then training, then evaluation, then deployment, rather than 50 hand-written separate DAGs, a maintenance burden that grows with every new product line, or one giant monolithic DAG, where one product line's failure blocks or confuses the status of the other 49.

Parallelism strategy. The 50 product lines are naturally independent and should run with real concurrency, capped against actual cluster and database capacity rather than one finishing fully before the next starts. Within a single product line's own pipeline, the five stages are strictly sequential, each depends on the one before it, so the real parallelism opportunity is across product lines, not within any one of them.

Caching and memoization. If multiple product lines' feature generation reads overlapping source data or computes overlapping features, a shared demographic feature block used by 30 of the 50 models, for instance, compute and cache that shared block once, keyed by date and by the exact feature-set identity, and have each product line's own feature-generation step read the cached result when its key matches, instead of recomputing the same aggregation against the same roughly 100 million rows 30 separate times.

Resource isolation. Give each product line's pipeline its own pool or quota, or at minimum group product lines into a small number of tiers by resource intensity, so one unusually large or slow product line cannot starve the other 49's capacity. Separate the typically central-processing-unit- and memory-heavy feature-generation and validation stages' resource pool from the typically GPU-heavy training stage's pool, matching the same resource-pool-per-workload-shape pattern used for any heterogeneous compute mix.

Scaling orchestration for team growth. As more product lines and more contributing teams are added over time, the shared feature-computation layer needs its own clear ownership and a documented, versioned contract that other teams' pipelines read from, rather than each new team reimplementing its own copy of the same shared logic. Dynamically mapping over a discovered list of active product lines, instead of a hardcoded list of 50 baked into the DAG's own code, means onboarding product line 51 becomes a configuration or registration change, not a code change, keeping the platform's own maintenance burden roughly flat as the model count grows.

Worked example

30 of the 50 product lines share the same demographic feature block, and computing it requires a full scan of the roughly 100-million-row source table taking 6 minutes. Without caching, that is:

$30 \times 6 = 180 \text{ minutes of redundant compute}$

recomputing the identical result 30 separate times. With caching, computed once and read from cache the other 29 times, the total cost for that shared block drops to 6 minutes, a savings of:

$180 - 6 = 174 \text{ minutes of compute per day}$

compounding daily. The remaining 20 product lines have genuinely distinct feature requirements and are unaffected by this specific caching layer, each still computing its own features independently.

Trade-offs and pitfalls

A shared feature cache introduces a genuine coordination dependency between product lines that did not exist before: if the shared feature block's computation is wrong or delayed, 30 product lines are now affected instead of one, a real blast-radius cost traded against the compute savings above.

A cache key missing a dimension that actually varies, a region-specific version of what looked like a universal feature, for example, can silently serve stale or wrong cached data to some product lines; the cache key has to be genuinely complete for the correctness guarantee to hold, not just keyed on date because that is the obvious dimension.

Treating all 50 product lines as equally important when allocating shared or contended resources ignores that they likely are not; resource isolation tiers should reflect actual business priority, not be uniform by default just because the pipeline shape is uniform.

Direct answer

Cron-like scheduling, periodic sensor polling, and event-driven triggers trade latency against cost and operational complexity, in that order: a fixed schedule is the cheapest and simplest but only as fresh as its interval, a polling sensor narrows that latency at the cost of wasted checks, and an event-driven trigger gets the lowest latency but requires the most infrastructure to wire up. For a new ingestion job, I default to a plain schedule unless there is a concrete freshness requirement the schedule cannot meet, and only reach for event-driven triggering once polling's own overhead becomes the bottleneck.

Structured elaboration

Cron-like schedule_interval. The orchestrator runs the DAG on a fixed cadence regardless of whether the data is actually ready; correctness for a daily ETL relies on the source reliably landing before the scheduled run time, with enough margin built in. Latency is bounded by the interval itself: if data can land anywhere in a 24-hour window and the schedule runs once a day, the worst case is close to a full day's delay. Cost and complexity are both minimal, since there is nothing to poll and nothing to listen for, just a time expression.

Periodic sensor polling. A sensor task repeatedly checks a condition (does this S3 prefix exist, has this API returned a new value) on its own poll interval, and only lets downstream tasks proceed once the condition is true, or gives up after a configured timeout. This narrows the latency window from "up to a full schedule interval" to "up to one poll interval," which matters when the source's actual arrival time varies. The cost is that every poll consumes resources (an orchestrator worker slot in the classic polling mode, or an API call against the source) even on the vast majority of checks that find nothing new; tightening the poll interval to reduce latency directly increases that overhead. Most modern orchestrators mitigate part of this with a reschedule mode that releases the worker slot between polls instead of holding it for the entire wait, trading a bit of scheduling overhead for much better resource usage during long waits.

Event-driven triggers. Instead of the orchestrator repeatedly asking "is it ready yet," the source system (or a piece of glue infrastructure sitting in front of it) pushes a notification the moment the condition becomes true: an S3 event notification, a message on a queue, a webhook call. This gets latency down to close to the event's own occurrence, without the wasted-check cost of polling, but it requires standing up and operating that notification path: a queue or event bus, a listener or webhook endpoint that can trigger a DAG run outside its normal schedule, and monitoring for that new piece of infrastructure itself, since a silently-broken event listener is a much quieter failure than a sensor that visibly times out.

Deciding for a new ingestion job. Ask three questions in order:

1. Does the data arrive on a knowable, fixed schedule? If yes, use a plain schedule; nothing else is justified.
2. If arrival time varies, how much does the resulting delay actually cost the business? If a few hours of possible delay is acceptable, sensor polling with a reasonable poll interval (for example, every 10-15 minutes) is usually the right level of investment.
3. If delay needs to be minutes or less, or the source system already emits events, only then justify the added infrastructure of an event-driven trigger, since building and operating that infrastructure is not free, and it is easy to over-invest in low-latency triggering for a pipeline that feeds a dashboard nobody looks at more than once a day anyway.

Worked example

Two concrete ingestion jobs illustrate the decision:

Daily ETL from a vendor's nightly export. The vendor reliably produces a file by 03:00 UTC every night. A plain schedule_interval of 0 4 * * * (04:00 UTC, one hour of margin) is correct here: the arrival time is known and stable, a sensor would be polling for a file that is virtually always already there by the time it starts checking, and event-driven triggering would add infrastructure to save, at most, an hour of latency on a report nobody consumes before business hours anyway.

S3 file arrival for an ingestion job with variable timing. Upstream files land at unpredictable times within a business day, anywhere from 09:00 to 17:00, and downstream consumers want the data within roughly 30 minutes of arrival. A sensor polling every 5 minutes in reschedule mode bounds latency to about 5 minutes in the worst case, well inside the 30-minute requirement, at a modest cost: over an 8-hour window that is roughly 96 poll checks, most of which find nothing, but each check is cheap (a lightweight existence check against the S3 API) and reschedule mode means the orchestrator is not holding a worker slot idle between checks. If the freshness requirement tightened to under a minute, or if the ingestion volume grew to hundreds of prefixes being watched at once (making 96 x N poll checks a real cost), that is the point where switching to S3 event notifications feeding a queue that triggers the DAG directly would be worth the added infrastructure; at 30-minute tolerance and a handful of prefixes, it is not.

Trade-offs and pitfalls

The most common mistake is reaching for event-driven infrastructure by default because it sounds like the more sophisticated answer, when the actual freshness requirement does not need it; the added queue, listener, and trigger path is a new production system to operate and monitor, and every extra moving part is a new thing that can silently break.

The opposite mistake is running a sensor in classic poke mode (holding a worker slot for the entire wait) at a tight poll interval against a large number of prefixes or sources, which can exhaust the orchestrator's worker pool during a busy window and starve unrelated pipelines, not just the one doing the waiting; reschedule mode (or moving to an event-driven design) is the fix, not just reducing the poll frequency, which only trades latency for the same underlying resource problem at a slower rate.

A subtler pitfall with plain cron scheduling is under-provisioning the margin: scheduling the run at exactly the vendor's typical delivery time, with no buffer, means any day the vendor is even slightly late produces a hard failure instead of a graceful wait, which is really a sensor's job description, not a fixed schedule's.

Direct answer

A 6-month revenue-bug reprocessing plan has to answer seven things before any compute runs: exactly what is affected (scope), how much it costs to fix (compute estimate), how the fix will actually execute without corrupting anything mid-run (partition-level reprocessing), how the team will know the fix is correct before trusting it (validation), how every downstream artifact gets updated consistently (historical dashboards and reports), who needs to know and when (stakeholder communications), and how all of this happens without breaking the systems that depend on this data every day in the meantime (scheduling). Skipping any one of these turns a data-quality incident into a second incident caused by the fix itself.

Structured elaboration

1. Scope identification. Determine precisely which partitions (which days, which specific tables and columns) the bug actually affected, not an approximate range. For a calculation bug, this usually means identifying the exact code change or logic flaw, then determining the first date it was live and whether it affected all revenue records or only a subset (a specific product line, a specific region, a specific transaction type). Getting this wrong in either direction is costly: too narrow misses genuinely affected data, too broad reprocesses correct data unnecessarily, wasting compute and risking new bugs on data that never needed touching.

2. Compute estimates. Once scope is known, estimate total reprocessing cost: number of partitions x per-partition compute cost, plus the downstream tables built on top of the affected data (which also need reprocessing once the base data changes). This estimate is what the scheduling and prioritization decisions in the rest of the plan are actually built on, so it needs to happen before, not after, committing to a specific timeline with stakeholders.

3. Partition-level reprocessing. Reprocess one logical partition (day) at a time, each an independent, idempotent operation (overwrite that day's data, not append), so partial progress is safe: if the reprocessing is interrupted after 90 of 180 days, the table is in a valid state, just with 90 days still reflecting the old bug, not a corrupted intermediate state. Bound concurrency (a max_active_runs or pool limit) so the reprocessing does not contend with the live, ongoing daily pipeline running the same infrastructure.

4. Validation tests. Before committing the full 6-month reprocess, validate a small sample: pick a handful of representative days (including edge cases like month boundaries or unusually high-volume days) and manually verify the corrected revenue figures against an independent calculation or a known-correct reference. Only after that sample passes does the full range proceed. This is the single highest-leverage step for avoiding the worst outcome (reprocessing 6 months with a fix that itself has a bug), and rushing past it under time pressure is the most common and most expensive mistake in incidents like this.

5. Updating historical dashboards and reports. Identify every downstream artifact built on the affected revenue data, not just the base table: aggregation tables, cached dashboard extracts, previously-generated static reports (PDFs or exports that were already sent to stakeholders and will not auto-update). Each needs its own explicit update or reissue plan; a corrected base table with a stale downstream aggregate still built on the old numbers leaves two systems disagreeing, which is often more confusing to a stakeholder than the original bug.

6. Stakeholder communications. Communicate before, during, and after: before, so finance and any other consumer of these numbers knows a correction is coming and roughly when (especially important if externally-reported figures, like a filed report, might be affected and need their own separate correction process); during, if the reprocessing spans multiple days, a status update on progress; after, a clear summary of what changed, by how much, and confirmation the numbers are now final. Coordinating with dashboard owners specifically before publishing corrected metrics, not just notifying them after the fact, avoids a dashboard owner being blindsided by a metric shifting under a report they are actively presenting or distributing.

7. Scheduling to minimize downstream impact. Run the reprocessing during low-traffic windows where possible, throttle concurrency to protect the live daily pipeline's own resource needs, and sequence it to avoid a specific high-visibility moment (a board meeting, a quarter-end close) where the underlying numbers shifting mid-review would be maximally disruptive, even if the fix itself is ready sooner.

Avoiding double counting. A specific, easy-to-miss risk worth calling out on its own: if the reprocessing job or its retries are not carefully scoped to overwrite exactly the affected partitions, a partial rerun or an overlapping manual retry can double-apply the correction, for example if a job is retried after a timeout without confirming whether the first attempt's write had already landed, producing revenue figures that are now wrong in the opposite direction. Guarding against this means the same partition-scoped, idempotent-overwrite discipline from point 3, verified explicitly as part of the validation step in point 4, not assumed.

Worked example

Scope identification confirms the bug affected revenue_by_transaction for the full 180 days (6 months), specifically undercounting revenue on transactions using a specific discount code path, roughly 8% of all transactions, understating total revenue by an average of 3% per affected day (varying with how many discount-path transactions occurred that day).

Compute estimate: 180 daily partitions at roughly 4 minutes of compute each for the base table, plus two downstream aggregation tables (weekly_revenue_summary, monthly_revenue_summary) that also need reprocessing once the daily base changes, adding roughly 26 weekly partitions and 6 monthly partitions at a few minutes each. Total estimated compute: roughly 12-14 hours sequential, planned to run at max_active_runs=4 over 3 nights to stay clear of the live pipeline's own daytime resource needs.

Validation: 5 representative days (including the highest-volume day and a month-boundary day) are manually recomputed against raw transaction logs; all 5 match the corrected pipeline's output within rounding tolerance, and the full 180-day reprocess proceeds.

Reprocessing executes in partition-scoped, overwrite mode, one day at a time within each night's batch, so a night-2 interruption after 60 of the remaining 120 days leaves the table in a valid, if partially corrected, state.

Downstream: weekly_revenue_summary and monthly_revenue_summary are rebuilt for every affected week and month once their underlying daily data is corrected; a cached dashboard extract used by an executive summary tool is manually refreshed, since it does not auto-update from the base tables; two previously-distributed quarterly PDF reports covering the affected period are flagged for reissue with a correction note, coordinated directly with the finance team that originally distributed them.

Communication: finance and the two affected dashboard owners are notified 48 hours before reprocessing begins, with a specific date range and expected completion; a status update goes out after night 1 (60 of 180 days complete); a closing summary states the corrected total revenue impact and confirms all downstream artifacts, including the flagged PDFs, have been addressed.

Trade-offs and pitfalls

Skipping the validation sample under pressure to "just fix it now" is the single most expensive mistake this plan guards against: reprocessing 180 days with a fix that itself has a bug costs far more in compute, stakeholder trust, and a second incident than the 5-day sample check would have cost to catch it first.

Reprocessing the base table but forgetting a downstream artifact, especially an already-distributed static report that will never auto-update, leaves stakeholders looking at inconsistent numbers depending on which artifact they happen to check, which is often more damaging to trust in the data than the original bug, since it looks like the correction itself is unreliable.

Notifying dashboard owners only after publishing corrected metrics, rather than coordinating with them beforehand, risks a dashboard owner being surprised by a shifting number in the middle of presenting it, which is exactly the scenario advance coordination is meant to prevent, and is a real reputational cost distinct from the data correctness itself.

Finally, underestimating double-counting risk from retries or overlapping manual reruns during a multi-night reprocessing effort is a subtle, easy-to-miss failure mode precisely because it produces a plausible-looking (if now wrong in the opposite direction) number rather than an obvious error, making it more likely to slip through a validation pass that only checks "does this look more correct than before" rather than checking the exact expected value.

Direct answer

In Apache Airflow, a DAG (Directed Acyclic Graph) is the static definition of a workflow: a Python object describing a set of tasks and the dependencies between them, with no cycles, so there is always a valid order to execute them in. The DAG object itself is just a blueprint; every time the scheduler decides it is time to run that workflow, it creates a separate DAG run, a live instance with its own state, its own logical time window, and its own set of task instances. Getting comfortable with that distinction, definition versus run, is what makes decisions like catchup=True versus catchup=False make sense, since catchup is really a question about how many DAG runs to create for a definition that already existed in the past.

Structured elaboration

Key components:

• DAG object. Created with DAG(dag_id=..., schedule_interval=..., start_date=..., catchup=...) (or the newer schedule= parameter in current Airflow versions), this is the container that holds the tasks and their dependency structure. It is defined once, in Python, and the scheduler re-parses this file periodically to discover the DAG and its schedule; the DAG object does not itself represent any single execution.
• Tasks and operators. A task is one node in the DAG, a unit of work. An operator is the template that defines what kind of work that task does: PythonOperator runs a Python callable, BashOperator runs a shell command, S3ToGCSOperator copies between two cloud storage systems, and so on. Instantiating an operator with specific arguments and adding it to a DAG produces a task; the same operator class can produce many differently-configured tasks.
• schedule_interval (or schedule in current Airflow). Defines the cadence the DAG is expected to run on: a cron expression (0 6 * * *), a preset (@daily), or a timedelta. This, combined with start_date, determines the sequence of logical time windows the DAG is scheduled to cover.
• execution_date and run_id. execution_date identifies which logical time window a given DAG run represents, not when the run actually executed physically. This is the single most confusing thing about Airflow to a newcomer: a daily DAG whose execution_date is January 1st actually starts running at the end of that interval, effectively January 2nd, because the interval it represents (all of January 1st) has to have finished before the DAG can process it. run_id is the unique identifier for that specific DAG run (for a scheduled run, it typically embeds the execution date). Current Airflow versions (2.2+) have renamed this concept to logical_date, with data_interval_start and data_interval_end marking the actual window explicitly, because "execution date" was a persistently confusing name for something that is not when execution happens; the underlying concept is the same, and both names are common to see depending on the Airflow version a codebase was written against.

DAG definition vs. DAG run. The DAG definition is the code: it exists once, is version-controlled, and describes the shape of the workflow and its intended schedule. A DAG run is created by the scheduler each time a scheduled interval completes (or is manually triggered), and it is a stateful object: it has a status (running, success, failed), a logical date identifying which interval it covers, and a collection of task instances, one per task in the definition, each with its own state. Multiple DAG runs for the same DAG definition can exist concurrently (a normal daily run plus a manually triggered backfill run for a past date, for example), each independently tracking its own task states, all governed by the same underlying task graph.

catchup=True vs. catchup=False. This setting controls what happens when a DAG's start_date is in the past relative to when the DAG is first deployed or enabled (or after it has been paused for a while):

• catchup=True (the historical Airflow default): the scheduler creates a DAG run for every scheduled interval between start_date and now that has not already run, in order. Deploying a daily DAG with a start_date 30 days ago and catchup=True produces 30 sequential DAG runs, each for its own logical day, effectively backfilling the DAG's entire history automatically.
• catchup=False: the scheduler skips all past intervals and only creates a DAG run for the most recent completed scheduled interval going forward. The same 30-day-old DAG with catchup=False produces a single DAG run, for the most recent completed interval, and none of the 29 earlier days are ever automatically run.

catchup=False is the far more common choice in practice for pipelines where historical backfilling should be a deliberate, reviewed action (using the explicit backfill command or UI trigger) rather than something that happens automatically and silently the moment a DAG is deployed or re-enabled after being paused, since an accidental 30-run catchup against a shared downstream resource is a common and disruptive real-world incident.

Worked example

Consider a small ML pipeline DAG with five tasks: ingest_data, validate_data, extract_features, train_model, evaluate_model, each depending on the one before it.

graph LR
    A[ingest_data] --> B[validate_data]
    B --> C[extract_features]
    C --> D[train_model]
    D --> E[evaluate_model]

This is the DAG definition: five PythonOperator tasks, a linear dependency chain, schedule_interval="@daily", start_date set to 10 days ago, catchup=False.

• DAG definition: exists once in the repository, unchanged whether it has run zero times or a thousand times.
• DAG runs: because catchup=False, deploying this today produces exactly one DAG run, for the most recent completed daily interval (yesterday, since today's interval has not finished yet). That run has its own run_id, its own execution_date (or logical_date) marking which day it covers, and five task instances, one per task, each independently tracked as running, success, or failed. Tomorrow, the scheduler creates a second, separate DAG run for the next completed interval; the DAG definition itself does not change between these runs, only the run-level state does.
• If this same DAG had been deployed with catchup=True instead, that first deployment would have immediately created 10 DAG runs, one for each of the past 10 days, and the scheduler would work through them (by default sequentially, respecting max_active_runs) in addition to the current day going forward. If train_model is an expensive GPU job, that means 10 unplanned expensive runs firing back-to-back the moment the DAG is turned on, which is exactly the kind of surprise catchup=False is meant to prevent for a workflow like this.

Trade-offs and pitfalls

The most common beginner confusion is treating execution_date as "when the code ran." It is not; it is the logical interval the run represents, and for a daily DAG, that means the run for a given day only starts after that day has fully elapsed. Code that reads execution_date and assumes it means "now" (for example, fetching "today's" data using execution_date inside a task meant to process yesterday's completed interval) is a frequent source of off-by-one-day bugs.

A second pitfall is leaving catchup at its default without thinking about it. The historical default is True (governed by the catchup_by_default config setting), so a newly deployed DAG with a start_date set far in the past, or a DAG that gets unpaused after sitting disabled for weeks, can silently trigger a large, unplanned batch of historical runs against live downstream systems. Explicitly setting catchup=False for pipelines that should never auto-backfill, and reserving intentional backfills for the explicit backfill tooling, avoids this class of incident entirely.

A third, more architectural trade-off worth naming: whether to model a workflow like the one above as one large DAG with many tasks, or as several smaller DAGs chained together. One large DAG (as shown) gives a single, clear view of the whole dependency chain and simple cross-task data passing, but a failure or a slow task anywhere in it holds up the entire pipeline's status, and it becomes harder for different owners to iterate on different parts independently. Splitting into smaller DAGs (an ingestion DAG, a feature DAG, a training DAG, connected by sensors or dataset-based triggering) gives each stage independent scheduling, ownership, and failure isolation, at the cost of needing an explicit cross-DAG dependency mechanism instead of Airflow's built-in same-DAG task dependencies. For a small five-task pipeline owned by one team, a single DAG is usually the right call; the split becomes worth it once different stages have genuinely different schedules, owners, or blast-radius requirements.

Direct answer

The architectural difference underneath every practical difference between the two: Airflow models a pipeline as a graph of tasks (units of execution), while Dagster models it as a graph of assets (the actual data objects a pipeline produces), with tasks as the mechanism that materializes them. That single choice ripples into developer experience, observability, testing, and deployment. Recommend Airflow when the team's mental model is genuinely task-centric (orchestrate arbitrary steps, many of which are not really "data assets" per se) and when the ecosystem's breadth of existing integrations matters most; recommend Dagster when the pipeline's real unit of value is the data it produces, and being able to reason about, test, and monitor at that level pays off, which is common for feature pipelines and other ML-adjacent data products.

Structured elaboration

Developer experience, in more depth. Airflow's DAG-as-Python-file model is simple to start with and has an enormous operator ecosystem (a pre-built integration for nearly any system a pipeline might need to talk to), but that same breadth means operator quality and API consistency vary across providers, since they are maintained by many different contributors. Dagster's asset-first API is more opinionated and type-aware from the start, which raises the initial number of concepts a new user has to learn (assets, ops, resources, jobs, schedules, sensors are all distinct, related concepts), but tends to produce more consistent, more testable code once a team is past that initial learning curve.

Observability, in more depth. Airflow's UI is built around DAG runs and task instances: it answers "did this task succeed, and when." Dagster's UI is built around the asset graph: it answers "is this specific piece of data fresh, and what upstream asset is stale or missing if it's not," which is a more directly useful question for a data consumer who cares about a specific table or feature set, not about which internal tasks happened to produce it.

Dataset/type awareness. This is the core architectural distinction stated above, made concrete: in Dagster, an asset can carry a declared type, and Dagster can check that what a downstream asset receives actually matches what it expects, catching a class of bug (a schema or shape mismatch between pipeline stages) that Airflow's task-centric model has no native mechanism to catch, since Airflow tasks pass data through XCom or external storage with no built-in typing.

Testing support. Both frameworks let you unit-test the business logic inside a task or op as plain Python, which is the majority of what actually needs testing. The difference is at the pipeline-structure level: Dagster's asset graph is a first-class object that can be validated and partially executed in tests (materialize just this asset and its direct dependencies, in isolation), while testing an Airflow DAG's structure (are the dependencies wired correctly, does the schedule behave as expected) typically requires more custom test scaffolding, since the DAG object itself is not designed around being partially executed for testing.

Deployment model. Airflow has the larger, more mature ecosystem of both managed offerings and self-hosted deployment patterns, reflecting its longer track record and larger install base; teams needing a specific compliance posture, a specific cloud provider's managed service, or deep operational familiarity already present on the team often lean this way by default. Dagster's deployment ecosystem (Dagster Cloud, plus self-hosted) is smaller but has matured significantly, and Dagster's software-defined-assets model is a genuine design differentiator, not just a smaller Airflow.

Worked example

A general-purpose ETL platform team, many source systems, mostly straightforward extract/load/transform sequencing, existing Airflow expertise on the team. Recommend Airflow: the pipeline's actual complexity is in sequencing many heterogeneous sources, which Airflow's operator ecosystem directly supports, and the team's existing operational familiarity with Airflow is a real, non-trivial advantage that a rewrite in a different tool would spend real time and risk to obtain only a marginal architectural benefit for a workload that is not particularly asset-model-shaped.

A machine learning feature pipeline: raw events to engineered features to a feature store, consumed by multiple downstream training jobs, where "is this feature fresh and correctly typed" is the actual question data scientists ask daily. Recommend Dagster: the pipeline's real unit of value is explicitly the data assets (each engineered feature), not the tasks that produce them, and Dagster's asset-centric observability directly answers the question data scientists actually have ("is feature_x fresh as of today, and does it match the schema training expects") without them needing to translate from "which tasks succeeded" to "is my feature usable." The type-awareness also catches a real, common ML pipeline failure mode early: a feature whose shape or type silently drifted between the pipeline and what a training job expects, which Dagster's typed asset graph can surface as a build-time or run-time check rather than a downstream training failure discovered much later.

Trade-offs and pitfalls

Choosing Dagster purely because "asset-based" sounds like the more modern architecture, for a pipeline that is genuinely task-shaped (many heterogeneous, loosely-related steps with no strong shared data-asset identity), adds conceptual overhead without a corresponding benefit; the asset model earns its complexity specifically when the pipeline's actual product is a well-defined set of data assets, not universally.

Staying on Airflow purely out of inertia for a pipeline that is genuinely asset-shaped (a feature pipeline, a data product with many downstream consumers who care about freshness and type correctness) means building, by hand, observability and validation that Dagster provides natively, which is a real, recurring engineering cost paid on every such pipeline, not a one-time migration cost avoided.

Finally, underestimating Airflow's ecosystem breadth when evaluating Dagster is a common mistake in the other direction: a source system with a mature, well-tested Airflow provider but no equivalent first-class Dagster integration can mean building custom integration code in Dagster that Airflow would have gotten for free, which is a real cost that should be weighed against the asset-model benefits on a case-by-case basis, not assumed away.