Workflow Orchestration and Scheduling Questions

Direct answer

Three broad trigger families cover almost every way a machine learning (ML) pipeline gets started: time-based scheduling (a cron or nightly clock), data-availability triggers (a new file or partition landing), and event-driven triggers (a message on a broker). Each one trades detection latency against operational complexity, so choosing between them is really a decision about how tightly the pipeline's start needs to track the actual arrival of new data versus how simple the scheduling should stay.

Structured elaboration

On detection latency, qualitatively. As a general ordering, not a measured claim about any specific deployment: event-driven triggers typically detect new data within seconds of it existing, since the message itself is the signal; data-availability sensors are bounded by their poll interval, typically minutes; time-based triggers are bounded by the schedule's own granularity, typically hours for a nightly job. This ordering is why event-driven and data-availability triggers both exist as alternatives to a plain clock in the first place, not because a clock is a worse mechanism in general, but because it cannot react faster than its own configured interval by construction.

Worked example

A team runs a daily fraud-detection retraining pipeline that currently uses a nightly cron trigger at 03:00. They notice that on roughly one day in ten, the upstream labeled-events file is not fully written until 03:20, so the 03:00 run silently trains on a stale, incomplete file without erroring. Switching to a data-availability trigger, a sensor watching for the specific file plus a completion marker (a zero-byte _SUCCESS file written only after the upstream job finishes), removes the race entirely: the pipeline now starts the moment the real data is ready, whether that happens to be 02:50 or 03:35 on a slow day, instead of on a fixed clock that assumes the data is always ready by 03:00. If the team later adds a requirement to retrain incrementally through the day as new fraud labels arrive, rather than once nightly, they would layer an event-driven trigger on top, consuming a labeled-event stream directly, since neither the clock nor the daily-file sensor can react at that granularity.

Trade-offs and pitfalls

Combining triggers is common and often the right answer, not a sign of indecision: a data-availability trigger paired with a cron-based SLA-miss fallback catches the case where the upstream file never arrives at all, something the availability trigger alone cannot detect on its own since it simply waits forever with no complaint.

Treating "the file exists" as equivalent to "the file is complete" is one of the most common data-availability-trigger bugs; a completion marker or an explicit size/row-count check is needed alongside existence, not existence alone.

Event-driven designs that do not handle duplicate or out-of-order message delivery correctly can trigger the same logical pipeline run twice for what was really one upstream event, which is a correctness bug specific to this trigger type that neither of the other two triggers is exposed to in the same way.

Direct answer

For feature generation and model serving specifically, batch orchestration (Airflow-style) trades latency for simplicity: features are as fresh as the last scheduled run, but backfilling, reprocessing, and correctness are comparatively easy to reason about. Streaming architectures (Kafka plus a stream processor) trade that simplicity for freshness: features can update within seconds, at the cost of continuous infrastructure to operate, real state-management complexity, and a much harder reprocessing story when historical feature values need to be corrected. Most real ML platforms use both, choosing per feature based on how much the specific serving use case actually needs sub-minute freshness, not adopting one architecture wholesale.

Structured elaboration

Latency. Batch-orchestrated feature pipelines are bounded by the schedule interval, typically minutes to hours; a model serving on features that are, say, an hour stale is common and often fine (a user's purchase history from an hour ago is still a good predictor). Streaming pipelines can deliver features within seconds of the triggering event, which matters specifically when the model's prediction quality genuinely depends on very recent behavior (a fraud model reacting to a transaction happening right now, not an hour-old snapshot).

State management. A batch job typically recomputes its output from a full or incremental read of source data each run, with any needed state (a running total, a rolling window) either recomputed from scratch or carried forward via an explicit checkpoint table the task itself manages; the orchestrator has no built-in notion of state between runs. A stream processor maintains state natively as a first-class capability (windowed aggregates, running counts per key, joins between live streams), continuously, as records flow through, which is architecturally the reason streaming exists as a distinct approach rather than just "batch, but more often."

Backfilling and reprocessing complexity. This is where batch has a real, durable advantage for feature pipelines specifically: reprocessing a historical window in a batch pipeline is the same mechanism as a normal run, just scoped to past logical dates, and it is comparatively straightforward to reason about (rerun the DAG for the affected dates, the output converges if writes are idempotent). Reprocessing a stream processing job's historical output is fundamentally harder: it usually means replaying the source event stream from an earlier offset through the (possibly changed) processing logic, which requires the raw events to still be retained that far back, and re-deriving what the stream processor's internal state would have looked like at each point in time, not just recomputing a final answer the way a batch job does.

Exactly-once guarantees. At the level relevant to choosing between these two architectures (not the deep mechanics of achieving it, which is its own topic): batch pipelines get an exactly-once-equivalent property comparatively cheaply, since a scheduled or backfilled run that writes to a partition-scoped, overwrite-not-append destination naturally converges to the same state no matter how many times it runs, which is a property of the write pattern, not something the orchestrator or the data layer has to actively guarantee mid-stream. Streaming pipelines have to work harder for the same property, since a stream processor consuming from a message queue can, without careful design, process the same event more than once (on a restart, a rebalance, or a retry) and needs its own mechanisms to avoid that showing up as a duplicated or double-counted feature value; this additional correctness burden is a real cost of the streaming approach, not a detail to gloss over when comparing the two.

Operational overhead. A batch orchestrator is, comparatively, a "fire and check" system: it runs on a schedule, and between runs there is nothing actively executing that needs monitoring. A streaming architecture is a continuously-running production system (the stream processing job itself, plus the message broker it reads from) that needs its own dedicated monitoring (consumer lag, backpressure, checkpoint health) around the clock, not just when a scheduled run happens to be executing; this is a standing operational commitment, not a one-time setup cost.

Worked example

A recommendation model uses two feature groups: user_purchase_history_7d (a rolling 7-day aggregate, updated daily is entirely sufficient since purchase patterns do not meaningfully shift within a day) and user_session_activity_5min (a rolling 5-minute window of in-session clicks, used to adjust recommendations live during an active browsing session).

• user_purchase_history_7d is built as a daily batch job, orchestrated by Airflow, reading the last 7 days of purchase events and writing a partitioned, overwrite-per-day feature table. When a bug is found and the last 30 days need correcting, the fix is a standard backfill: rerun the DAG for the affected 30 logical dates, each overwriting its own partition, no different in kind from the pipeline's normal operation.
• user_session_activity_5min is built as a Flink job consuming a click-event Kafka topic, maintaining a sliding 5-minute window per user session in the stream processor's own managed state, and writing the current window's aggregate to a low-latency feature store on every update. If a bug is found in this job's windowing logic, correcting the last 30 days of historical values (if that value is even retained and queried historically at all, which is less common for a feature this ephemeral) would require replaying 30 days of raw click events from Kafka, assuming that retention window is even available, through the corrected logic, a meaningfully heavier operation than the batch job's equivalent fix.
• The two are coordinated, not competing: the orchestrator triggers and monitors the daily batch job on its own schedule, and separately runs a lightweight, low-frequency health-check task against the streaming job (is it running, what is its consumer lag), rather than trying to make one architecture serve both freshness requirements.

Trade-offs and pitfalls

Building every feature pipeline as streaming by default, on the reasoning that "fresher is always better," is the most common over-engineering mistake in ML feature platforms: the added operational burden and the much harder reprocessing story are real, ongoing costs that many features (like the 7-day purchase history above) never actually need to pay for, since the model's prediction quality is not meaningfully different whether that feature is an hour or a day stale.

Underestimating the reprocessing cost of a streaming feature until the first time a bug actually requires it is a common, expensive surprise: teams that have only ever run a streaming feature pipeline forward, never needing to correct historical output, often discover during the first real incident that doing so requires event retention and replay tooling that was never built, because the forward-only happy path never exercised that need.

Assuming batch's exactly-once-equivalent property is automatic rather than designed is a subtler pitfall: it only holds if the batch job's writes are genuinely partition-scoped and overwrite-based; a batch job that appends instead of overwrites loses this property just as thoroughly as a naively-implemented streaming job would, so the advantage described above is a property of the write pattern each approach commonly uses, not an inherent guarantee of "batch" as a category.

Direct answer

A minimal scheduler for retraining jobs needs three pieces working together from a JSON-described dependency graph: a topological sort that resolves a valid execution order, the same sort doubling as cycle detection since a graph with a cycle can never be fully ordered, and a backfill layer that replays that resolved order once per missed date, oldest date first. The sketch below implements exactly this with Kahn's algorithm and explains how to extend it toward distributed execution and retries without changing its core shape.

Structured elaboration

Approach. resolve_order builds a successor list and an in-degree count for every task from the declared dependency pairs, then repeatedly pulls tasks with zero remaining in-degree, in a stable, sorted order for determinism, appending each to the result and decrementing its successors' in-degree. If every task is eventually appended, the order is valid; if any task never reaches zero in-degree, the remaining tasks form a cycle, and resolve_order raises CycleError naming exactly which tasks never resolved, rather than looping forever or silently returning a partial, incorrect order.

Key points. Backfilling is deliberately not a separate scheduling algorithm: backfill calls the exact same resolve_order once per missed date, sorted oldest first, so the DAG's own dependency logic never has two different code paths for "normal run" versus "backfill run." This sketch does not model a depends_on_past-style dependency between different dates' runs of the same task; each date's replay is fully independent of every other date's, which is called out explicitly rather than silently assumed to be handled.

Worked example

"""
Minimal scheduler for retraining jobs: accepts a JSON DAG, resolves execution order via
Kahn's algorithm (which doubles as cycle detection), and supports backfilling missed runs
for specific dates by replaying that same resolved order once per missed date.
"""
import json
from collections import deque


class CycleError(Exception):
    pass


def resolve_order(dag_json):
    """dag_json: {"tasks": [str, ...], "deps": [[upstream, downstream], ...]}"""
    tasks = dag_json["tasks"]
    deps = dag_json["deps"]

    succs = {t: [] for t in tasks}
    indeg = {t: 0 for t in tasks}
    for a, b in deps:
        succs[a].append(b)
        indeg[b] += 1

    queue = deque(sorted(t for t in tasks if indeg[t] == 0))
    order = []
    while queue:
        t = queue.popleft()
        order.append(t)
        for s in sorted(succs[t]):
            indeg[s] -= 1
            if indeg[s] == 0:
                queue.append(s)

    if len(order) != len(tasks):
        unresolved = [t for t in tasks if t not in order]
        raise CycleError(f"cycle detected, tasks never reach indegree 0: {unresolved}")
    return order


def run_dag_for_date(dag_json, run_date, executed_log):
    order = resolve_order(dag_json)
    for task in order:
        executed_log.append((run_date, task))


def backfill(dag_json, missed_dates):
    """Replays the DAG's resolved order once per missed date, oldest date first, so a
    downstream date-partitioned dependency a task might have on 'yesterday's output' is
    honored at the RUN level even though this minimal sketch does not model that dependency
    explicitly (see the answer's extension notes for depends_on_past)."""
    executed_log = []
    for d in sorted(missed_dates):
        run_dag_for_date(dag_json, d, executed_log)
    return executed_log


if __name__ == "__main__":
    dag_json_text = """
    {
      "tasks": ["fetch_features", "validate", "train", "evaluate", "register"],
      "deps": [
        ["fetch_features", "validate"],
        ["validate", "train"],
        ["train", "evaluate"],
        ["evaluate", "register"]
      ]
    }
    """
    dag = json.loads(dag_json_text)

    print("=== resolved order (single run) ===")
    order = resolve_order(dag)
    print(order)
    assert order == ["fetch_features", "validate", "train", "evaluate", "register"]

    print()
    print("=== backfill 3 missed dates, out of order input ===")
    missed = ["2026-07-22", "2026-07-20", "2026-07-21"]
    log = backfill(dag, missed)
    for run_date, task in log:
        print(f"  {run_date}: {task}")
    assert [d for d, _ in log][:5] == ["2026-07-20"] * 5, "must process oldest date first"
    assert len(log) == len(missed) * len(dag["tasks"])
    print("total steps executed:", len(log), "=", len(missed), "dates x", len(dag["tasks"]), "tasks")

    print()
    print("=== cycle detection ===")
    bad_dag = {
        "tasks": ["a", "b", "c"],
        "deps": [["a", "b"], ["b", "c"], ["c", "a"]],
    }
    try:
        resolve_order(bad_dag)
        print("ERROR: should have raised")
    except CycleError as e:
        print("raised CycleError as expected:", e)

Output (actual, from running the block above with python3, stdlib only):

=== resolved order (single run) ===
['fetch_features', 'validate', 'train', 'evaluate', 'register']

=== backfill 3 missed dates, out of order input ===
  2026-07-20: fetch_features
  2026-07-20: validate
  2026-07-20: train
  2026-07-20: evaluate
  2026-07-20: register
  2026-07-21: fetch_features
  2026-07-21: validate
  2026-07-21: train
  2026-07-21: evaluate
  2026-07-21: register
  2026-07-22: fetch_features
  2026-07-22: validate
  2026-07-22: train
  2026-07-22: evaluate
  2026-07-22: register
total steps executed: 15 = 3 dates x 5 tasks

=== cycle detection ===
raised CycleError as expected: cycle detected, tasks never reach indegree 0: ['a', 'b', 'c']

The single-run order matches the linear chain declared in deps exactly. The backfill case is given missed_dates deliberately out of order (2026-07-22, 2026-07-20, 2026-07-21) and the output confirms 2026-07-20 is processed first regardless of input order, proving the sorted(missed_dates) step actually enforces oldest-first replay rather than merely happening to look right. The cycle case, a 3-node cycle with no valid starting point at all, correctly raises CycleError naming all three unresolved tasks, rather than hanging in an infinite loop or silently returning an incomplete order.

Key points, complexity, and edge cases

Complexity: resolve_order is Kahn's algorithm, $O(V + E)$ in the number of tasks and dependency edges, dominated by the sorted insertions into the queue; backfill runs it once per missed date, so a backfill over $m$ dates costs $O(m \times (V + E))$.

Edge cases: an empty tasks list with no deps resolves to an empty order without error, since the queue starts with every zero-in-degree task, all of them, and immediately empties. A task named in deps but not listed in tasks would raise a KeyError from the dictionary comprehensions building succs and indeg, a fail-loud behavior appropriate for a malformed input rather than silently ignoring the dangling reference. The cycle case demonstrates that CycleError correctly reports every unresolved task, not just one, which matters for diagnosing a real malformed DAG definition where identifying the full cycle, not just a single node in it, is what a human actually needs to fix it.

Trade-offs and pitfalls

This scheduler resolves and replays a full DAG once per missed date sequentially; a genuinely distributed extension would need to submit each date's resolved task order to a real worker pool, ideally allowing multiple dates to run their independent stages concurrently rather than one full date finishing before the next starts, since nothing in this dependency structure inherently requires sequential dates to be processed one at a time, only that within a single date, tasks respect their own topological order.

Extending toward real retries needs per-task state (pending, running, succeeded, failed) tracked externally, not just an in-memory executed_log list; a crash partway through a backfill with no persisted state would have to restart every date from scratch.

The explicit choice not to model depends_on_past here is a real scoping decision, not an oversight: adding it would mean a date's train task cannot start until the previous date's train task has succeeded, which changes the scheduler from "N independent per-date replays" into a single larger dependency graph spanning dates, a genuinely different and more complex problem than this sketch solves.

Direct answer

A task is idempotent if running it multiple times with the same input produces the same end state as running it exactly once: no duplicated rows, no double-counted totals, no repeated side effects. An orchestrator depends on this because its entire safety net, retries, replays, and backfills, works by re-executing a task, sometimes more than once for the same logical unit of work; if a task is not idempotent, every one of those safety mechanisms becomes a correctness risk instead of a recovery tool. Turning a non-idempotent write into an idempotent one is usually a matter of replacing "add to what's there" with "set what should be there," and validating it in CI means actually running the task twice against the same input and asserting the output is identical both times, not just asserting it by inspection.

Structured elaboration

What idempotency means precisely for an ETL task. Given the same input data and the same logical run identifier (the same logical date, the same partition), executing the task 1 time or 5 times leaves the target system in the same observable state. This is a property of the task's write behavior specifically; a task that only reads and computes, with no external side effect, is trivially idempotent, since re-running it just recomputes the same answer without touching anything.

Why the orchestrator depends on it. Three of the orchestrator's core mechanisms all boil down to "run this task again":

• Retries. A task that fails partway through (after some side effect already happened) gets re-executed automatically. If the write is not idempotent, the retry compounds the partial side effect from the failed attempt instead of correcting it.
• Replays, meaning a manual rerun of a specific task instance, for example after diagnosing and fixing a bug, without waiting for a full backfill.
• Backfills. Re-running a historical logical interval, sometimes because the interval genuinely never ran, but often because it DID run and produced wrong output that needs replacing, not adding to.
  In every one of these cases, the orchestrator has no way to know or enforce whether the underlying write is safe to repeat; it can only trigger the task again. Idempotency is what makes "trigger it again" a safe, general-purpose recovery mechanism instead of something that has to be reasoned about case by case, task by task, every time something goes wrong.

Turning a non-idempotent write into an idempotent one. A common non-idempotent shape is an append-only insert: a task that runs INSERT INTO daily_totals (date, region, total) VALUES (...) for each computed row. Run it twice for the same date, and the table now has two rows for that date and region, silently doubling anything downstream that sums the table. The idempotent version replaces that pattern with one of two mechanisms:

1. Delete-then-insert, scoped to the exact logical unit being written. Before inserting, delete any existing rows for that specific (date, region) key, then insert fresh. Running it N times leaves exactly the rows from the last run, since each run first clears its own prior output.
2. Upsert (insert-or-replace) keyed on the natural identifier. INSERT ... ON CONFLICT (date, region) DO UPDATE SET total = EXCLUDED.total (or the equivalent MERGE statement) achieves the same end state in one statement, without a separate delete step, and is generally preferred when the underlying table supports it, since it avoids a window where the delete has happened but the insert has not.
   Both approaches share the same principle: the write is scoped to overwrite exactly the rows this task run is responsible for, identified by a stable key, rather than blindly adding new rows every time.

Trade-offs the fix introduces. Delete-then-insert needs the delete's scope to be exactly right; too narrow (missing a key column) leaves stale rows behind after a schema or logic change, too broad (deleting more than this run's own output) can destroy another task's data if two tasks share the same table. Both delete-then-insert and upsert require a stable natural key to scope or conflict on, which is not always available for genuinely append-only event data (a raw event log with no natural deduplication key); in that case, idempotency has to be achieved differently, typically by making the destination path or partition itself unique per run (so a rerun overwrites the same file/partition rather than needing row-level conflict resolution) rather than by upserting individual rows. There is also a performance cost: an upsert or a scoped delete-then-insert is generally more expensive than a blind append, since the database has to check for existing matches, which matters at high write volume.

Validating idempotency in CI. Rather than reasoning about it by reading the code, run the task twice against the same fixed input in a test database or table and assert the resulting state is identical after both runs, not just that the second run did not error. Concretely: seed a test table, run the write function once, capture the full resulting state (row count, and ideally a hash or sorted dump of the actual row contents, not just the count, since a delete-then-insert bug that leaves the row count right but the values stale would pass a count-only check), run the write function a second time with the same input, and assert the captured state is unchanged. This is a stronger test than checking the function does not raise on a second call, since a non-idempotent function can run successfully twice while still producing wrong (duplicated or doubled) data.

Worked example

Take a preprocessing step in a feature pipeline that computes a daily aggregate and writes it to a daily_features table, originally implemented as a non-idempotent append:

Non-idempotent version: the task computes feature_value for (user_id, feature_date) and executes a plain INSERT for every row. Run once: 10,000 rows, one per user for that date. Retried after a transient failure partway through a 10,000-row batch (say, 6,000 rows already inserted before the failure): the retry reruns the whole computation and inserts all 10,000 rows again, so the 6,000 users caught in the partial first attempt now have 2 rows each and the other 4,000 have 1, a silently inconsistent table where a naive AVG(feature_value) for those 6,000 users is now computed from duplicated identical values (which happens not to change the average, but does change any query that instead uses COUNT or SUM, both of which are now wrong for exactly the users caught in the partial retry).

Idempotent version: the write becomes INSERT INTO daily_features (user_id, feature_date, feature_value) VALUES (...) ON CONFLICT (user_id, feature_date) DO UPDATE SET feature_value = EXCLUDED.feature_value. The same retry scenario now: the first, partial attempt inserts 6,000 rows; the retry recomputes and writes all 10,000, of which the first 6,000 hit the ON CONFLICT path and are updated in place (not duplicated), and the remaining 4,000 are freshly inserted. End state after the retry: exactly 10,000 rows, one per user, matching what a single clean run would have produced, regardless of exactly where the original attempt failed.

CI validation for this task: seed a test database with a small fixture of 50 users, run the write function once, capture a sorted dump of (user_id, feature_date, feature_value) for all rows, run the write function a second time with the identical input, capture the same dump again, and assert the two dumps are byte-identical and the row count is still 50. This test would fail loudly against the non-idempotent version (100 rows the second time, dump mismatch) and pass against the upsert version (50 rows both times, identical dump), which is exactly the distinction a row-count-only or "did it error" check would miss.

Trade-offs and pitfalls

The most common mistake is validating idempotency by re-reading the code and reasoning "this looks like it should be safe" rather than actually running it twice; a delete-then-insert whose delete clause is missing one key column looks correct on a read-through and only fails when actually exercised with a second run against real data.

A second common mistake is testing idempotency by checking that the row count is unchanged after a second run, without checking the row values; a bug where the delete step correctly removes stale rows but the insert step recomputes with a subtly different formula on the second pass produces the right count with wrong values, and a count-only test gives false confidence.

Finally, upsert-based idempotency assumes the natural key used for conflict resolution is genuinely stable and unique for the task's logical scope; if two different upstream sources can legitimately both produce a row for the same (user_id, feature_date) key for different reasons, an upsert silently overwrites one with the other instead of raising the conflict as an error, which trades away a signal that might have indicated two pipelines are stepping on the same output when they should not be.

Direct answer

A workflow orchestrator like Airflow or Dagster coordinates discrete, scheduled or triggered units of work; it is built around the idea that a task starts, runs, and finishes. A native stream processor like Flink or Spark Structured Streaming runs one continuous, long-lived job that never "finishes" in the same sense, processing an unbounded flow of records as they arrive. Use a workflow orchestrator when the actual freshness requirement tolerates minutes to hours of latency and the work naturally decomposes into discrete runs; reach for native stream processing only when the requirement is genuinely sub-minute and the extra operational complexity of a continuously-running, stateful system is actually justified by that requirement.

Structured elaboration

Latency expectations. An orchestrator's latency floor is set by its scheduling and task-startup overhead, typically seconds to low minutes at best for a triggered run, and by the schedule interval itself if the pipeline is on a fixed cadence rather than event-triggered. A stream processor's latency floor is much lower, often sub-second to a few seconds, because records are processed continuously as they arrive rather than waiting for the next scheduled or triggered batch. The practical question is not "which is faster" in the abstract, it is whether the business requirement actually needs the lower floor; a dashboard refreshed every 15 minutes gets no benefit from sub-second processing underneath it.

Stateful processing. A stream processor maintains state across the unbounded input as a first-class capability: running aggregates, session windows, joins between two live streams, all kept in the processor's own managed state store as records flow through. An orchestrator has no equivalent built-in concept; each triggered task run is generally treated as stateless from the orchestrator's point of view; any state a batch pipeline needs across runs (a high-water mark, a running total) has to be stored and read back explicitly by the task itself, in a database or file, not something the orchestrator manages for it.

Checkpointing. This is one of the clearest architectural differences between the two. A stream processor periodically checkpoints its internal state so that if the job crashes, it can resume from the last checkpoint instead of reprocessing the entire stream from the beginning; this is a core, continuously-operating internal mechanism of the stream processing engine itself. An orchestrator has no equivalent concept for the tasks it triggers, since its unit of recovery is the whole task run, not a continuous internal state; when a batch task fails, the orchestrator's answer is "retry the task" or "backfill the interval," treating the task as atomic, not "resume the task from partway through." This is a real reason coordinating a stream processor from an external, discrete-run-oriented orchestrator is architecturally awkward: the orchestrator's model (start, run, finish, maybe retry) does not map cleanly onto a job that runs continuously and manages its own internal recovery state, so orchestrators are typically used to manage a stream processing job's lifecycle (deploy it, restart it if the whole job dies, alert on backpressure or lag) rather than to trigger and sequence its individual units of work the way it does for a batch task.

When to use which. Use a workflow orchestrator for the large majority of data pipelines: anything on a schedule, anything that naturally decomposes into "extract this, transform that, load the other," anything where minutes of latency is genuinely fine. Reach for native stream processing when at least one of these holds: the business requirement is measured in seconds, not minutes; the computation genuinely needs continuously-updated state (a running fraud score, a live session window) rather than periodic recomputation; or the input volume and pattern make "wait and batch" itself impractical (a continuous, unbounded firehose with no natural batch boundary).

Coordinating batch jobs with streaming outputs. The common real pattern is not "pick one," it is both, glued together: a stream processor continuously writes its output into a sink (a table, a topic, an object store path), and a workflow orchestrator runs scheduled batch jobs that read from that sink on its own cadence, for example an hourly rollup job that aggregates the last hour of stream-processed events into a reporting table, or a daily job that reconciles the stream's output against a batch source of truth. The orchestrator does not manage the stream processor's own internal continuous execution; it manages the discrete, scheduled work that consumes the stream's output.

Worked example

A team needs two related capabilities: a live fraud-risk score that must update within seconds of a transaction (a genuine sub-minute freshness requirement, since a fraud check that lags by even a minute is too slow to block a bad transaction before it settles), and an hourly summary table of flagged transactions for an analyst dashboard, where a 15-30 minute lag is entirely acceptable.

• The fraud-risk score is built as a native stream processing job (Flink or Spark Structured Streaming), maintaining a running per-account risk state, checkpointing its internal state periodically so a job restart resumes from the last checkpoint instead of reprocessing the full input stream, and emitting a risk score per transaction with sub-second latency. This genuinely needs continuous state and sub-minute freshness; an orchestrator triggering a batch job every few minutes could not meet the transaction-blocking latency requirement no matter how tightly scheduled.
• The hourly summary table is a workflow-orchestrated batch job, scheduled @hourly, that reads the stream processor's output (landed in a table or object store path) for the completed hour and aggregates it. This does not need the stream processor's continuous state model at all; it is a discrete, scheduled unit of work with a clear start and finish, exactly what an orchestrator is built for.
• The orchestrator's role with respect to the stream processor itself is limited to lifecycle management: a separate, low-frequency monitoring task checks the stream job's health (is it running, what is its consumer lag) and alerts if it has died or fallen behind, rather than the orchestrator triggering individual "runs" of the streaming job the way it triggers the hourly batch task.

Trade-offs and pitfalls

Building a stream processing system for a requirement that does not actually need sub-minute latency is the most common over-engineering mistake in this space: it adds a continuously-running, stateful production system (with its own scaling, checkpointing, and failure-recovery operational burden) to solve a freshness requirement that a well-tuned orchestrated batch job on a tight schedule could have met with far less operational surface area.

The opposite mistake, trying to squeeze a genuinely sub-minute requirement out of an orchestrator by scheduling batch jobs as frequently as the system allows, usually fails on both latency and reliability: task startup and scheduling overhead alone can eat a meaningful fraction of a one-minute budget, and running the same DAG every minute at scale creates its own resource-contention and observability problems.

Treating "coordinate batch with streaming outputs" as needing the orchestrator to somehow manage the stream processor's individual micro-batches or internal checkpoints is a category error: the orchestrator's job is to manage the discrete, scheduled consumers of the stream's output, not the stream processor's own continuous execution, which manages its own internal recovery independently of anything the orchestrator does.