Workflow Orchestration and Scheduling Questions
Orchestrating multi-step data workflows with DAG schedulers (Airflow, Dagster, and similar tools): dependency management between tasks, scheduling strategies (cron-based, sensor and trigger patterns, event-driven runs), and backfills or catch-up runs for time-partitioned data. Covers task-level retries and idempotent task design, so a scheduler can safely re-run a failed step, plus SLA tracking and alerting when a run is late or missing. The core concern is coordination: given a set of dependent tasks that must run in some order on some schedule, how do you trigger, sequence, and re-run them reliably. This is distinct from whether the data itself stays correct across a failure (exactly-once processing, deduplication, checkpointing, and dead-letter handling for corrupted or poison messages, which is a data-consistency concern) and from how a specific compute engine executes a task internally (Spark or Hadoop mechanics). The operational glue of a data platform: getting the right task to run at the right time, in the right order, with visibility into failures.
You are migrating 300 on-premises Airflow DAGs that use custom operators and local filesystem dependencies to a managed cloud Composer/MWAA environment. Provide a migration plan covering inventory and dependency analysis, refactoring custom operators, replacing local disk usage with cloud object storage, secrets integration, CI/CD adjustments, validation, rollback strategy, and how you'd prioritize and stage the migration to reduce risk.
Sample Answer
Direct answer
Migrating 300 on-premises Airflow DAGs with custom operators and local-filesystem dependencies to a managed cloud service is fundamentally a dependency-inventory problem before it is a cutover problem. Every DAG's actual filesystem and custom-operator dependencies need to be mapped and reclassified for cloud object storage before any DAG moves, because "it worked on-premises" says nothing about whether it will work once local disk access and host-level assumptions are gone.
Structured elaboration
Inventory and dependency analysis. Programmatically scan all 300 DAGs for local filesystem access patterns, direct file opens, hardcoded paths, or an assumption that a temporary directory persists across tasks, which does not hold once workers are ephemeral and not guaranteed to share a host. Scan separately for custom operator and plugin usage, and for any host-level assumption, a system-wide package or a mounted network drive. Produce a per-DAG risk and complexity score from this scan, and let it drive the staging plan below, rather than guessing which DAGs are safe to move first.
Refactoring custom operators. Classify each one as compatible as-is, needs adaptation, or needs a full rewrite against the target managed service's actual constraints: a specific Airflow version, a specific set of pre-installed providers, and no host-level file system write access beyond what the managed service itself exposes.
Replacing local disk usage with cloud object storage. Any DAG reading or writing local files between tasks, a common on-premises pattern, silently breaks under a managed, ephemeral-worker execution model, since two tasks are never guaranteed to run on the same host. That pattern needs replacing with an object-storage-backed one: write to a bucket path, and pass only that path through cross-communication (XCom), not the file's own bytes. This is frequently the single most common silent breakage in this exact kind of migration, since it can pass a quick single-task smoke test on one host and still fail once tasks are genuinely distributed in production.
Secrets integration. On-premises secrets often live in a local Airflow connections database or in host-level environment variables. Migrate these to the target cloud's actual secrets-management integration, never as a lift-and-shift of plaintext values into the new platform's connection interface as a "temporary" step that then quietly never gets revisited.
Continuous integration and deployment (CI/CD) adjustments. The deployment pipeline needs to target the new platform's real deployment mechanism, typically an object-storage bucket sync for DAG files, not a file copy to a server. Any CI step that validated DAGs against the on-premises environment's specific installed packages needs to validate against the target environment's package set instead, catching a missing dependency before deployment rather than at first production run.
Validation. For each migrated DAG, require a defined equivalence test, a parallel run comparing on-premises and cloud output on the same input, before that DAG is considered migrated, not simply "it deployed without error."
Rollback strategy. Keep the on-premises Airflow instance running and authoritative for each DAG until its cloud equivalent has passed validation and run cleanly in production for a defined stabilization period. Rolling one DAG back means re-enabling its on-premises schedule and disabling the cloud one, which stays cheap and fast only for as long as the on-premises instance has not already been decommissioned.
Prioritizing and staging to reduce risk. Migrate DAGs with no custom operators and no local-disk dependencies first, the lowest-risk group, which also validates the migration pipeline and CI/CD mechanics themselves. Then DAGs needing adaptation but no local-disk refactor. Then DAGs needing both a local-disk refactor and a custom-operator rewrite last, since that group carries this migration's two genuinely riskiest dimensions at once.
Worked example
The inventory scan of 300 DAGs finds 165 with no custom operators and no local-disk dependencies (wave 1), 98 with custom operators needing adaptation but no local-disk dependencies (wave 2), and 37 with local-disk dependencies requiring the object-storage refactor, 12 of which also need a custom-operator rewrite (wave 3, the most complex group).
Wave 1's 165 DAGs migrate over 3 weeks. Equivalence testing during this wave catches 4 of the 165 with a subtle behavior difference, traced to a timezone-handling default that differs between the on-premises Airflow version and the managed service's version. Once fixed and re-validated for those 4, the same class of bug is proactively checked against the remaining 135 DAGs across waves 2 and 3, rather than being independently rediscovered dozens more times.
Trade-offs and pitfalls
Keeping the on-premises instance running for the entire migration means paying for both platforms simultaneously for the full duration, a real, ongoing cost that should be weighed against how much risk it is actually still reducing once later waves have stabilized; decommissioning earlier waves' on-premises twins progressively, rather than all at once at the very end, can meaningfully cut this cost without giving up the safety net where it still matters.
Treating the local-disk-to-object-storage refactor as a purely mechanical, one-time task can miss cases where the local-disk usage was hiding an implicit ordering assumption, a task that only worked because it happened to run on the same host as an earlier task that wrote a file it read. Object storage alone does not fix that; the underlying task dependency has to be made explicit too, or the same assumption can re-manifest as a race condition, one task reading before another task's object-storage write has actually finished propagating.
In Airflow, what is the difference between an Operator and a Sensor? Provide examples of each and discuss trade-offs between poll-based sensors and event-driven approaches (e.g., webhooks or message triggers), including resource usage and reliability.
Sample Answer
Direct answer
An Operator is a task that does work: it executes something (a query, a script, a copy) and finishes. A Sensor is a specialized operator that does not do the pipeline's real work at all; it repeatedly checks whether a condition is true (a file exists, a partition landed, an external job finished) and only succeeds once that condition holds, so downstream tasks can safely proceed. Both are subclasses of the same base task abstraction, but a Sensor's entire job is waiting, not producing output.
Structured elaboration
Operators, with examples. BashOperator runs a shell command; PythonOperator runs a Python callable; PostgresOperator (or the newer SQL-execution operators) runs a SQL statement against a database; provider-specific operators like S3ToGCSOperator move data between two systems. Each of these starts, does its work, and reaches a terminal state (success or failure) once, without repeatedly re-checking anything.
Sensors, with examples. FileSensor checks whether a file exists at a given path; S3KeySensor checks whether an object matching a key or prefix exists in an Amazon S3 bucket; ExternalTaskSensor checks whether a task in a different DAG has reached a given state; HttpSensor polls an HTTP endpoint until a response matches a condition. A Sensor is really an Operator with a poke() method the scheduler calls repeatedly (at poke_interval seconds apart) until it returns true or the sensor's timeout is reached.
Poll-based (poke) sensors vs. event-driven approaches. A polling sensor is active: it repeatedly asks "is it ready yet," on its own schedule, regardless of whether the answer has changed since the last check. An event-driven approach flips this: the source system pushes a notification (a webhook, a message on a queue, a cloud storage event notification) the moment the condition becomes true, and a listener reacts to it, without ever having to guess when to check.
Trade-offs, on resource usage. Classic poke-mode sensors hold a worker slot for their entire wait, which means a sensor waiting for hours ties up that slot the whole time, even though it is idle almost all of it; running many such sensors concurrently can exhaust the orchestrator's available worker capacity and starve unrelated tasks. Reschedule mode (available for most modern sensors) fixes the worst of this: the sensor releases its worker slot between pokes and only re-claims one at the next scheduled check, trading a small amount of scheduling overhead for dramatically better resource usage during long waits. Event-driven approaches avoid the polling cost entirely, since nothing is checked until the event fires, but they require standing up and running the notification infrastructure itself (a queue, an event bus, a webhook receiver), which is its own resource and operational cost, just a different kind.
Trade-offs, on reliability. A polling sensor's failure mode is visible and self-limiting: if the condition never becomes true, the sensor eventually times out and fails loudly, which is easy to alert on. An event-driven trigger's failure mode is quieter and more dangerous: if the event notification pipeline itself breaks (a webhook silently stops firing, a queue subscription is misconfigured), nothing ever triggers the DAG, and there is no timeout to notice, because from the orchestrator's point of view, the trigger simply never arrived; there is no task sitting there actively failing to alert on. This asymmetry is the main reliability argument for keeping at least a coarse polling or scheduled fallback check even in an otherwise event-driven design, purely as a safety net that can detect "the event-driven path silently broke."
Worked example
A DAG needs to start processing once a specific S3 object, s3://raw-bucket/orders/2026-07-29/complete.flag, appears, signaling that an upstream job finished writing that day's data.
Poll-based approach: an S3KeySensor configured with poke_interval=300 (5 minutes) and timeout=14400 (4 hours), in reschedule mode. Every 5 minutes, the sensor checks whether the flag object exists; between checks, it releases its worker slot. If the flag never appears within 4 hours, the sensor fails, and that failure is directly visible in the orchestrator's UI with a clear "sensor timed out" reason, easy to alert on and easy for an on-call engineer to diagnose (check whether the upstream job actually ran).
Event-driven approach: an S3 event notification configured to publish to a queue whenever an object matching that prefix is created, with a listener that triggers the DAG run the moment the message arrives. Latency drops from up to 5 minutes (the poll interval) to close to the actual event time, and no worker slot is held waiting. But if the S3 bucket's event notification configuration is accidentally removed, or the queue subscription silently stops delivering, the DAG simply never triggers, with no error anywhere in the orchestrator to alert on, since as far as the orchestrator is concerned, nothing ever asked it to run. Detecting that failure mode requires monitoring the notification pipeline itself (queue depth, delivery metrics, or a separate low-frequency polling check as a safety net) rather than relying on the orchestrator's own failure/timeout mechanism, which never gets the chance to fire.
Trade-offs and pitfalls
Leaving a sensor in classic poke mode instead of reschedule mode is one of the most common resource-planning mistakes in production Airflow deployments: it works fine at small scale, then quietly exhausts the worker pool once enough long-waiting sensors are running concurrently, starving unrelated DAGs that have nothing to do with the ones actually waiting.
Setting poke_interval too aggressively (checking every few seconds for a condition that realistically changes over minutes to hours) wastes resources and can even trip rate limits on the system being polled (a cloud storage API, a database), without materially improving effective latency, since the real bottleneck is usually how long the upstream process actually takes, not the polling granularity.
The event-driven reliability gap described above is the pitfall most often missed in practice: teams adopt event-driven triggering for its latency win and correctly reason about the happy path, but do not build monitoring for the silent-failure case, so the very failure mode event-driven design is most vulnerable to (the trigger path itself breaking) is the one least likely to be caught quickly.
Finally, a timeout set too generously on a sensor (or omitted, defaulting to a very long wait) turns a genuine upstream problem into a slow-motion incident: the sensor sits there for hours before finally failing and alerting, when a tighter timeout tuned to the upstream process's actual expected duration would have surfaced the problem much sooner.
Design a robust orchestration strategy for complex DAGs in Airflow with dynamic branching, parameterized backfills, SLA-based retries, and safe idempotent backfills. Include patterns like TaskFlow, sensors, external task sensors, XCom usage, and strategies for state reconciliation after failures.
Sample Answer
Direct answer
A robust strategy for complex production DAGs combines several mechanisms that are often used in isolation but are strongest together: dynamic branching for conditional logic decided by real data, parameterized backfills that reuse the exact scheduled-run code path, retries that escalate based on remaining deadline slack rather than a fixed count, and explicit cross-communication (XCom) usage kept small and typed through TaskFlow, backed by state reconciliation that actively resolves whatever a failure left behind instead of only alerting a human to go look.
Structured elaboration
Dynamic branching. Use a branch operator, or TaskFlow's branching decorator, to choose the downstream path at run time based on data, not a static configuration value, for example routing to a fast path for a small dataset or a chunked path for a large one depending on a row count discovered during the run. Push the branching decision itself to XCom, so which path was taken, and why, is auditable afterward, not only observable in the moment from the rendered graph.
Parameterized backfills. A backfill run should be driven by a run configuration, accepting a target date or date range, and use the exact same DAG code path as a normal scheduled run, never a separate "backfill mode" branch of logic. A distinct backfill code path is itself a source of divergence bugs: the backfill path quietly doing something subtly different from the path that actually gets tested every day.
Service-level-agreement-based retries. Tie retry policy to how much time genuinely remains against the DAG's own deadline, not a fixed count and delay applied uniformly. A task failing with 3 hours of slack left can afford several attempts with real exponential backoff; the same task failing with 10 minutes of slack left should escalate to a human immediately, since another retry cannot possibly finish before the deadline anyway.
Safe, idempotent backfills. Apply a staging-plus-atomic-swap or MERGE pattern to every date in a backfill range individually, so re-running a backfill, whether from a partial failure or a manual re-trigger, converges to the same correct state per date rather than duplicating or corrupting any single date's data.
TaskFlow. Use the @task decorator for Python-native steps needing explicit, typed data hand-off, since TaskFlow's automatic XCom wiring, a task's return value becoming the next task's input through the dependency graph the code already expresses, removes an entire class of "forgot to push or pull the right XCom key" bugs that classic operators leave open.
Sensors. Bound every sensor with an explicit timeout, and prefer reschedule or deferrable mode over blocking poke mode for anything that might wait more than a couple of minutes, so a long wait does not tie up a worker slot the entire time.
External task sensors. Reserve this specifically for its real purpose, waiting on a task in another DAG within the same Airflow deployment, and map its logical-date correspondence carefully when the two DAGs run on different schedules; a naive same-logical-date assumption breaks silently the moment the two DAGs' schedules diverge, a daily DAG waiting on an hourly one, for instance.
XCom usage. Keep payloads small, identifiers, paths, and counts, not large datasets themselves, which belong in object storage with only the resulting path passed through XCom. XCom is backed by the metadata database by default, and pushing large payloads through it degrades the same database every other scheduling decision also depends on.
State reconciliation after failures. On a DAG-run retry, never assume a clean slate. Explicitly check the actual state of anything the DAG may have partially written and reconcile before proceeding: a task that partially wrote 3 of 5 expected output partitions before crashing should, on retry, detect and complete only the missing 2, not blindly redo all 5, and not silently leave the state ambiguous either.
Worked example
A parameterized backfill for a 30-day range is triggered, using the same code path as scheduled runs. Day 14 fails partway through, having already written 3 of 5 output partitions before crashing on partition 4 from a transient error.
On retry, state reconciliation checks the existing output for day 14, confirms 3 of 5 partitions are already correctly written, verified against a row-count expectation, not merely "the file exists", and the retry logic recomputes only partitions 4 and 5, cutting the redone work for that specific day by:
53=60%
Meanwhile, the other 29 days in the backfill range proceed entirely independently and are unaffected by day 14's failure, since the backfill's mapping over dates isolates each date's own failure to that date alone.
Trade-offs and pitfalls
Dynamic branching decided by run-time data makes a DAG's actual behavior harder to predict from reading the code alone, since you also have to know what data will be seen; that real cost to debuggability has to be offset by good logging of which branch was actually taken and why, per the auditability point above.
Keeping XCom payloads small is a discipline that is easy to violate under deadline pressure, pushing a whole dataframe through because it is faster to write in the moment, and the cost, metadata-database degradation, is not immediately visible when it happens, only once it accumulates across many DAGs sharing the same database.
State reconciliation logic itself needs to be simpler and more thoroughly tested than the pipeline's main-line logic, since it only ever runs during the exact failure scenarios that are hardest to reproduce in normal testing, and a bug inside the reconciliation logic compounds an already-bad situation instead of fixing it.
Design an SLA/SLO tracking and enforcement system for pipeline outputs, e.g., 'report A must be available by 04:00 daily'. Include how to model SLAs at dataset/partition level, integrate SLA checks into the orchestrator, send alerts, and perform automated remediation or prioritization when SLAs are at risk or breached.
Sample Answer
Direct answer
Model SLAs as data (a declarative registry keyed by dataset and, where needed, partition granularity), not as scattered logic embedded inside each individual DAG, so a single system can evaluate every pipeline's SLA state consistently, alert with the same severity model everywhere, and take automated action (remediation, reprioritization, or gating downstream consumers) based on that shared evaluation, rather than each team reinventing SLA enforcement independently and inconsistently.
Structured elaboration
Modeling SLAs at dataset and partition level. An SLA registry table: dataset_name, partition_granularity (daily/hourly/etc), deadline_expression (e.g. '04:00 local, next-day'), severity, owner_team. Critically, the SLA is attached to the dataset the consumer actually cares about ("report A must be available by 04:00"), not to a specific task or DAG run, since a consumer of report_A does not know or care which of several upstream tasks might be responsible for a delay; the SLA check evaluates dataset readiness (has today's partition of report_A been marked complete), which may itself depend on a chain of upstream tasks and datasets the registry does not need to know about directly.
Integrating SLA checks into the orchestrator. A lightweight, centrally-scheduled SLA-evaluation job (not one embedded per DAG) runs on a short interval (every few minutes), reading the registry and checking, for each active SLA, whether the relevant dataset's most recent partition has a completion marker recorded and comparing that against the SLA's deadline expression. This centralizes SLA logic in one place rather than duplicating deadline-checking code inside every producing DAG, which both reduces duplicated logic and makes the SLA registry the single source of truth a dashboard or a downstream system can query directly, instead of having to interrogate each DAG's own internal state individually.
Sending alerts. Two severities, evaluated continuously as the deadline approaches: an at-risk warning once the dataset is not yet ready and less than some buffer remains before the deadline (enough time left to still make it, but worth surfacing), and a breach alert once the deadline has actually passed with the dataset still not ready. Route both to the owning team from the registry's owner_team field, not a generic shared channel, so the alert reaches whoever can actually act on it.
Automated remediation and prioritization when SLAs are at risk or breached. Before or alongside alerting: for an at-risk SLA, the system can automatically raise the priority of the specific upstream tasks that dataset depends on (if the orchestrator's scheduling supports per-run priority adjustment), so a pipeline approaching its deadline gets preferential access to shared resources over lower-priority work currently competing with it, without a human needing to manually intervene. For a breach, if a known-safe automated fix exists (a retry of a specifically-identified failed upstream task, for example), attempt it before or alongside paging, the same automated-mitigation-before-paging pattern used elsewhere in SLA design.
Gating downstream consumers until the SLA is met or remedied. Beyond alerting, the registry can drive an explicit gate: a downstream DAG or a consumer-facing system can check the registry's own readiness state for a dataset before proceeding, rather than trusting its own internal timing assumptions about when upstream data is normally ready. This means a downstream report that would otherwise run on a fixed schedule and risk publishing against stale or incomplete upstream data can instead wait on, or explicitly flag, an unmet SLA, giving the platform a single enforcement point (the registry's readiness state) that both alerting and gating consume consistently, rather than two independently-implemented sources of truth about the same underlying question that could disagree.
Worked example
graph LR
A[SLA registry: report_A, daily, deadline 04:00] --> B[SLA evaluator: runs every 5 min]
B -->|dataset not ready, 30 min before deadline| C[At-risk: raise priority on upstream tasks]
B -->|deadline passed, still not ready| D[Breach: attempt known-safe remediation]
D -->|fails| E[Page owner_team]
B --> F[Downstream consumer gate: checks registry before publishing]
report_A's SLA: daily, deadline 04:00, owner data-eng. At 03:30, the evaluator finds report_A's dataset not yet marked ready; this is 30 minutes before the deadline, inside the at-risk window, so the system automatically raises the priority of report_A's specific upstream tasks (still in progress, running behind due to unrelated contention from another pipeline sharing the same pool), letting them access resources ahead of the lower-priority work they were competing with. At 03:50, report_A's dataset completes and is marked ready, before the 04:00 deadline, so no breach alert fires; the automated priority boost resolved the at-risk state without a human needing to intervene.
A different day: the same at-risk trigger fires at 03:30, but this time the upstream delay is caused by a genuine failure (an upstream task erroring out, not just resource contention), so the priority boost alone does not resolve it. At 04:00, the deadline passes with report_A still not ready; a breach alert fires to data-eng, and because the specific failed task is identifiable, an automated remediation attempt (retry that task) runs alongside the page, so by the time the on-call engineer opens the alert, they see both "breached" and "auto-retry already attempted, still failing," rather than needing to discover that themselves. Meanwhile, a downstream dashboard that reads report_A checks the registry before rendering and displays "data delayed, SLA breached at 04:00" instead of silently showing yesterday's data as if it were current, since it is gated on the registry's readiness state rather than assuming freshness from its own fixed refresh schedule.
Trade-offs and pitfalls
Embedding SLA logic separately inside each producing DAG, rather than centralizing it in a registry-driven evaluator, is the most common way SLA enforcement becomes inconsistent across a platform: different teams implement slightly different deadline logic, alert differently, and a dashboard trying to show "which SLAs are currently at risk across the whole platform" has no single place to query, since the answer is scattered across many DAGs' individually-implemented logic.
Automated remediation without a clear boundary on what it is allowed to attempt is a real risk: a remediation step that retries broadly (not a specific, identified failed task) can mask a genuine, persistent problem behind repeated automatic retries that never actually fix anything, delaying the point at which a human realizes intervention is needed; remediation should be scoped to known-safe, specifically-identified actions, not a blanket "just try again" applied to the whole pipeline.
Gating downstream consumers on the registry without a defined emergency-override path creates its own risk: a downstream consumer that refuses to ever proceed on an unmet SLA, with no way for an authorized operator to override that gate in a genuine emergency (a partial-but-urgently-needed result), can turn a data-quality problem into an availability problem for the downstream system too; the gate needs an explicit, audited override mechanism, not just a hard block.
Finally, setting the at-risk buffer too short (close to the deadline itself) gives the automated remediation and priority-boost mechanisms too little runway to actually help before the deadline passes, turning "at-risk" into a formality that always resolves into a breach anyway; the buffer needs to be sized against how long the platform's remediation mechanisms genuinely take to have an effect, not an arbitrary round number.
An hourly-aggregate DAG failed 3 hours ago and only has partial outputs for some hours, but a report needs to publish now. What temporary strategies would you use to provide reasonable values while making sure consumers understand the caveats? Discuss short-term workarounds and the longer-term fix.
Sample Answer
Direct answer
When a report has to publish now but some hours are missing, the right response is never to silently substitute a guess and publish as if the data were complete; it is to choose an explicit, clearly-labeled stand-in value for the missing hours (the last known-good comparable value, or an interpolation, whichever is more defensible for this specific metric), publish with a visible caveat naming exactly which hours are incomplete, and separately schedule the real fix (the actual backfill) so the report is silently corrected, and visibly re-confirmed as final, once it lands.
Structured elaboration
Short-term workaround options, and how to choose between them.
- Carry-forward (use the last known-good value for the missing hour). Appropriate when the metric is relatively stable hour to hour and a slightly stale value is a reasonable approximation, for example a slowly-changing inventory count.
- Interpolation between the surrounding known hours. Appropriate when the metric has a smooth intra-day trend (traffic ramping through the day) and a straight-line or comparable estimate between the hour before and after the gap is more accurate than simply repeating the last value.
- Explicit "pending" placeholder, with the report showing a partial total or a clearly marked gap rather than any estimated number. Appropriate when the metric is high-stakes enough (a financial figure, a compliance number) that ANY estimated value risks being mistaken for a real one, and an honest gap is safer than a plausible-looking guess.
- Exclude the affected hours from the reporting window entirely, showing a smaller but fully real window (for example, a 21-hour total instead of 24, clearly labeled), when the report's consumer would rather have a smaller, fully accurate number than a full-window number partly built on estimates.
The right choice depends on what the number is actually used for: a directional trend dashboard can tolerate carry-forward or interpolation with a caveat; a number feeding a financial or compliance decision should generally use the placeholder-or-exclude approach, since being wrong with confidence is worse than being visibly incomplete.
Making sure consumers understand the caveats. A workaround value with no visible flag defeats the entire point of choosing a careful, defensible estimate; the report itself needs an explicit, visible marker (a footnote, an asterisk on the affected hours, a banner stating "hours 14-16 are provisional, pending pipeline reprocessing, expected corrected by [time]") so anyone reading it knows exactly which numbers to treat with caution and roughly when a corrected version will be available. This should be as specific as possible: naming the affected hours and giving a real expected-resolution time is far more useful to a stakeholder than a generic "data may be incomplete" disclaimer that does not say which part.
Longer-term fix. Once the underlying DAG failure is diagnosed and resolved, backfill exactly the affected hours (not the whole day, to avoid unnecessary reprocessing of hours that were already correct), overwrite the report's provisional numbers with the real, recomputed values, and send a follow-up notice confirming the report is now final and correct, referencing the earlier caveat so anyone who saw the provisional version knows it has since been resolved. This closing step is often skipped once the immediate pressure is off, but it is what actually closes the loop for stakeholders who specifically remember seeing a caveat and are waiting to know whether it was ever resolved.
Worked example
An hourly sales-aggregate DAG failed 3 hours ago; hours 0-13 completed normally, hours 14-16 are missing, and hour 17 (the current hour) has not run yet. A daily summary report is due to publish now, needing a full 0-17 window.
Given this is a sales figure feeding an internal operations dashboard (directional, not a financial-close number), the team chooses interpolation for the 3 missing hours, using the average of hour 13's and hour 17's... except hour 17 has not run yet either, so interpolation between two known points is not actually available. Recognizing this, the workaround shifts: hours 14-16 use carry-forward from hour 13 (the last fully known value), and the report explicitly shows hour 17 as excluded from the total rather than estimated, since there is no surrounding data to interpolate from yet. The published report shows a clear banner: "hours 14-16 use a provisional carry-forward estimate pending pipeline reprocessing; hour 17 is not yet available and is excluded from this total; expected fully corrected by [end of day]."
Two hours later, the underlying DAG failure (a transient database connection issue) is fixed, and a backfill runs for hours 14, 15, 16, and the now-completed 17, producing the real, correct values. The report is republished with all hours final, and a follow-up notice states: "the earlier provisional figures for hours 14-16 have now been corrected with actual data; today's report is final."
Trade-offs and pitfalls
Choosing to interpolate or carry-forward without checking whether the surrounding hours are actually available (as almost happened in the worked example) is a real, easy mistake under time pressure: an interpolation computed from one real value and one still-missing value is not actually an interpolation, it is a guess dressed up as one, and shipping it without noticing the second input was also missing would have produced a specific, confidently-wrong number rather than an honestly-labeled gap.
Publishing a workaround value without a visible caveat, on the reasoning that "we'll fix it soon anyway so it doesn't matter," is the single most damaging mistake in this scenario: a stakeholder who acts on an unflagged, temporarily-wrong number before the fix lands has already made a decision based on it, and there is no way to retroactively un-make that decision the way there is a way to retroactively correct the report's numbers.
Forgetting the closing "this has now been corrected" notice leaves stakeholders who saw the caveat with no way to know whether the number they are now looking at is the corrected final version or still the provisional one, which quietly erodes trust in every future caveat the team publishes, since "we'll let you know when it's fixed" starts to feel like an empty promise if it is not consistently followed through.
Unlock Full Question Bank
Get access to all Workflow Orchestration and Scheduling interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.