Workflow Orchestration and Scheduling Questions

Orchestrating multi-step data workflows with DAG schedulers (Airflow, Dagster, and similar tools): dependency management between tasks, scheduling strategies (cron-based, sensor and trigger patterns, event-driven runs), and backfills or catch-up runs for time-partitioned data. Covers task-level retries and idempotent task design, so a scheduler can safely re-run a failed step, plus SLA tracking and alerting when a run is late or missing. The core concern is coordination: given a set of dependent tasks that must run in some order on some schedule, how do you trigger, sequence, and re-run them reliably. This is distinct from whether the data itself stays correct across a failure (exactly-once processing, deduplication, checkpointing, and dead-letter handling for corrupted or poison messages, which is a data-consistency concern) and from how a specific compute engine executes a task internally (Spark or Hadoop mechanics). The operational glue of a data platform: getting the right task to run at the right time, in the right order, with visibility into failures.

HardSystem Design
48 practiced

You are migrating 300 on-premises Airflow DAGs that use custom operators and local filesystem dependencies to a managed cloud Composer/MWAA environment. Provide a migration plan covering inventory and dependency analysis, refactoring custom operators, replacing local disk usage with cloud object storage, secrets integration, CI/CD adjustments, validation, rollback strategy, and how you'd prioritize and stage the migration to reduce risk.

EasyTechnical
45 practiced

In Airflow, what is the difference between an Operator and a Sensor? Provide examples of each and discuss trade-offs between poll-based sensors and event-driven approaches (e.g., webhooks or message triggers), including resource usage and reliability.

HardSystem Design
51 practiced

Design a robust orchestration strategy for complex DAGs in Airflow with dynamic branching, parameterized backfills, SLA-based retries, and safe idempotent backfills. Include patterns like TaskFlow, sensors, external task sensors, XCom usage, and strategies for state reconciliation after failures.

HardSystem Design
52 practiced

Design an SLA/SLO tracking and enforcement system for pipeline outputs, e.g., 'report A must be available by 04:00 daily'. Include how to model SLAs at dataset/partition level, integrate SLA checks into the orchestrator, send alerts, and perform automated remediation or prioritization when SLAs are at risk or breached.

MediumTechnical
54 practiced

An hourly-aggregate DAG failed 3 hours ago and only has partial outputs for some hours, but a report needs to publish now. What temporary strategies would you use to provide reasonable values while making sure consumers understand the caveats? Discuss short-term workarounds and the longer-term fix.

Unlock Full Question Bank

Get access to all Workflow Orchestration and Scheduling interview questions and detailed answers.

Sign in to Continue

Join thousands of developers preparing for their dream job.