Workflow Orchestration and Scheduling Questions

Direct answer

For a daily task that occasionally hits transient API timeouts, I would configure the orchestrator to retry that specific task automatically with exponential backoff and a small, bounded attempt count (for example, 3-4 attempts, doubling delay, capped), rather than failing the whole run on the first timeout. Before retrying blindly, the task needs to classify the failure: a timeout or a 5xx (server error) response is worth retrying, while a 4xx (client error) like a 400 (bad request) or 401 (unauthorized) is not, since retrying an error caused by a malformed request or bad credentials just reproduces the same failure. Finally, because the retry re-sends a request the external system may have already partially processed, the task itself must be written so that running it twice with the same input has the same effect as running it once, which is what makes it safe for the orchestrator to retry in the first place.

Structured elaboration

Retry and backoff strategy. Exponential backoff is the right default here specifically because the failure mode is a flaky external API: a fixed short delay retries at the same cadence regardless of whether the API is still struggling, while exponential backoff (1 minute, 2, 4, capped at, say, 10) gives the API's own transient issue time to clear and avoids retrying into an API that is still overloaded. Add jitter (a small random offset added to each delay) if many parallel task instances could hit the same endpoint at once, so they do not all retry in lockstep and re-create the same burst of load that caused the timeout. Bound the attempt count (3-4 is typical) and set a retry_delay ceiling; an unbounded or very long-tail retry policy just delays the eventual alert if the API is down for a real, non-transient reason.

Distinguishing retryable vs. non-retryable errors. This is the step that separates a correct retry policy from a wasteful one:

• Retryable: connection timeouts, socket errors, HTTP 429 (rate limited, especially if a Retry-After header is present and should be honored), HTTP 5xx server errors. These indicate the request was probably fine and the problem is transient, on the server side, or load-related.
• Non-retryable: HTTP 4xx client errors other than 429, such as 400 (malformed request), 401/403 (authentication or authorization failure), 404 (resource does not exist). These indicate the request itself was wrong; retrying it produces the identical error every time and just delays surfacing a real bug or a stale credential that needs a person to fix.
• Practically, this means the task's error-handling code needs to inspect the exception type or status code and either raise a retryable exception (which the orchestrator's retry policy catches) or fail immediately without consuming a retry attempt, rather than treating every exception the same way.

Avoiding duplicate side effects on retry. A retry re-executes the entire task, which means any side effect the task already performed before it failed (a partial write, a record inserted, a downstream system notified) can happen again on the retry. Three patterns address this:

1. Idempotency keys. If the external API supports it, send a unique, stable key with each logical request (for example, derived from the task's own logical run identifier plus the record being processed) so the API can recognize and de-duplicate a retried request that already succeeded server-side, even if the original response was lost due to the timeout.
2. Idempotent writes on the receiving end. Instead of an "append" or "increment" operation, use an "upsert" (insert-or-replace) keyed by a natural identifier, so writing the same record twice leaves the same end state as writing it once.
3. Check-before-act. Before performing the side-effecting call, check whether it was already done (for example, query whether today's record already exists) and skip if so; this is weaker than an idempotency key (it has its own race condition if two attempts overlap) but is often the only option when the external API has no native idempotency support.

The general principle: a task is only genuinely safe to retry if a person could run it three times in a row on the same input and the world would look the same as if it ran once. If that is not true, retries need to be disabled for that task, or the side-effecting part needs to be redesigned before an automatic retry policy is turned on.

Worked example

Task: call a third-party pricing API once daily to fetch and store the day's exchange rates, one HTTP POST per currency pair, writing each result into a fx_rates table.

• Backoff config: 4 attempts, exponential backoff starting at 30 seconds and doubling (30s, 60s, 120s), capped at 120s, small jitter (up to +/-10s) added to each delay since the same DAG triggers all currency-pair tasks at once and they would otherwise retry in a synchronized burst against the same API.
• Error classification: the task's HTTP client wraps the call so that a timeout or a 503 raises a RetryableAPIError, which the orchestrator's retry policy is configured to catch; a 401 (expired API key) raises a FatalAPIError instead, which is not retried and immediately alerts, since retrying with the same expired key would fail identically four more times and only delay someone noticing the key needs rotating.
• Duplicate-write safety: the write to fx_rates uses an upsert keyed on (currency_pair, business_date). On attempt 1, the POST to the pricing API times out after the API had, in fact, already computed and internally logged the rate server-side (a common timeout failure mode: the response never made it back, but the work happened). On attempt 2 (after the 30-second backoff), the task calls the API again; the API's own idempotency key (derived from currency_pair + business_date, sent identically on both attempts) lets it recognize the retry and return the same rate rather than recomputing or double-billing an external usage-metered API call. The task then writes that rate via the upsert. Whether attempt 1's write silently succeeded after the client had already given up, or only attempt 2's write ever landed, the fx_rates table ends up with exactly one row for that currency pair and date either way, so the retry is safe regardless of which side of the timeout the original request actually failed on.

Trade-offs and pitfalls

Retrying without classifying the error first is the single most common mistake: a 401 or 400 will fail identically on every attempt, so blind retries just burn the retry budget and delay the alert that would have told someone the credential expired or the request payload is malformed, sometimes by 10+ minutes across four backed-off attempts.

Idempotency keys and upserts add real design cost: the external API must support idempotency keys (not all do), and an upsert requires a natural key that uniquely identifies the logical unit of work, which is not always available (a pure "append a log line" operation has no natural dedup key without adding one). When neither is available, the fallback (check-before-act) is weaker and can itself race under concurrent retries, so it should be treated as a last resort, not a default.

Finally, jitter is easy to skip because it does not matter for a single task in isolation, but it matters a great deal the moment many parallel task instances share a downstream dependency, since synchronized retries without jitter can turn a brief API hiccup into a self-inflicted thundering-herd retry storm against the very system that is already struggling.

Direct answer

The architectural difference underneath every practical difference between the two: Airflow models a pipeline as a graph of tasks (units of execution), while Dagster models it as a graph of assets (the actual data objects a pipeline produces), with tasks as the mechanism that materializes them. That single choice ripples into developer experience, observability, testing, and deployment. Recommend Airflow when the team's mental model is genuinely task-centric (orchestrate arbitrary steps, many of which are not really "data assets" per se) and when the ecosystem's breadth of existing integrations matters most; recommend Dagster when the pipeline's real unit of value is the data it produces, and being able to reason about, test, and monitor at that level pays off, which is common for feature pipelines and other ML-adjacent data products.

Structured elaboration

Developer experience, in more depth. Airflow's DAG-as-Python-file model is simple to start with and has an enormous operator ecosystem (a pre-built integration for nearly any system a pipeline might need to talk to), but that same breadth means operator quality and API consistency vary across providers, since they are maintained by many different contributors. Dagster's asset-first API is more opinionated and type-aware from the start, which raises the initial number of concepts a new user has to learn (assets, ops, resources, jobs, schedules, sensors are all distinct, related concepts), but tends to produce more consistent, more testable code once a team is past that initial learning curve.

Observability, in more depth. Airflow's UI is built around DAG runs and task instances: it answers "did this task succeed, and when." Dagster's UI is built around the asset graph: it answers "is this specific piece of data fresh, and what upstream asset is stale or missing if it's not," which is a more directly useful question for a data consumer who cares about a specific table or feature set, not about which internal tasks happened to produce it.

Dataset/type awareness. This is the core architectural distinction stated above, made concrete: in Dagster, an asset can carry a declared type, and Dagster can check that what a downstream asset receives actually matches what it expects, catching a class of bug (a schema or shape mismatch between pipeline stages) that Airflow's task-centric model has no native mechanism to catch, since Airflow tasks pass data through XCom or external storage with no built-in typing.

Testing support. Both frameworks let you unit-test the business logic inside a task or op as plain Python, which is the majority of what actually needs testing. The difference is at the pipeline-structure level: Dagster's asset graph is a first-class object that can be validated and partially executed in tests (materialize just this asset and its direct dependencies, in isolation), while testing an Airflow DAG's structure (are the dependencies wired correctly, does the schedule behave as expected) typically requires more custom test scaffolding, since the DAG object itself is not designed around being partially executed for testing.

Deployment model. Airflow has the larger, more mature ecosystem of both managed offerings and self-hosted deployment patterns, reflecting its longer track record and larger install base; teams needing a specific compliance posture, a specific cloud provider's managed service, or deep operational familiarity already present on the team often lean this way by default. Dagster's deployment ecosystem (Dagster Cloud, plus self-hosted) is smaller but has matured significantly, and Dagster's software-defined-assets model is a genuine design differentiator, not just a smaller Airflow.

Worked example

A general-purpose ETL platform team, many source systems, mostly straightforward extract/load/transform sequencing, existing Airflow expertise on the team. Recommend Airflow: the pipeline's actual complexity is in sequencing many heterogeneous sources, which Airflow's operator ecosystem directly supports, and the team's existing operational familiarity with Airflow is a real, non-trivial advantage that a rewrite in a different tool would spend real time and risk to obtain only a marginal architectural benefit for a workload that is not particularly asset-model-shaped.

A machine learning feature pipeline: raw events to engineered features to a feature store, consumed by multiple downstream training jobs, where "is this feature fresh and correctly typed" is the actual question data scientists ask daily. Recommend Dagster: the pipeline's real unit of value is explicitly the data assets (each engineered feature), not the tasks that produce them, and Dagster's asset-centric observability directly answers the question data scientists actually have ("is feature_x fresh as of today, and does it match the schema training expects") without them needing to translate from "which tasks succeeded" to "is my feature usable." The type-awareness also catches a real, common ML pipeline failure mode early: a feature whose shape or type silently drifted between the pipeline and what a training job expects, which Dagster's typed asset graph can surface as a build-time or run-time check rather than a downstream training failure discovered much later.

Trade-offs and pitfalls

Choosing Dagster purely because "asset-based" sounds like the more modern architecture, for a pipeline that is genuinely task-shaped (many heterogeneous, loosely-related steps with no strong shared data-asset identity), adds conceptual overhead without a corresponding benefit; the asset model earns its complexity specifically when the pipeline's actual product is a well-defined set of data assets, not universally.

Staying on Airflow purely out of inertia for a pipeline that is genuinely asset-shaped (a feature pipeline, a data product with many downstream consumers who care about freshness and type correctness) means building, by hand, observability and validation that Dagster provides natively, which is a real, recurring engineering cost paid on every such pipeline, not a one-time migration cost avoided.

Finally, underestimating Airflow's ecosystem breadth when evaluating Dagster is a common mistake in the other direction: a source system with a mature, well-tested Airflow provider but no equivalent first-class Dagster integration can mean building custom integration code in Dagster that Airflow would have gotten for free, which is a real cost that should be weighed against the asset-model benefits on a case-by-case basis, not assumed away.

Direct answer

Cron-like scheduling, periodic sensor polling, and event-driven triggers trade latency against cost and operational complexity, in that order: a fixed schedule is the cheapest and simplest but only as fresh as its interval, a polling sensor narrows that latency at the cost of wasted checks, and an event-driven trigger gets the lowest latency but requires the most infrastructure to wire up. For a new ingestion job, I default to a plain schedule unless there is a concrete freshness requirement the schedule cannot meet, and only reach for event-driven triggering once polling's own overhead becomes the bottleneck.

Structured elaboration

Cron-like schedule_interval. The orchestrator runs the DAG on a fixed cadence regardless of whether the data is actually ready; correctness for a daily ETL relies on the source reliably landing before the scheduled run time, with enough margin built in. Latency is bounded by the interval itself: if data can land anywhere in a 24-hour window and the schedule runs once a day, the worst case is close to a full day's delay. Cost and complexity are both minimal, since there is nothing to poll and nothing to listen for, just a time expression.

Periodic sensor polling. A sensor task repeatedly checks a condition (does this S3 prefix exist, has this API returned a new value) on its own poll interval, and only lets downstream tasks proceed once the condition is true, or gives up after a configured timeout. This narrows the latency window from "up to a full schedule interval" to "up to one poll interval," which matters when the source's actual arrival time varies. The cost is that every poll consumes resources (an orchestrator worker slot in the classic polling mode, or an API call against the source) even on the vast majority of checks that find nothing new; tightening the poll interval to reduce latency directly increases that overhead. Most modern orchestrators mitigate part of this with a reschedule mode that releases the worker slot between polls instead of holding it for the entire wait, trading a bit of scheduling overhead for much better resource usage during long waits.

Event-driven triggers. Instead of the orchestrator repeatedly asking "is it ready yet," the source system (or a piece of glue infrastructure sitting in front of it) pushes a notification the moment the condition becomes true: an S3 event notification, a message on a queue, a webhook call. This gets latency down to close to the event's own occurrence, without the wasted-check cost of polling, but it requires standing up and operating that notification path: a queue or event bus, a listener or webhook endpoint that can trigger a DAG run outside its normal schedule, and monitoring for that new piece of infrastructure itself, since a silently-broken event listener is a much quieter failure than a sensor that visibly times out.

Deciding for a new ingestion job. Ask three questions in order:

1. Does the data arrive on a knowable, fixed schedule? If yes, use a plain schedule; nothing else is justified.
2. If arrival time varies, how much does the resulting delay actually cost the business? If a few hours of possible delay is acceptable, sensor polling with a reasonable poll interval (for example, every 10-15 minutes) is usually the right level of investment.
3. If delay needs to be minutes or less, or the source system already emits events, only then justify the added infrastructure of an event-driven trigger, since building and operating that infrastructure is not free, and it is easy to over-invest in low-latency triggering for a pipeline that feeds a dashboard nobody looks at more than once a day anyway.

Worked example

Two concrete ingestion jobs illustrate the decision:

Daily ETL from a vendor's nightly export. The vendor reliably produces a file by 03:00 UTC every night. A plain schedule_interval of 0 4 * * * (04:00 UTC, one hour of margin) is correct here: the arrival time is known and stable, a sensor would be polling for a file that is virtually always already there by the time it starts checking, and event-driven triggering would add infrastructure to save, at most, an hour of latency on a report nobody consumes before business hours anyway.

S3 file arrival for an ingestion job with variable timing. Upstream files land at unpredictable times within a business day, anywhere from 09:00 to 17:00, and downstream consumers want the data within roughly 30 minutes of arrival. A sensor polling every 5 minutes in reschedule mode bounds latency to about 5 minutes in the worst case, well inside the 30-minute requirement, at a modest cost: over an 8-hour window that is roughly 96 poll checks, most of which find nothing, but each check is cheap (a lightweight existence check against the S3 API) and reschedule mode means the orchestrator is not holding a worker slot idle between checks. If the freshness requirement tightened to under a minute, or if the ingestion volume grew to hundreds of prefixes being watched at once (making 96 x N poll checks a real cost), that is the point where switching to S3 event notifications feeding a queue that triggers the DAG directly would be worth the added infrastructure; at 30-minute tolerance and a handful of prefixes, it is not.

Trade-offs and pitfalls

The most common mistake is reaching for event-driven infrastructure by default because it sounds like the more sophisticated answer, when the actual freshness requirement does not need it; the added queue, listener, and trigger path is a new production system to operate and monitor, and every extra moving part is a new thing that can silently break.

The opposite mistake is running a sensor in classic poke mode (holding a worker slot for the entire wait) at a tight poll interval against a large number of prefixes or sources, which can exhaust the orchestrator's worker pool during a busy window and starve unrelated pipelines, not just the one doing the waiting; reschedule mode (or moving to an event-driven design) is the fix, not just reducing the poll frequency, which only trades latency for the same underlying resource problem at a slower rate.

A subtler pitfall with plain cron scheduling is under-provisioning the margin: scheduling the run at exactly the vendor's typical delivery time, with no buffer, means any day the vendor is even slightly late produces a hard failure instead of a graceful wait, which is really a sensor's job description, not a fixed schedule's.

Direct answer

Reprocess the 7 affected days into a new, versioned copy of the dataset rather than modifying the existing one in place, and only make that new version visible to consumers with a single atomic pointer switch once every affected partition has finished and passed validation. This is what guarantees the "old or fully reprocessed, never a mixture" requirement: consumers always read through a stable reference (a view, an alias, a pointer) that flips from one complete, immutable version to the next in one operation, never a state where some days behind that reference are old and others are new.

Structured elaboration

Dataset versioning. Instead of overwriting the live table's affected partitions directly, write the reprocessed 7 days into a new version, identified by an explicit version tag or timestamp (revenue_v2, or a version column baked into the storage layout). The existing version (revenue_v1) stays fully intact and untouched throughout reprocessing, which is what makes the "consumers see old or fully reprocessed, not a mixture" guarantee possible: there is always exactly one complete, internally consistent version available to read, whichever one is currently designated current.

Compaction. Reprocessing due to a schema change often means the new version's physical layout differs from the old (new columns, a different partitioning scheme, or simply the accumulated small files a targeted 7-day rewrite produces). Compact the newly-written partitions into an efficient file layout (merging small files, optimizing for the query patterns consumers actually use) as part of the reprocessing job itself, before the version is promoted to current, not as a follow-up cleanup task after consumers are already reading it, since compaction after promotion risks a performance regression window right when consumers start using the new version.

Atomic visibility via a version pointer. Consumers should never query revenue_v2 (or v1) directly; they query a stable reference, for example a view revenue_current that points to whichever version is designated current, or a metadata record an application-layer client checks before choosing which physical table to read. Promoting the new version is a single, fast metadata operation (repointing the view, or updating one row in a version-registry table), not a data-copying operation, which is what keeps the switch atomic: from a consumer's perspective, one query sees v1 in full, the very next query sees v2 in full, with no window where a query could see some of both.

Consumer notifications. Notify affected consumers (specifically, systems or teams that read this dataset, not necessarily every stakeholder) ahead of the promotion, stating what changed (a schema addition, corrected values for the affected 7 days) and roughly when the switch will happen, so any consumer with its own caching layer or schema expectations can prepare rather than being surprised by values or columns changing underneath a query it just ran. A machine-readable notification (a message to a topic other systems can subscribe to, not just a Slack post aimed at humans) is worth having if any consumer is itself an automated system rather than a person checking a dashboard.

Worked example

graph TD
    A[Reprocess 7 affected days into a NEW versioned copy] --> B[Compact new partitions]
    B --> C[Run validation against the new version]
    C -->|pass| D[Atomically repoint revenue_current view to the new version]
    C -->|fail| E[Halt, alert, old version stays current]
    D --> F[Notify consumers: switch complete]

The revenue table has a schema change (a new discount_tier column) affecting the last 7 days. The reprocessing job writes those 7 days into revenue_v7 (the current live version is revenue_v6), leaving revenue_v6 completely untouched. Compaction runs on the newly-written 7 partitions, merging what would otherwise be several small files per day into a query-efficient layout. Validation confirms revenue_v7's 7 reprocessed days match expected row counts and the new discount_tier column is populated correctly for a spot-checked sample. Once validation passes, a single CREATE OR REPLACE VIEW revenue_current AS SELECT * FROM revenue_v7 (or the equivalent for the storage layer in use) executes atomically: any query against revenue_current issued a moment before this statement reads entirely from v6, and any query issued a moment after reads entirely from v7, with no query ever observing a mixture of the two. Consumers, notified 24 hours ahead of the planned switch time, know to expect the new column and the corrected 7 days; a machine-readable event is also published to a topic an automated downstream reconciliation job subscribes to, so that job re-validates its own derived output against the new version without a human needing to trigger it.

Trade-offs and pitfalls

Reprocessing in place (overwriting the live table's 7 affected partitions directly, one at a time, rather than into a separate version) is the most common shortcut that violates the atomicity requirement: a consumer querying across all 7 days mid-reprocess sees some days already corrected and others still old, exactly the mixed state the question explicitly asks to avoid, and this happens silently, with no error to signal it.

Versioning has a real storage cost: keeping the old version fully intact alongside the new one temporarily doubles storage for the affected partitions, which is a deliberate, worthwhile trade for the atomicity guarantee but should be sized and time-boxed (retire the old version once the new one has been current and validated in production for some period, not kept indefinitely by default) rather than left as an unbounded, growing cost.

Skipping compaction before promotion is a subtler pitfall: it makes the atomic switch itself fast and clean, but leaves consumers hitting a newly-promoted version with a suboptimal file layout at exactly the moment they start relying on it, which reads as a performance regression coinciding suspiciously with the schema change, even though the two are actually unrelated causes that happened to land at the same time.

Finally, treating consumer notification as optional because "the switch is atomic, so nothing breaks" misses the point: atomicity prevents a mixed, inconsistent read, but it does not prevent a consumer's own downstream logic from breaking on an unexpected new column or on values that shifted for the corrected days, so notification remains necessary even though the mechanism itself is safe.

Direct answer

Tracking 100,000 prefixes for file arrival without 100,000 live sensors means replacing per-prefix polling with a small set of shared, batched List calls against a common parent prefix, combined with event-driven detection for anything wired to storage event notifications, plus a slow, cheap safety-net sweep that exists purely to catch what the event path misses. The design below proves this at a representative scale of 2,000 prefixes and shows the batching arithmetic separately for why the same mechanism holds at the full 100,000.

Structured elaboration

Approach. Three components share one last-known-state cache, so nothing is ever double-reported. An event path fires fast for prefixes wired to storage event notifications (ObjectCreated events, delivered at-least-once, typically within seconds, but never instantaneous or guaranteed-ordered). A poll-only adaptive sweep is the primary detection path for prefixes with no event wiring: one shared, batched List call per sweep, covering every not-yet-detected poll-only prefix at once, with an exponentially backed-off interval (capped at 64 ticks) that resets to fast whenever the sweep actually finds something new. A safety-net sweep runs on a fixed, slow, hourly cadence across every not-yet-detected prefix regardless of event wiring, deliberately cheap and infrequent, since its job is reliability insurance against a dropped or delayed event, not speed.

Key points. The critical technique that avoids spawning one sensor per prefix is that a single List call against the common parent prefix returns everything currently present in one paginated response (up to 1,000 keys per page), so the cost of checking is bounded by how many objects actually exist under that prefix, not by how many prefixes are being watched. At the full 100,000-prefix scale, one complete sweep costs:

$\lceil 100000 / 1000 \rceil = 100 \text{ paginated List calls}$

versus 100,000 individual per-prefix calls without batching, a 1,000 times reduction per full sweep, the same mechanism demonstrated at the smaller, executed scale below.

Recovery after downtime. On restart, both the poll-only sweep and the safety-net sweep are forced immediately, independent of their normal schedules. Because each sweep re-lists reality and diffs it against the cache rather than trusting any assumption about what happened while it was down, every prefix that actually arrived during the outage is caught in that first forced sweep, with a detection latency bounded by the outage length rather than being lost.

Consistency. Since December 2020, Amazon S3 provides strong read-after-write consistency for all operations, including a List immediately after a Put, automatically and for all buckets, so a sweep is guaranteed to see an object the moment it has finished uploading, with no eventual-consistency caveat to design around on the storage side. The remaining consistency risk in this design is entirely on the detection-pipeline side, a dropped event or a sweep that has not run yet, not on whether a completed upload is visible when listed.

Worked example

"""
Simulated reconciliation scheduler for S3 'file-available' monitoring at scale.
Representative N=2000 prefixes (disclosed scale-down from the question's 100k target; the
per-call batching arithmetic for why the mechanism holds at 100k is shown separately in the
answer as plain division, since 100k ticks of per-object bookkeeping would not demonstrate
anything the arithmetic doesn't already show).

Detection has three components sharing ONE last-known-state cache (`detected_tick` /
`known_present`), so nothing is ever double-reported:

1. EVENT path (fast, primary for event-enabled prefixes): an S3 Event Notification
   (ObjectCreated) fires a short fixed delivery delay after the object lands (S3 events are
   at-least-once, typically delivered within seconds, not instantaneous or ordered). Modeled
   here as fully DROPPED (not queued) during the downtime window, deliberately, to stress-test
   the safety net below rather than assume perfect delivery-pipeline durability.
2. POLL-ONLY adaptive sweep (primary for prefixes with no event wiring): a single shared,
   adaptively-backed-off batched List call (pages = ceil(objects currently under the common
   parent prefix / 1000), NOT one call per candidate prefix) scanning only the not-yet-event-
   wired prefixes. Backs off exponentially (cap 64 ticks) when a sweep finds nothing new,
   resets to 1 tick when it does.
3. SAFETY-NET sweep (insurance, not a fast path): a separate, fixed hourly (60-tick) batched
   List call scanning ALL not-yet-detected prefixes regardless of event wiring. Exists purely
   to bound the damage from a dropped or delayed event, deliberately slow and cheap since its
   job is reliability, not speed.

DOWNTIME: ticks 700-900. No events are processed and neither sweep runs. On restart (tick
900) BOTH sweeps are forced immediately, independent of their normal schedules, to demonstrate
recovery: every prefix that actually arrived during the outage must be caught with a bounded
latency, not lost.
"""
import random

random.seed(42)

TOTAL_TICKS = 1440          # one day at 1-minute resolution
N_PREFIXES = 2000           # representative scaled-down N, see module docstring
EVENT_FRACTION = 0.7
MIN_INTERVAL = 1
MAX_INTERVAL = 64
SAFETY_NET_INTERVAL = 60    # hourly
PAGE_SIZE = 1000            # ListObjectsV2 real page size cap
DOWNTIME_START, DOWNTIME_END = 700, 900

prefixes = [f"p{i:05d}" for i in range(N_PREFIXES)]
arrival_tick = {p: random.randint(0, TOTAL_TICKS - 1) for p in prefixes}
is_event_enabled = {p: (random.random() < EVENT_FRACTION) for p in prefixes}
event_delay = 1

detected_tick = {}
known_present = set()

event_calls = 0
poll_sweep_calls = 0
poll_sweep_pages = 0
safety_sweep_calls = 0
safety_sweep_pages = 0

poll_interval = MIN_INTERVAL
next_poll_sweep = 0
next_safety_sweep = 0

event_enabled = [p for p in prefixes if is_event_enabled[p]]
poll_only = [p for p in prefixes if not is_event_enabled[p]]

for tick in range(TOTAL_TICKS):
    downtime = DOWNTIME_START <= tick < DOWNTIME_END

    if not downtime:
        for p in event_enabled:
            if p not in detected_tick and arrival_tick[p] + event_delay == tick:
                detected_tick[p] = tick
                known_present.add(p)
                event_calls += 1

    do_poll_sweep = (not downtime) and (tick == next_poll_sweep or tick == DOWNTIME_END)
    if do_poll_sweep:
        poll_sweep_calls += 1
        currently_arrived = {p for p in poll_only if arrival_tick[p] <= tick}
        newly_found = currently_arrived - known_present
        poll_sweep_pages += max(1, -(-len(currently_arrived) // PAGE_SIZE))
        for p in newly_found:
            detected_tick[p] = tick
        known_present |= newly_found
        poll_interval = MIN_INTERVAL if newly_found else min(poll_interval * 2, MAX_INTERVAL)
        next_poll_sweep = tick + poll_interval

    do_safety_sweep = (not downtime) and (tick == next_safety_sweep or tick == DOWNTIME_END)
    if do_safety_sweep:
        safety_sweep_calls += 1
        currently_arrived_all = {p for p in prefixes if arrival_tick[p] <= tick}
        newly_found_all = currently_arrived_all - known_present
        safety_sweep_pages += max(1, -(-len(currently_arrived_all) // PAGE_SIZE))
        for p in newly_found_all:
            detected_tick[p] = tick
        known_present |= newly_found_all
        next_safety_sweep = tick + SAFETY_NET_INTERVAL

undetected = [p for p in prefixes if p not in detected_tick]
downtime_arrivals = [p for p in prefixes if DOWNTIME_START <= arrival_tick[p] < DOWNTIME_END]
downtime_all_caught_at_restart = all(detected_tick.get(p) == DOWNTIME_END for p in downtime_arrivals)
max_poll_only_latency = max(
    (detected_tick[p] - arrival_tick[p] for p in poll_only if p in detected_tick), default=0)
max_event_latency = max(
    (detected_tick[p] - arrival_tick[p] for p in event_enabled
     if p in detected_tick and not (DOWNTIME_START <= arrival_tick[p] < DOWNTIME_END)), default=0)
events_caught_by_safety_net = sum(
    1 for p in event_enabled if DOWNTIME_START <= arrival_tick.get(p, -1) < DOWNTIME_END)

naive_calls = N_PREFIXES * TOTAL_TICKS

print("N_PREFIXES:", N_PREFIXES, "| event_enabled:", len(event_enabled), "| poll_only:", len(poll_only))
print("event notifications processed:", event_calls)
print("poll-only adaptive sweeps run:", poll_sweep_calls, "| paginated List calls:", poll_sweep_pages)
print("safety-net sweeps run:", safety_sweep_calls, "| paginated List calls:", safety_sweep_pages)
print("total API calls (events + both sweeps' pages):", event_calls + poll_sweep_pages + safety_sweep_pages)
print("naive per-prefix-per-tick call count (no batching/backoff, for comparison):", naive_calls)
print("undetected prefixes at end of sim:", len(undetected), undetected)
print("prefixes that arrived during the downtime window:", len(downtime_arrivals),
      "(", events_caught_by_safety_net, "of these were event-enabled, i.e. their event was dropped by the outage )")
print("all downtime-window arrivals caught exactly at the restart-forced sweep:", downtime_all_caught_at_restart)
print("max detection latency, event path, non-downtime arrivals (ticks):", max_event_latency)
print("max detection latency, poll-only prefixes (ticks):", max_poll_only_latency)

Output (actual, from running the block above with python3, stdlib only, seed=42):

N_PREFIXES: 2000 | event_enabled: 1394 | poll_only: 606
event notifications processed: 1180
poll-only adaptive sweeps run: 649 | paginated List calls: 649
safety-net sweeps run: 21 | paginated List calls: 30
total API calls (events + both sweeps' pages): 1859
naive per-prefix-per-tick call count (no batching/backoff, for comparison): 2880000
undetected prefixes at end of sim: 4 ['p00868', 'p00978', 'p01592', 'p01851']
prefixes that arrived during the downtime window: 266 ( 184 of these were event-enabled, i.e. their event was dropped by the outage )
all downtime-window arrivals caught exactly at the restart-forced sweep: True
max detection latency, event path, non-downtime arrivals (ticks): 201
max detection latency, poll-only prefixes (ticks): 197

Total API load across the whole simulated day is 1,859 calls, against a naive one-call-per-prefix-per-tick baseline of 2,880,000, roughly a 1,550 times reduction, while every single downtime-window arrival (266 of them) is still caught, exactly at the forced restart sweep, proving the safety net actually works rather than merely running without error. The 4 prefixes still undetected at the very end of the simulated day all arrived at tick 1439, the last tick simulated, one tick too late for either their event delay or the next sweep to fire before the simulation window itself ends; this is an artifact of the simulation stopping at exactly one day, not a defect in the design, since a continuously running system would simply catch them on the next tick. The 201-tick worst-case event-path latency belongs to a single prefix whose object landed at tick 699, one tick before the downtime window, but whose event notification, delayed by the fixed 1-tick delivery lag, would have fired at tick 700, squarely inside the outage, so it too was only caught by the forced restart sweep at tick 900, a genuine and informative edge case: an outage's effective blast radius on the event path extends slightly earlier than its own start time, by however long event delivery normally takes.

Key points, complexity, and edge cases

Complexity: each poll-only or safety-net sweep is a single batched call costing $O(\lceil k / 1000 \rceil)$ paginated List requests, where $k$ is the number of objects currently under the scanned prefix, not $O(N)$ in the number of watched prefixes; this decoupling of API-call count from prefix count is the entire mechanism that avoids one sensor per prefix. The simulation loop itself is $O(T \times N)$ in the worst case, $T$ ticks times $N$ prefixes, purely as an artifact of simulating every tick explicitly in Python; a real deployment has no equivalent per-tick cost, only per-sweep cost.

Edge cases: a prefix whose object lands in the final simulated tick has no later tick in which its event delay or next sweep can fire, the 4 permanently-undetected prefixes above, an artifact of the simulation window ending, not a defect in the mechanism, disclosed explicitly rather than adjusted away. A prefix whose object lands one tick before downtime begins, but whose event's fixed delivery delay would push the actual notification into the downtime window, is still correctly caught, just later, by the forced restart sweep, demonstrated by the 201-tick worst-case event-path latency traced to exactly this case.

Trade-offs and pitfalls

Modeling event delivery as fully dropped during downtime, rather than durably queued (as a real SNS/SQS-backed pipeline usually is), was a deliberate choice to stress-test the safety net; a real deployment backed by a durable queue would lose far fewer, possibly zero, events during an outage of the consumer alone, but the safety-net sweep is still worth keeping, since it also protects against outages of the event pipeline itself, not only the consumer, and against silent misconfiguration where a new prefix was never wired to event notifications in the first place.

The safety-net sweep's fixed hourly cadence is a deliberate trade: making it faster would shrink worst-case latency for a dropped event but would erode the very cost advantage that justifies not just running the poll-only sweep against everything all the time.

The adaptive poll-only sweep's exponential backoff means a prefix that arrives just after the interval has grown large waits up to that full interval before being caught; this is the direct cost of adaptivity, and a system with a tighter latency requirement for poll-only prefixes specifically would need a lower MAX_INTERVAL cap, trading away some of the call-count savings for tighter worst-case latency.