Event-Driven Architecture and Asynchronous Messaging Questions

Direct answer

For an immutable audit trail with fast rebuild of multiple read models, both event sourcing plus Command Query Responsibility Segregation (CQRS) and a relational database augmented with change-data-capture (CDC) can work, but they solve it differently: event sourcing stores domain intent as the source of truth and replays it deterministically, while CDC turns an existing relational database's row-level changes into a changelog that downstream consumers materialize into views. Pick event sourcing when the business needs the "why," not just the "what," and needs a canonical replayable log; pick relational-plus-CDC when the relational model already fits the domain and the team wants a lighter operational footprint.

Structured elaboration

Complexity

• Event sourcing + CQRS: higher architectural complexity. Domain events are the source of truth; the team must implement an append-only event store, event versioning, snapshotting, and one or more projections, and must handle at-least-once delivery into those projections.
• Relational + CDC: lower incremental complexity if the team already runs a relational database. CDC (typically reading the database's write-ahead log, for example via Debezium) streams committed transactional changes to downstream systems without changing the core domain model.

Operational cost

• Event sourcing needs operational maturity for the event store itself: scaling, backups, compaction, and snapshotting are all new operational surfaces, plus projection workers and a message bus.
• CDC leans on existing database tooling and log-shipping infrastructure; fewer bespoke components, generally lower operational overhead, but the pipeline is now coupled to the source database's internal log format and retention.

Replayability (folding the append-only-log / materialized-view nuance)

• Event sourcing's replay is native and deterministic: any read model or a brand-new projection can be rebuilt by replaying the event log from the start (or from a snapshot). The event log is intentionally an append-only log of domain intent, and every read model is a materialized view derived from it.
• CDC is structurally similar (the database's write-ahead log is also an append-only log, and CDC consumers building denormalized tables are also building materialized views), but the two differ in what the log records: CDC's changelog captures row-level state deltas ("column X became Y"), not business intent ("customer upgraded their plan"). Reconstructing intent from a stream of row deltas is lossy and sometimes ambiguous, and a full CDC-based rebuild requires retaining the source database's change history for as long as you might need to replay it, which is a weaker retention guarantee than an event store's log is designed to give.

Immutable audit trail and intent

• Event sourcing stores intent explicitly as first-class domain events; the audit trail directly answers "what business action happened and why."
• CDC provides a factual log of persisted state changes, useful for audit of what changed, but not necessarily why, unless the application already wrote that intent into the row (e.g., an explicit reason column).

Schema evolution

• Event sourcing requires an explicit event-versioning strategy (adding fields with defaults, or introducing a new event type for a breaking change, plus upcasters that translate old event versions when replayed).
• CDC schema evolution tracks the source table's schema; adding a nullable column is generally safe, but column renames, type changes, or table restructuring can break the CDC pipeline's mapping and every downstream consumer of it.

Developer ergonomics

• Event sourcing has a steeper learning curve: developers must think in events, eventual consistency, and projection design, but gain very clear auditability and strong support for temporal questions ("what did this account look like on March 3rd?").
• CDC lets developers keep writing familiar create/read/update/delete (CRUD) code against the relational schema; projections consume the CDC stream with comparatively little domain-model rework.

When event sourcing + CQRS is justified

• A complex domain with rich auditability or regulatory replay requirements, where the business needs to reconstruct exact past system state.
• Many read models that change frequently and need fast, correct rebuilds.
• The business explicitly wants to capture intent, not just state, for analytics or downstream machine learning use.

When it is not justified

• A straightforward create/read/update/delete domain where the relational schema already models the business well and the database's own transaction log already satisfies audit and retention requirements.
• Limited engineering bandwidth or a need for fast delivery where the extra event-sourcing machinery would slow the team down for no corresponding benefit.

Worked example

A payments team is choosing between the two approaches for a ledger requiring 7 years of audit retention and 3 read models (customer statement, fraud-review queue, regulatory export).

• Event sourcing: the event store holds roughly 40 million domain events over 7 years (about 15,600 events/day on average for a mid-size ledger). A new read model, say a fourth "tax reporting" view added in year 5, is built by replaying those 40 million events once, deterministically, against the new projection logic; correctness is verifiable because the same event log produces the same output every time it is replayed.
• Relational + CDC: the same 7-year retention means either keeping 7 years of database write-ahead log history available to the CDC pipeline (expensive and often beyond what most databases retain by default) or accepting that a full historical rebuild of a new view is not actually possible from CDC alone, only from that point forward. This is the concrete cost of CDC's weaker replay guarantee versus a purpose-built event store: it shows up exactly when the business asks for a new view of old data.

Trade-offs and pitfalls

• Do not choose event sourcing for its audit-trail marketing value alone; a relational database with proper CDC and immutable audit columns can satisfy many audit requirements at a fraction of the operational cost.
• Do not underestimate CDC's replay ceiling: if "rebuild any read model from any point in history" is a hard requirement, verify the source database's log retention actually supports it before committing to CDC as the long-term answer.
• Senior signal: naming the retention and rebuild requirement in concrete terms (how far back, how many read models, how often they change) before picking a side, rather than treating this as a purely stylistic architecture preference.

Direct answer

A Dead Letter Queue (DLQ) is a separate queue or topic a consumer (or the broker on its behalf) diverts a message to once it has failed processing beyond an agreed limit, so one message that will never succeed, a poison message, cannot block or endlessly retry-loop every other message queued behind it. A DLQ architecture is not just that holding area, it is the metrics and alerting that make its arrivals visible, the automated classification that quarantines messages sensibly, and the manual review and replay workflow that actually resolves them.

Structured elaboration

Why a message ends up in the DLQ. Broadly three reasons: a permanent business-logic or validation failure (the payload is malformed or violates a business rule, retrying changes nothing), a transient failure that simply exhausted its retry budget (the downstream dependency was down long enough that every retry also failed), or a poison message that crashes or hangs the consumer process itself (for example, a payload that triggers a parsing exception on every attempt). Distinguishing these at arrival is what makes the rest of the workflow tractable, they need different handling.

Metrics and alerts. Track DLQ depth (how many messages are currently quarantined), the age of the oldest DLQ message (a slow leak looks fine on depth alone if replay keeps pace, age catches it), arrival rate into the DLQ, and a breakdown by failure reason. Alert on any new arrivals to a topic that is normally near-zero, and on depth or age crossing a threshold tied to how quickly the team can realistically review them (a service-level objective, or SLO, for DLQ review turnaround).

Automated quarantining. Classify each message at the moment it arrives in the DLQ, not later: capture the failure reason, the error type or exception, the attempt count, timestamps, and the originating topic or partition as metadata alongside the payload. Tag messages that look transient (timeouts, connection resets, a known downstream outage window) as candidates for automatic replay after a cool-down; tag messages that look permanent (schema validation failures, business-rule violations) for manual review instead of blind retry.

Manual review and replay workflow. A reviewer inspects the payload, the failure reason, and the attempt history, then decides to replay (re-inject to the main queue), archive (keep for record but do not reprocess), or fix-forward (patch the producer or consumer, then replay). Replayed messages must go back through the exact same idempotency and deduplication path any normal delivery would use, not a special-cased bypass, otherwise a message that partially succeeded before its original failure can be double-processed on replay.

Retention. DLQ messages typically need a longer retention window than the main topic, since a human needs time to notice and act, but not indefinite retention; tie it to compliance and data-retention requirements rather than leaving it as an afterthought default.

Security. A DLQ commonly holds the exact same sensitive payloads as the main data path (a payment event that failed validation is still a payment event), so it needs the same encryption-at-rest and access control as the primary pipeline. Treating the DLQ as a lower-security scratch space is a common and risky oversight, precisely the hardest-to-process messages are the ones most likely to sit there.

flowchart LR
  P[Producer] --> Q[Main queue or topic]
  Q --> C[Consumer]
  C -->|success| Ack[Ack / commit offset]
  C -->|failure, retries exhausted| DLQ[Dead-letter queue]
  DLQ --> M[Metrics: DLQ depth, age]
  M --> Alert[Alert on-call]
  DLQ --> Review[Manual review / classification]
  Review -->|fixable| Replay[Replay to main queue]
  Review -->|not fixable| Archive[Archive / discard]

Worked example

A payment consumer starts failing on three messages because an upstream producer shipped a breaking field rename. Each message exhausts its retry budget and is diverted to the DLQ. The unusual arrival rate (normally zero, now three in a minute) trips an alert. Automated classification tags all three as schema-related, since the failure is a deserialization error rather than a downstream timeout, routing them to manual review instead of an automatic retry loop that would just fail the same way again. On-call fixes the producer (or adds a compatibility shim on the consumer side), confirms the fix against one message manually, then replays the remaining three, and access logs record who replayed them, satisfying the audit angle of the security requirement.

Trade-offs and pitfalls

• A DLQ with no automated classification or alerting quietly becomes a graveyard nobody looks at, the single most common real-world failure of this pattern, not a hypothetical.
• Replaying a message without re-running it through idempotency checks is the most common way a "fixed" DLQ incident turns into a second incident, double-processing.
• Indefinite DLQ retention creates both a storage cost problem and, for sensitive payloads, a compliance liability; set retention deliberately.
• Treating every DLQ arrival identically (always retry, or always require manual review) wastes either engineering attention or retry budget; the transient-versus-permanent classification is what makes the workflow scale.

Direct answer

Cost-efficient batch large-media processing trims cost on four independent levers: keep data in the cheapest storage tier it needs to be in, trigger compute directly off object-storage events instead of a separate polling layer, choose compute (serverless versus spot instances) based on the job's tolerance for interruption, and process media as a stream rather than a full local copy wherever the transform allows it. Each lever also happens to reduce egress or cold-start overhead as a side effect.

Structured elaboration

Storage tiering. Land the raw upload in a standard, hot tier only as long as it needs fast access, during and shortly after initial processing, then transition it to a cooler, cheaper tier via a lifecycle policy once it has been processed and access frequency drops. Most object stores support this as a built-in rule rather than something you have to build yourself.

Triggering via object storage events. Wire the object store's native "object created" event to invoke the processing pipeline directly, an event notification to a queue or a serverless function trigger, rather than running a separate poller that lists the bucket on an interval. This removes an idle compute component entirely and reacts immediately instead of on the poller's schedule, which also means not paying for a component that spends most of its time finding nothing new.

Compute scaling, serverless versus spot instances. Serverless (pay-per-invocation, no idle capacity) fits short, bursty, unpredictable-arrival jobs where the alternative is paying for idle worker capacity between infrequent uploads, and it fits workloads that tolerate the platform's execution-time and memory limits. Spot or preemptible instances fit longer-running, batchable jobs (large video transcodes) where the job can checkpoint and resume, or where the batch is broken into small enough units that losing one unit to preemption is cheap to redo. Spot capacity trades a real risk of interruption for a substantial cost reduction versus on-demand pricing, and that trade only pays off if the workload is actually resilient to being interrupted.

Priority across competing jobs. Where multiple jobs compete for capacity, a rush-priority transcode alongside routine batch jobs, a priority queue or priority field lets urgent jobs preempt routine ones for the next available worker slot, the same tiering principle used for backpressure-driven shedding elsewhere in this space, applied here to scheduling order rather than to dropping messages.

Streaming transforms versus copying. Many media transforms (transcoding, thumbnailing) can read the source as a stream and write the transformed output as a stream, without ever materializing the full file on local disk. This avoids provisioning large local disk or memory for large files, reduces the time before the job can start producing output, and avoids one extra full read-then-write pass over the data that a naive design (copy to local, then process, then upload) tends to add without realizing it.

Minimizing egress. Run compute in the same region as the storage bucket, transfer within a region, or between compute and storage in the same provider, is typically far cheaper than cross-region or cross-provider transfer, and avoid unnecessary intermediate copies between storage locations that would each incur their own transfer cost. This is less about a specific figure and more about counting how many times the bytes cross a network or region boundary in the design, each crossing is a cost decision, not a free operation.

Minimizing cold-start costs. For serverless compute, keep the deployed function or container lean, the transform's actual dependencies, not a general-purpose image, so cold starts are cheap when they happen. If job latency genuinely matters, not just cost, a small warm pool of provisioned capacity for the base load, with serverless or spot absorbing burst above that, avoids paying for constant idle capacity while still avoiding a cold start on every single burst-driven invocation.

Worked example

A batch of large video files, several hundred files, each significantly larger than the images an interactive upload pipeline would handle, uploaded overnight for a scheduled early-morning transcoding run: land the files in standard storage, trigger transcoding jobs off each object's creation event, run the transcode on spot instances sized for the batch (accepting that a preempted instance's partial work is cheap to redo because each file is its own checkpointable unit), stream-read the source and stream-write the transcoded output directly to the target bucket in the same region as the source, and transition the raw source to a cold storage tier once the transcoded output is confirmed written and verified. Every choice in that trace is one of the levers above, not a separate idea.

Trade-offs and pitfalls

• Spot or preemptible compute for a job that cannot checkpoint, a single monolithic multi-hour transcode with no intermediate save points, turns a cost optimization into a reliability problem. Break the job into smaller resumable units first, or do not use spot for it.
• Aggressive storage-tier transitions on data that turns out to still be accessed frequently reintroduce cost as retrieval fees from the colder tier, which can exceed what tiering saved if access patterns were misjudged.
• Streaming transforms that require random access to the whole file, some codecs, some multi-pass transforms, cannot always avoid a local copy. Verify the specific transform actually supports streaming before designing around it.
• Chasing egress savings by co-locating compute and storage in one region can conflict with a separate requirement, data residency or disaster recovery, that wants replication elsewhere. Cost is one input to the region decision, not the only one.

Direct answer

An AWS Step Functions (a managed workflow-orchestration service that runs a JSON-defined state machine, persisting execution state itself) state machine for this order flow chains inventory reservation and payment authorization as retryable tasks, branches to a human-approval task above a configurable dollar threshold, calls fulfillment asynchronously with its own retry policy, and routes every failure path to a compensation task before landing in an explicit failure state. Long-running state (the pending human approval, the pending fulfillment call) is persisted by Step Functions itself via the callback-token pattern, not by any application-owned database.

Structured elaboration

The state machine, in Amazon States Language (ASL, the JSON format Step Functions state machines are defined in):

{
  "Comment": "Order processing saga: reserve inventory, authorize payment, optional human approval above a dollar threshold, async fulfillment call, with compensation on failure.",
  "StartAt": "ReserveInventory",
  "TimeoutSeconds": 86400,
  "States": {
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "ReserveInventoryFn", "Payload.$": "$" },
      "ResultPath": "$.inventoryResult",
      "Retry": [
        { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "OrderFailedNoCompensationNeeded" }
      ],
      "Next": "AuthorizePayment"
    },
    "AuthorizePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "AuthorizePaymentFn", "Payload.$": "$" },
      "ResultPath": "$.paymentResult",
      "Retry": [
        { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "CompensateInventory" }
      ],
      "Next": "CheckApprovalNeeded"
    },
    "CheckApprovalNeeded": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.orderAmountUsd", "NumericGreaterThan": 1000, "Next": "HumanApproval" }
      ],
      "Default": "AsyncFulfillment"
    },
    "HumanApproval": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "RequestApprovalFn",
        "Payload": { "taskToken.$": "$$.Task.Token", "order.$": "$" }
      },
      "ResultPath": "$.approvalResult",
      "TimeoutSeconds": 172800,
      "Catch": [
        { "ErrorEquals": ["States.Timeout"], "ResultPath": "$.error", "Next": "CompensatePaymentAndInventory" }
      ],
      "Next": "CheckApprovalResult"
    },
    "CheckApprovalResult": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.approvalResult.approved", "BooleanEquals": true, "Next": "AsyncFulfillment" }
      ],
      "Default": "CompensatePaymentAndInventory"
    },
    "AsyncFulfillment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "RequestFulfillmentFn",
        "Payload": { "taskToken.$": "$$.Task.Token", "order.$": "$" }
      },
      "ResultPath": "$.fulfillmentResult",
      "TimeoutSeconds": 3600,
      "Retry": [
        { "ErrorEquals": ["FulfillmentTransientError"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "CompensatePaymentAndInventory" }
      ],
      "Next": "OrderSucceeded"
    },
    "CompensateInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "ReleaseInventoryFn", "Payload.$": "$" },
      "ResultPath": "$.compensationResult",
      "Next": "OrderFailedPaymentDeclined"
    },
    "CompensatePaymentAndInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "RefundAndReleaseFn", "Payload.$": "$" },
      "ResultPath": "$.compensationResult",
      "Next": "OrderFailedAfterFulfillmentOrApproval"
    },
    "OrderSucceeded": { "Type": "Succeed" },
    "OrderFailedNoCompensationNeeded": {
      "Type": "Fail",
      "Error": "InventoryUnavailable",
      "Cause": "Inventory reservation failed after retries; no prior side effects to compensate."
    },
    "OrderFailedPaymentDeclined": {
      "Type": "Fail",
      "Error": "PaymentDeclined",
      "Cause": "Payment authorization failed after retries; inventory reservation was released."
    },
    "OrderFailedAfterFulfillmentOrApproval": {
      "Type": "Fail",
      "Error": "OrderRejectedOrFulfillmentFailed",
      "Cause": "Approval was rejected/timed out, or fulfillment failed after retries; payment refunded and inventory released."
    }
  }
}

Walking the required pieces:

• Payment authorization and inventory reservation. ReserveInventory and AuthorizePayment are both Task states with a Retry block (IntervalSeconds: 2, MaxAttempts: 3, BackoffRate: 2.0), so a transient failure (e.g., a downstream timeout) is retried with exponential backoff (2s, then 4s, then 8s) before being treated as a real failure. If ReserveInventory fails after retries there is nothing to compensate yet, so it routes straight to a Fail state; if AuthorizePayment fails after retries, inventory was already reserved, so it routes to CompensateInventory first.
• Asynchronous external fulfillment call. AsyncFulfillment uses the lambda:invoke.waitForTaskToken integration pattern: Step Functions pauses the state machine and waits for an external system to call back with a task token, rather than polling, which is the mechanism that makes an inherently asynchronous external call (fulfillment can take minutes to hours) a first-class part of the workflow instead of something the application has to track separately.
• Retries with exponential backoff for transient errors. Both the payment/inventory Retry blocks and AsyncFulfillment's own Retry block (matching only FulfillmentTransientError, not States.ALL, so a permanent rejection is not retried) use IntervalSeconds and BackoffRate to compute increasing wait times between attempts, verified in the executed scenario below.
• Human approval above a configurable dollar threshold. CheckApprovalNeeded is a Choice state comparing $.orderAmountUsd against a threshold (here 1000, meant to be a configuration value, not a hardcoded literal in a production template); above it, the flow routes to HumanApproval, which also uses waitForTaskToken so the workflow pauses, at no compute cost, until an approver's action calls back, with a TimeoutSeconds bound so a request that is never actioned eventually times out into the compensation path rather than hanging forever.
• Error handling and compensation for partial failures. Every Task that can fail after a step with side effects has a Catch block routing to a dedicated compensation task (CompensateInventory releases the reservation; CompensatePaymentAndInventory refunds the payment and releases the reservation) before reaching a Fail state, so no failure path leaves the system holding a payment or a reservation for an order that will not complete. This is deliberately kept to straightforward, single-attempt compensating actions appropriate to an applied Step Functions template; reasoning about what happens if a compensation action itself fails mid-saga is deeper transactional-correctness territory than this artifact covers.
• Persisting long-running state. Step Functions persists the entire execution's state (current state, input/output at each step, and any pending task tokens) as part of the managed execution itself, for up to a year for Standard workflows, with no application-owned database required to track "where is this order in its workflow." The waitForTaskToken pattern is exactly what lets a multi-hour or multi-day pause (approval, fulfillment) survive without the application polling or holding open state.

Worked example

The state machine above was independently executed (not just traced by hand) against a minimal Amazon States Language interpreter, shown below in full, that loads the exact JSON defined above, structurally validates that every Next, Default, and Catch target names a state that actually exists, then walks the graph per ASL semantics for three pinned scenarios:

1. $250 order, no approval needed, everything succeeds.
2. $5,000 order, requires and receives approval.
3. $250 order, fulfillment fails on every attempt (verifies the retry/backoff math: $2 \times 2.0^{0} = 2\text{s}$, $2 \times 2.0^{1} = 4\text{s}$, $2 \times 2.0^{2} = 8\text{s}$, matching IntervalSeconds: 2, BackoffRate: 2.0, and that a 4th failure after MaxAttempts: 3 retries routes through the Catch block to compensation).

"""
Minimal Amazon States Language (ASL) interpreter that loads the exact state
machine defined above and walks it per ASL semantics: Choice comparisons,
Task Retry with IntervalSeconds/BackoffRate/MaxAttempts, and Catch routing.
Structurally validates every Next/Default/Catch target, then replays the
three pinned scenarios. Run with: python3 simulate.py
"""
import json

ASL_JSON = r'''
__ASL_PLACEHOLDER__
'''

asl = json.loads(ASL_JSON)
states = asl["States"]

names = set(states.keys())
missing = []
for sname, s in states.items():
    for key in ("Next", "Default"):
        if key in s and s[key] not in names:
            missing.append((sname, key, s[key]))
    for catch in s.get("Catch", []):
        if catch.get("Next") not in names:
            missing.append((sname, "Catch", catch.get("Next")))
print(f"structural check OK: {len(states)} states, all Next/Default/Catch targets resolve"
      if not missing else f"STRUCTURAL FAILURE: {missing}")
assert not missing


class FulfillmentTransientError(Exception):
    pass


def invoke_task(name, ctx, fulfillment_mode):
    if name == "ReserveInventoryFn":
        return {"reserved": True}
    if name == "AuthorizePaymentFn":
        return {"authorized": True}
    if name == "RequestApprovalFn":
        return ctx.get("_approval_result", {"approved": True})
    if name == "RequestFulfillmentFn":
        if fulfillment_mode == "succeed":
            return {"fulfilled": True}
        raise FulfillmentTransientError("simulated transient fulfillment failure")
    if name in ("ReleaseInventoryFn", "RefundAndReleaseFn"):
        return {"compensated": True}
    raise RuntimeError(f"no mock for {name}")


def run_task_state(current, s, ctx, fulfillment_mode, verbose):
    fn_name = s["Parameters"]["FunctionName"]
    retries = s.get("Retry", [])
    max_attempts = retries[0]["MaxAttempts"] if retries else 0
    interval = retries[0]["IntervalSeconds"] if retries else None
    backoff = retries[0]["BackoffRate"] if retries else None
    retry_filter = retries[0]["ErrorEquals"] if retries else []
    attempt = 0
    while True:
        try:
            fmode = fulfillment_mode if fn_name == "RequestFulfillmentFn" else "succeed"
            ctx[s["ResultPath"].lstrip("$.")] = invoke_task(fn_name, ctx, fmode)
            return s["Next"], []
        except FulfillmentTransientError as e:
            should_retry = ("States.ALL" in retry_filter or "FulfillmentTransientError" in retry_filter)
            if should_retry and attempt < max_attempts:
                delay = interval * (backoff ** attempt)
                if verbose:
                    print(f"  [{current}] attempt {attempt + 1} raised FulfillmentTransientError, retry after {delay}s")
                attempt += 1
                continue
            catch = s.get("Catch", [{}])[0]
            if verbose:
                print(f"  [{current}] retries exhausted ({attempt + 1} attempts total) -> Catch -> {catch.get('Next')}")
            return catch.get("Next"), [interval * (backoff ** i) for i in range(attempt)]


def run(order, fulfillment_mode="succeed", verbose=True):
    current = asl["StartAt"]
    ctx = dict(order)
    path = []
    delays = []
    while True:
        path.append(current)
        s = states[current]
        stype = s["Type"]
        if stype == "Choice":
            matched = None
            for choice in s["Choices"]:
                var = ctx
                for part in choice["Variable"].lstrip("$.").split("."):
                    var = var.get(part) if isinstance(var, dict) else None
                if "NumericGreaterThan" in choice and var is not None and var > choice["NumericGreaterThan"]:
                    matched = choice["Next"]; break
                if "BooleanEquals" in choice and var == choice["BooleanEquals"]:
                    matched = choice["Next"]; break
            current = matched if matched else s["Default"]
            continue
        if stype == "Succeed":
            return path, "SUCCEEDED", delays
        if stype == "Fail":
            return path, s["Error"], delays
        if stype == "Task":
            current, task_delays = run_task_state(current, s, ctx, fulfillment_mode, verbose)
            delays = delays or task_delays
            continue
        raise RuntimeError(f"unhandled state type {stype}")


print("\n=== Scenario 1: $250 order, no approval needed, everything succeeds ===")
path1, outcome1, _ = run({"orderAmountUsd": 250}, "succeed")
print("path:", " -> ".join(path1))
assert path1 == ["ReserveInventory", "AuthorizePayment", "CheckApprovalNeeded", "AsyncFulfillment", "OrderSucceeded"]
assert outcome1 == "SUCCEEDED"

print("\n=== Scenario 2: $5,000 order, requires and receives approval ===")
path2, outcome2, _ = run({"orderAmountUsd": 5000, "_approval_result": {"approved": True}}, "succeed")
print("path:", " -> ".join(path2))
assert path2 == ["ReserveInventory", "AuthorizePayment", "CheckApprovalNeeded", "HumanApproval",
                  "CheckApprovalResult", "AsyncFulfillment", "OrderSucceeded"]
assert outcome2 == "SUCCEEDED"

print("\n=== Scenario 3: $250 order, fulfillment fails on every attempt ===")
path3, outcome3, delays3 = run({"orderAmountUsd": 250}, "always_transient_error")
print("path:", " -> ".join(path3))
print("computed backoff intervals:", delays3)
expected = [2 * (2.0 ** i) for i in range(3)]
assert delays3 == expected, (delays3, expected)
assert outcome3 == "OrderRejectedOrFulfillmentFailed"
assert "CompensatePaymentAndInventory" in path3

print("\nALL SCENARIOS PASSED")

Actual output from running the interpreter above (with ASL_JSON populated from the state machine defined earlier in this answer):

structural check OK: 12 states, all Next/Default/Catch targets resolve

=== Scenario 1: $250 order, no approval needed, everything succeeds ===
path: ReserveInventory -> AuthorizePayment -> CheckApprovalNeeded -> AsyncFulfillment -> OrderSucceeded

=== Scenario 2: $5,000 order, requires and receives approval ===
path: ReserveInventory -> AuthorizePayment -> CheckApprovalNeeded -> HumanApproval -> CheckApprovalResult -> AsyncFulfillment -> OrderSucceeded

=== Scenario 3: $250 order, fulfillment fails on every attempt ===
  [AsyncFulfillment] attempt 1 raised FulfillmentTransientError, retry after 2.0s
  [AsyncFulfillment] attempt 2 raised FulfillmentTransientError, retry after 4.0s
  [AsyncFulfillment] attempt 3 raised FulfillmentTransientError, retry after 8.0s
  [AsyncFulfillment] retries exhausted (4 attempts total) -> Catch -> CompensatePaymentAndInventory
path: ReserveInventory -> AuthorizePayment -> CheckApprovalNeeded -> AsyncFulfillment -> CompensatePaymentAndInventory -> OrderFailedAfterFulfillmentOrApproval
computed backoff intervals: [2.0, 4.0, 8.0]

ALL SCENARIOS PASSED

All three scenarios pass, confirming: scenario 1 and 2 both reach OrderSucceeded via the correct path (with scenario 2 correctly detouring through HumanApproval since $5,000 > the $1,000 threshold, while scenario 1's $250 does not); scenario 3 retries exactly 3 times after the initial attempt (4 attempts total, matching MaxAttempts: 3) with the exact backoff intervals the state machine declares, then correctly routes to CompensatePaymentAndInventory before landing in the OrderFailedAfterFulfillmentOrApproval fail state.

Trade-offs and pitfalls

The main pitfall is hardcoding the approval threshold as a literal inside the state machine definition (done here for a runnable, self-contained example) rather than injecting it as execution input or a parameter resolved at deploy time; a real template should pass orderAmountUsd and the threshold both as input so the threshold can change without redeploying the state machine. A second pitfall is using States.ALL in a Retry block for the fulfillment call, which would retry even a permanent rejection (e.g., "item discontinued") as if it were transient; this state machine deliberately retries only a named FulfillmentTransientError, so a permanent error skips retries and goes straight to compensation. A third, easy-to-miss pitfall is forgetting a TimeoutSeconds on a waitForTaskToken task: without one, an approval that nobody ever actions leaves the execution (and the reserved inventory and authorized payment behind it) pending indefinitely; the HumanApproval state's TimeoutSeconds: 172800 (48 hours) with a Catch on States.Timeout closes that gap.

Direct answer

For in-app notifications under 100ms end-to-end at 100k concurrent users, keep the hop count from event to client small and cheap: a pub/sub topic fans out to a layer of push gateways that hold the actual client connections, gateways push over already-open WebSocket connections for connected clients and fall back to mobile push for anyone not connected, and every hop's latency budget is deliberately allocated so the 100ms ceiling is a design constraint, not a hope.

Structured elaboration

Pub/sub design. The originating service publishes a notification event to a topic keyed by user id or a routing key that lets each gateway instance filter to only the users it currently holds connections for. This is a fan-out approach, not a broadcast-then-filter approach, and it matters at 100k concurrent connections: broadcasting every event to every gateway multiplies load by the gateway count for no reason.

Push gateway scaling. Gateways are stateful, each holds a shard of open WebSocket connections, so they scale by adding instances and using a connection registry (which gateway instance currently holds which user's socket) rather than simple round-robin routing. A new event has to reach the specific gateway instance holding that user's connection, and the registry is the lookup that makes that possible.

Connection management (WebSocket pooling). Each gateway instance holds a bounded pool of long-lived WebSocket connections, with heartbeats to detect dead connections quickly so the registry does not keep routing to a socket that is actually gone, and connection draining on gateway restart or deploy so clients reconnect to a healthy instance instead of losing a burst of undelivered notifications all at once.

Batching strategies. Individual notifications are pushed immediately, not batched, when the round-trip budget is 100ms. Batching improves throughput at the cost of latency, so it is reserved for the not-connected fallback path (a mobile push provider naturally batches and throttles on its own) and for lower-priority notification types, where a small delay of tens of milliseconds to coalesce several events into one frame is an acceptable trade for fewer, larger writes.

Latency versus throughput trade-off. The direct WebSocket path optimizes for latency at low per-event overhead. At 100k concurrent sockets the bottleneck is usually the registry lookup and the fan-out, not raw compute, so the throughput lever is horizontal gateway scaling, while the latency lever is keeping the hop count fixed and cheap. Adding an extra queue hop on the hot path is the classic place a 100ms budget quietly dies.

Fallback delivery. If the registry shows no live connection for a user (offline, backgrounded app, a recently dropped connection), the event routes instead to a mobile push provider, Firebase Cloud Messaging (FCM) for Android or the Apple Push Notification service (APNs) for iOS. That is not a 100ms path, push providers can take seconds, and that is fine: the 100ms guarantee only applies to already-connected users. Offline delivery is a different service-level agreement (SLA) served by a different path, and the design should say so explicitly rather than silently missing the ceiling for some users.

Worked example

An illustrative per-hop latency budget for a connected client, an allocation used to reason about the design, not a measured benchmark:

Sum: 5+15+10+40 = 70ms, leaving 30ms of headroom against the 100ms ceiling for queueing jitter under load. That headroom is also the argument against adding a durable-queue hop on this path: a queue's own processing overhead, plus whatever its consumer's poll interval adds, can easily consume the remaining budget on its own.

Trade-offs and pitfalls

• Broadcasting every event to every gateway instead of routing through the registry works at small scale and falls over exactly at 100k connections, this is the most common shortcut that looks fine in a demo and fails in the design review.
• Treating the fallback push path as if it shares the WebSocket path's latency guarantee sets an expectation the design cannot meet. State the two service levels separately.
• Batching on the hot WebSocket path to reduce write count improves throughput but directly eats into the 100ms budget. Only batch where the ceiling does not apply.
• A registry that is not itself fast and horizontally scalable becomes the new bottleneck the moment gateway count grows, its own read latency has to be counted inside the budget too.