Approach: treat poison messages as a predictable failure class and combine queue features (visibility timeout, DLQ) with orchestrator-level throttles, per-message backoff, and safe reprocessing workflows. Goals: avoid retry storms, surface DLQ problems quickly, and allow controlled reprocessing.Design (components & behavior)- Queue + Dead-Letter Queue (DLQ): Configure redrive policy (maxReceiveCount) so messages failing N times move to DLQ automatically. Set DLQ retention and access controls.- Visibility timeouts & dynamic extension: Set initial visibility to expected processing time + margin. On long-running tasks, orchestrator periodically extends visibility. If handler fails early, leave message visible for quick retry (subject to backoff rules).- Per-message exponential backoff with jitter: On failure increment attempt counter (in message attributes or external metadata store). Use ChangeMessageVisibility to set visibility = base * 2^(attempt-1) ± jitter, capped to max_backoff (e.g., 1 hour). This pushes flaky messages out of circulation and avoids synchronized retries.- Prevent retry storms: enforce global concurrency limits (token bucket / semaphore) per queue/orchestrator worker pool; when DLQ rate spikes, move to circuit-breaker: stop pulling new messages or increase backoff for entire queue.- Idempotency & safe processing: require idempotent task handlers (dedupe keys, upserts, idempotency tokens) so retries are safe. Persist processing state to allow resume/skip.- Alerting & monitoring: emit metrics — DLQ rate, average receive count, visibility extension rate, processing error rate. Alert thresholds: DLQ rate > X/min or >Y% of incoming → urgent paging. Use dashboards and runbooks.- Safe reprocessing strategy: treat DLQ as staging area. Provide tooling to: - Inspect and annotate DLQ messages - Bulk requeue with controlled replay rate and increased backoff - Replay into a "canary" queue first (small throttle) to validate fixes - Track reprocessed message IDs to avoid double-processing- Operational playbook: automatic move to DLQ after N attempts, on DLQ alert: run quick triage (sample messages), root-cause, patch handler or data, then replay via canary with monitoring. If message is irrecoverable, archive for compliance.Example backoff pseudocode:python
# assume msg.attributes['attempts'] present
attempts = int(msg.attributes.get('attempts',0)) + 1
backoff = min(BASE * (2**(attempts-1)), MAX_BACKOFF)
backoff_with_jitter = backoff * (0.8 + random.random()*0.4)
change_message_visibility(msg, visibility_seconds=int(backoff_with_jitter))
update_attempts_metadata(msg, attempts)
if attempts >= MAX_ATTEMPTS:
send_to_dlq(msg)
