**Title**Flaky worker: intermittent failures processing SQS messages — “Worker 420: Message processing error 500/timeout”**Impact**Intermittent message loss and delayed downstream processing (~5–10% of messages over last 48h). Leads to delayed emails and incomplete analytics updates; customer-visible latency spikes.**Steps to reproduce**- No deterministic repro. Failure observed under production load (peak CPU/throughput). Suspected when queue depth > 2000 and messages with >3KB payloads.- Probable repro: send 5k messages with 4KB payloads while scaling down workers to 2; run for 10–20 minutes.**Sample log snippets**text
2026-03-02T14:03:21Z ERROR worker-420 ProcessingMessage id=abc123 svc=worker error="context deadline exceeded"
2026-03-02T14:03:21Z WARN worker-420 Message id=abc123 retry=1 visibilityTimeout=30s
2026-03-02T14:03:22Z ERROR worker-421 DB commit failed id=def456 error="pq: canceling statement due to statement timeout"
**Environment & timestamps**- Prod cluster: aws eu-west-1, ECS Fargate (service=worker-svc, task def v2.3.1)- DB: RDS Postgres 13 (db.t3.medium)- Queue: AWS SQS standard, visibility timeout 30s- Observed window: 2026-03-01 09:00 UTC — 2026-03-02 16:00 UTC**Tentative hypotheses**1. Short DB statement timeouts under load causing retries and eventual visibility expiry.2. Worker CPU/memory throttling on Fargate tasks leading to goroutine starvation/timeouts.3. SQS visibility timeout too short for worst-case processing.4. Non-idempotent processing causing partial failures on retry.**Suggested priority**P1 — affects customer-facing flows and leads to message loss/delay; mitigate quickly.**Recommended next steps**- Increase SQS visibility timeout to 2x typical processing time; monitor impact.- Temporarily increase DB statement_timeout and RDS instance size; enable slow query logging.- Add tracing/span IDs in logs to correlate message lifecycle.- Run load test (5k messages, 2 workers) to reproduce; capture CPU, memory, DB metrics.- Review idempotency of handlers; implement at-least-once safe processing if missing.