Webhooks and Event-Driven Integration Questions

Requirements (clarified)

• Functional: deliver webhooks to partner endpoints with subscription management, retries, DLQ, partner-facing dashboard.
• Non-functional: at-least-once, authenticated (signing), replay prevention, observable, scalable to large partner count.

High-level architecture
API Layer (Subscription Management) → Event Ingest/Router → Delivery Service (worker pool) → Retry Scheduler (backoff + jitter) → DLQ (persistent)
Supporting: Key Store (partner secrets), Dedup store, Delivery DB, Metrics/Tracing, Partner Dashboard (read-only views + replays)

Components & responsibilities

1. Subscription API: create/validate subscriptions (endpoint URL, secret, filters, retry policy).
2. Event Router: fan-out incoming events to subscribers (writes Delivery records).
3. Delivery Service: picks delivery task, constructs payload, signs with HMAC-SHA256(secret, payload|timestamp|nonce), sends HTTP(S) POST, expects 2xx. On success mark delivered; on failure record error and enqueue retry.
4. Retry Scheduler: stores next_attempt, backoff = base * 2^attempt * random(0.5–1.5). Uses delayed queue (e.g., Redis streams/DelayQueue or Kafka + scheduler).
5. Dedup / Replay prevention: include event_id, timestamp, nonce in signature; Delivery DB + dedup cache store delivered_event_ids per subscription with TTL; reject duplicates.
6. DLQ: after max_attempts, mark DLQ and surface in dashboard; allow manual requeue.
7. Secrets store: rotateable, access-controlled (KMS).
8. Metrics & Observability: Prometheus + Grafana: success rate, P50/P99 latency, retry counts, DLQ size, per-partner metrics. Traces via OpenTelemetry.
9. Dashboard: partner login, list deliveries, payload, headers, attempts, timestamps, retry schedule, replay button, download logs.

Data models (simplified)

• subscriptions(subscription_id, partner_id, endpoint_url, secret_id, filters(json), created_at, active, retry_policy)
• events(event_id, type, payload, created_at, source)
• deliveries(delivery_id, event_id, subscription_id, status{pending,delivered,failed,dlq}, attempts, next_attempt_at, last_error, signed_headers, created_at, updated_at)
• secrets(secret_id, partner_id, kms_key_id, rotated_at)
• dedup_cache(event_id, subscription_id) stored in Redis with TTL

Delivery flow

1. Event arrives -> router queries subscriptions -> write deliveries (status=pending).
2. Worker picks delivery, checks dedup cache -> sign payload -> POST -> on 2xx mark delivered, emit metric; else increment attempts, compute backoff+jitter, set next_attempt_at, re-enqueue or DLQ when attempts exhausted.

Trade-offs

• At-least-once vs exactly-once: choose at-least-once for simplicity; idempotency handled by partners via event_id.
• Delay queue implementation: Redis streams simpler; Kafka + scheduler more durable at scale.
• Per-partner throttling to avoid noisy neighbors.

Security notes

• Use HMAC with timestamp+nonce, allow ±allowed_skew, replay cache.
• Enforce TLS, validate certificate pinning optionally, rate-limit per subscription.

This design provides resiliency, security, observability and a partner UX to inspect and remediate failed deliveries.

Requirements (functional + non‑functional):

• Receive partner HTTP(S) POSTs reliably at scale, verify authenticity, prevent replay, ensure idempotent processing, surface failures for retries/alerting, and detect suspicious activity.

High-level architecture:
Partner → API Gateway (TLS, rate limiting, mTLS optional) → Webhook Receiver Service → Validation Layer → Processing Queue → Worker(s) → Persistent Store / Downstream systems → DLQ / Alerting / Monitoring

Authentication & verification

• Transport: enforce TLS 1.2+ and HSTS. Offer optional mTLS for high-trust partners.
• Message signature: require partner to sign payload with HMAC-SHA256 using a shared secret; include headers:
  • X-Partner-Id: partner identifier
  • X-Signature: v1=hex(hmac_sha256(secret, timestamp + "." + body)), v=version
  • X-Timestamp: unix seconds (or ISO8601)
  • X-Idempotency-Key: UUID (optional)
• Key management: support key versioning (kid) in header, rotate keys periodically, store secrets encrypted (KMS).

Signature verification steps

1. Validate partner exists and retrieve active secret by partner id and kid.
2. Ensure |now - X-Timestamp| <= allowed window (e.g., 300s).
3. Compute expected = HMAC(secret, timestamp + "." + body); constant-time compare to X-Signature.
4. If mismatch → 401 and log.

Replay protection & idempotency

• Timestamp window prevents old messages. Additionally store recent signature hashes or nonce in a fast store (Redis) with TTL = window.
• Idempotency: require or accept X-Idempotency-Key. Persist processed message keys (combination of partner id + idempotency-key or signature hash) in durable store (Redis for short TTL or RDBMS for long-lived dedupe) and return cached result if duplicate.
• If no idempotency key, use signature hash as dedupe key.

Failure handling & retries

• Receiver should validate and enqueue quickly (respond 2xx on accept). Use durable queue (Kafka/SQS) for processing.
• If downstream processing fails: retry with exponential backoff; after N attempts, send message to DLQ and generate alert.
• For synchronous validation failures (bad signature/timestamp), respond 4xx without retry.
• Provide partner-facing retry guidelines and a webhook replay endpoint for manual replays (authenticated).

Monitoring & suspicious activity detection

• Metrics: total requests, successful verifications, failed verifications, replay attempts, duplicate deliveries, processing failures, processing latency, queue depth.
• Logs: structured logs with partner id, request id, signature status (do not log raw secrets or full payloads).
• Alerts: thresholds for spikes in failed verifications, repeated bad signatures from same partner, sudden increase in traffic or replay attempts.
• Automated actions: throttle/rate‑limit or temporarily suspend partner keys on repeated failures; require rekeying or mTLS for high-risk partners.
• Audit trail: immutable storage of webhook receipts (hash of payload + metadata) for forensic analysis.

Operational considerations

• Scale: API Gateway autoscale + horizontally scaled receivers; Redis/KV for dedupe with consistent hashing; workers scale by queue partitions.
• Security: rotate keys, KMS encryption, least privilege for services, WAF for common threats, protect logs for PII.
• Documentation & onboarding: provide sample signing code (HMAC example), test endpoints, and key rotation process.
• Example HMAC snippet (pseudo): expectedSig = HMAC_SHA256(secret, timestamp + "." + body); compare const-time.

This design balances security, reliability and operability: fast validation at edge, durable asynchronous processing, strong replay/idempotency controls, and monitoring + automated mitigation for suspicious activity.

Start with trade-offs:

• Webhooks (push)

  • Pros: low-latency updates, minimal client CPU/network when idle, efficient at scale for active events.
  • Cons: operational complexity — must manage delivery retries, backoffs, security (signing, TLS), and scale outbound connections. Harder for clients behind NAT/firewalls or intermittent networks.

• Polling

  • Pros: very simple for clients, predictable server load patterns, easier for unreachable clients.
  • Cons: higher latency or wasted requests (if frequent), increased cost for many clients; inefficient when most clients are idle.

Recommendation for many low-traffic clients

• Use a hybrid, cost-aware pattern: primary webhooks with a graceful polling fallback.
  • Register client endpoints and attempt webhook delivery with exponential backoff and queued retries; batch or coalesce events per client to reduce calls.
  • Require a lightweight heartbeat/lease: if webhook failures exceed threshold (client offline), mark endpoint “unreachable” and switch the client to infrequent polling (e.g., every 5–30 minutes) or let client perform on-demand poll when it wakes.
  • Provide a “sync” endpoint and a Last-Modified/ETag cursor so polls are cheap (only return deltas).
  • Monitor delivery metrics and expose dashboards/alerts; use a dead-letter queue or notification to admins when repeated failures occur.

Why this balances cost and timeliness

• Active/available clients get near-real-time updates via webhooks (low latency, low cost). Inactive/unreachable clients avoid costly retry storms by switching to sparse polling, keeping operational cost down while preserving eventual consistency.