Webhooks and Event-Driven Integration Questions

Endpoint design (summary):

• Single POST /webhook/{provider} endpoint with provider-specific secret lookup.
• Use HTTPS, require Content-Type: application/json, and introspect headers: X-Signature, X-Timestamp, X-Nonce (if provided).
• Return 2xx for success, 4xx for client errors (invalid signature), 5xx for transient processing errors so sender retries per their policy. Log and alert on repeated failures.

Approach:

1. Validate HMAC signature.
2. Defend against replay with timestamp window and nonce dedupe.
3. Process event idempotently using event_id storage.
4. Respond quickly; process heavier work async.
5. Surface persistent failures via retry counters, DLQ, and alerts.

Python pseudocode:

import hmac, hashlib, time, json
from typing import Dict

SECRET_STORE = {"provider": b"supersecret"}
NONCE_TTL = 300  # seconds
TIMESTAMP_SKEW = 300
# Simple stores for examples; use Redis/DB in prod
SEEN_NONCES = {}        # nonce -> expiry_ts
PROCESSED_EVENTS = {}   # event_id -> result/status
RETRY_COUNTER = {}      # event_id -> attempts

def now(): return int(time.time())

def validate_hmac(body: bytes, signature: str, secret: bytes) -> bool:
    mac = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(mac, signature)

def is_replay(nonce: str, ts: int) -> bool:
    if abs(now() - ts) > TIMESTAMP_SKEW:
        return True
    expiry = SEEN_NONCES.get(nonce)
    if expiry and expiry > now():
        return True
    SEEN_NONCES[nonce] = now() + NONCE_TTL
    return False

def idempotent_process(event: Dict) -> bool:
    eid = event.get("id")
    if eid in PROCESSED_EVENTS:
        return True  # already processed
    # business processing (should be transactional)
    success = enqueue_work(event)  # quick enqueue for async processing
    if success:
        PROCESSED_EVENTS[eid] = "accepted"
    return success

def handle_webhook(request):
    provider = request.path_params.get("provider")
    secret = SECRET_STORE.get(provider)
    if not secret:
        return Response(404)

    body = request.body  # bytes
    sig = request.headers.get("X-Signature", "")
    ts = int(request.headers.get("X-Timestamp", "0"))
    nonce = request.headers.get("X-Nonce", "")

    if not validate_hmac(body, sig, secret):
        return Response(400, "invalid signature")

    if is_replay(nonce, ts):
        return Response(400, "replay or timestamp skew")

    event = json.loads(body.decode())
    eid = event.get("id")
    if not eid:
        return Response(400, "missing id")

    if not idempotent_process(event):
        # transient error: ask sender to retry by returning 5xx
        RETRY_COUNTER[eid] = RETRY_COUNTER.get(eid, 0) + 1
        if RETRY_COUNTER[eid] > 5:
            move_to_dlq(event)           # persistent failure surface
            alert_on_failure(event)
            return Response(200, "accepted but moved to DLQ")
        return Response(503, "temporary failure")
    return Response(200, "ok")

Key notes:

• Use secure compare for HMAC to avoid timing attacks.
• Persist nonces and processed event IDs in Redis/DB with TTL to survive restarts.
• Keep processing fast—hand off to worker for retries and exponential backoff.
• Surface persistent failures via DLQ, metrics, and pager/alerting.

Requirements:

• At-least-once delivery from partners → must dedupe and be idempotent.
• Low latency for happy-path processing.
• Scale to many partners and high QPS.
• Handle duplicates and out-of-order events (where semantics matter).

Design summary:

• Front door: HTTP endpoint (stateless) → validate, extract partner-id, event-id, sequence/ts, payload.
• Fast dedupe check in a distributed in-memory store (Redis cluster with strong TTL support).
• Push events that pass dedupe into an internal processing queue (Kafka or SQS) for application processing.
• Ack webhook ASAP (200) when accepted (not when processed) to minimize partner latency.
• Retry/backoff: client-side (if we call external) use exponential backoff with full jitter; for partners, rely on their retries—we keep idempotency guarantees.

Deduplication strategy:

• Use an idempotency key: partner-id + event-id if provided. If no event-id, use hash(partner-id + payload).
• Store a dedupe record in Redis as key -> metadata {status, seq, first_seen_ts} with TTL.
• TTL = max(event retention window, partner retry window) + safety buffer (e.g., 24-72 hours depending on business).
• If duplicate received and status == processed, immediately drop/ack. If in-flight, optionally wait/fast-path check.

Pseudocode (Python-like):

import time, hashlib
def idempotency_key(partner_id, event):
    if 'event_id' in event:
        return f"{partner_id}:id:{event['event_id']}"
    return f"{partner_id}:hash:{hashlib.sha256(serialize(event)).hexdigest()}"

def handle_webhook(partner_id, event):
    key = idempotency_key(partner_id, event)
    now = int(time.time())
    # Try set NX with small value to mark reception; store seq if present
    inserted = redis.set(key, json.dumps({'status':'queued','seq':event.get('seq'), 'ts':now}), nx=True, ex=DEDUP_TTL)
    if not inserted:
        meta = json.loads(redis.get(key))
        if meta['status']=='processed':
            return 200  # duplicate, already done
        # if seq present and incoming.seq <= meta.seq: duplicate/out-of-order -> ack/drop
        if event.get('seq') and meta.get('seq') and event['seq'] <= meta['seq']:
            return 200
        # else allow (maybe out-of-order later)
    enqueue_to_kafka(topic=partner_id, payload=event, meta_key=key)
    return 200

def worker_consume(msg):
    key = msg.meta_key
    # process idempotently: re-check Redis to ensure not double-processing
    meta = json.loads(redis.get(key))
    if meta['status']=='processed':
        return
    success = process_event(msg.payload)
    if success:
        redis.set(key, json.dumps({'status':'processed','seq':msg.payload.get('seq'),'ts':int(time.time())}), ex=DEDUP_TTL)

Out-of-order handling:

• Prefer that partners send sequence numbers or timestamps. Keep per-partner watermark/latest-seq in Redis.
• If an event arrives with seq > watermark+1, buffer in a small in-memory store (or Kafka partitioned by partner) for short TTL to wait for missing earlier events.
• If missing events don't arrive within buffering TTL, process higher-seq events and mark a gap (business decision).
• Partition Kafka by partner-id to keep ordering per partner and have consumers handle ordered processing.

Retry/backoff (when calling partners or downstream systems):

• Exponential backoff with full jitter:
  base = 100ms, max = 30s, attempt i → sleep = random(0, min(max, base * 2^i))
• Use circuit breaker for downstream failures; move to DLQ after N attempts.
• Prefer async retries via scheduled retry queue (Redis/ZSet or Kafka with delay) rather than blocking worker.

Scaling & operational notes:

• Use Redis cluster (shard by partner or key hash) and set dedupe TTL sensible to business.
• Partition Kafka by partner-id for ordering and parallelism.
• Monitor dedupe hit-rate, queue lag, buffer drop rates.
• Consider compacted persistent store (Cassandra) for long-term dedupe audit if required.

Trade-offs:

• Short TTL reduces memory but may allow late duplicates to be reprocessed.
• Strong ordering buffering increases latency; choose per-partner policy (strict vs best-effort).
• Redis provides low-latency dedupe; for extreme scale use a bloom filter + backing store to reduce writes (probabilistic).

Requirements:

• Functional: accept events, deliver to subscriber HTTP endpoints with at-least-once semantics, support batching, per-subscriber rate limits, secret-based signing.
• Non-functional: high throughput (millions/day), low latency, multi-region resilience, observability (metrics, logs, tracing), admin + subscriber retry dashboard.

High-level architecture:
API Ingest -> Event Store (Kafka) -> Dispatcher service(s) -> Rate limiter -> Batcher -> Delivery workers -> DLQ store -> Retry controller -> Secret Manager
Monitoring + Dashboard + Alerting

Components:

1. Ingest API: validates, authenticates producers, attaches event metadata, writes to partitioned Kafka topic keyed by subscriber id for ordering guarantees.
2. Dispatcher: reads events per subscriber partition, enforces per-subscriber rate limits (token-bucket via Redis or local leaky-bucket with centralized tokens), groups events into batches (configurable size/time window).
3. Delivery Worker: sends HTTP requests with HMAC signature from Secret Manager (secrets stored in KMS + cached in secure vault). Uses async HTTP client with connection pooling, follows exponential backoff with full jitter (sleep = random_between(0, base * 2^attempt)), caps max backoff, and circuit-breaker per endpoint to avoid cascading failures.
4. Retry Controller & DLQ: delivery failures (network 5xx, connection errors, or non-retriable codes depending on subscriber config) get retried until max attempts or TTL. After exhaustion, events go to per-subscriber DLQ (durable DB or Kafka topic). Subscribers can requeue from DLQ via dashboard or API.
5. Idempotency & At-least-once: include event-id and attempt-id; subscribers should handle dedup. To reduce duplicates, maintain short-lived delivery cache (Redis) to avoid immediate duplicate deliveries across retries.
6. Batching: batch size/time trade-off; use adaptive batching based on endpoint latency to maximize throughput while respecting rate limits.
7. Secret Management: store signing keys in KMS (AWS KMS/GCP KMS) and rotate keys; delivery service fetches keys with least privilege, caches and auto-refreshes.
8. Monitoring & Dashboard: Prometheus metrics (success rate, latencies, retries, DLQ counts), distributed tracing (OpenTelemetry), logs (structured). Subscriber-facing dashboard shows delivery history, retry attempts, failure reasons, allow manual retry/replay, update rate-limit or webhook config.
9. Scalability & Resilience: Dispatcher & workers stateless and autoscaled. Kafka partitions scale with subscribers. Use multiple regions with active-active or active-passive failover and geo-routing.
10. Trade-offs: At-least-once simplifies guarantees but pushes dedup responsibility to subscribers. Strong ordering per-subscriber requires single-partition consumption which limits parallelism per subscriber — mitigate by per-topic-per-subscription sharding for high-volume subscribers.

Edge cases:

• Slow endpoints: circuit-breaker + backpressure to avoid queue buildup; move high-volume endpoints to dedicated partitions.
• Bursts: token-bucket smoothing and DLQ overflow throttling.
• Secret compromise: immediate key rotation and revoke, mark subscriber endpoints disabled.

This design provides resilient at-least-once delivery, robust retries with jitter, manageable DLQ workflows, throughput via batching, per-subscriber rate control, secure secret handling, and full observability plus a subscriber retry dashboard.

Webhook fundamentals: a webhook is an HTTP callback—your service makes an HTTP POST to a receiver URL when an event occurs. It's push-based, low-latency, and simple to integrate.

Key design concerns:

• Retries & delivery guarantees: assume networks fail; use at-least-once delivery by retrying, but design for duplicates. Persist events and retry until success or expiry.
• Idempotency: include an event ID and type so receivers can dedupe. Receivers should store processed IDs or use idempotency keys.
• Authentication/signature verification: sign payloads (HMAC SHA256) and include timestamp to prevent replay attacks. Receiver verifies signature and timestamp window.
• Delivery semantics: most webhooks provide at-least-once; exactly-once is impractical without coordination.
• Polite retry/backoff: exponential backoff with jitter, capped retries and total time window. Example: retries at 1s, 2s, 4s, 8s,... up to 1 hour, add ±20% jitter, stop after N attempts or 24h, and notify owner on permanent failure.

Sample payload:

{
  "id": "evt_01F...A",
  "type": "order.created",
  "timestamp": 1699999999,
  "data": {
    "order_id": "ord_123",
    "amount": 1999,
    "currency": "USD"
  }
}

How a receiver should validate & acknowledge:

• Validate HTTP: accept only POST, require TLS (https).
• Verify signature: compute HMAC(secret, timestamp + "." + body) and compare to header (e.g., X-Signature). Reject if timestamp outside allowed window (e.g., ±5 minutes).
• Parse and validate schema: check required fields and types.
• Idempotency check: look up event.id; if seen, return 200 OK without reprocessing.
• Acknowledge: on successful validation & processing return 200 OK (or 2xx). For temporary failures return 5xx so sender retries; for invalid/auth failures return 4xx to stop retries.
• Monitoring: log failures, expose dead-letter or webhook dashboard for manual retry.

Approach: Use HMAC-SHA256 over canonicalized payload + timestamp + nonce; include timestamp and nonce in headers. Server verifies signature, timestamp freshness, and nonce uniqueness (cached short-term). Support key rotation by accepting multiple active secrets with key IDs (kid) in header. For high-security partners, use RSA/ECDSA signatures with the partner’s public key and certificate rotation/verification.

Signing (sender):

import hmac, hashlib, time, uuid
def sign(payload, secret, kid):
    ts = str(int(time.time()))
    nonce = uuid.uuid4().hex
    canonical = ts + "." + nonce + "." + payload  # deterministic
    sig = hmac.new(secret.encode(),''.join([canonical]).encode(),hashlib.sha256).hexdigest()
    return {"X-Signature": sig, "X-Timestamp": ts, "X-Nonce": nonce, "X-Kid": kid}

Verification (receiver pseudocode):

def verify(headers, payload, key_store, nonce_cache, max_age=300):
    ts = int(headers['X-Timestamp']); nonce = headers['X-Nonce']; sig = headers['X-Signature']; kid = headers['X-Kid']
    if abs(now()-ts) > max_age: return False  # replay/time window
    if nonce_cache.exists(nonce): return False  # replay
    secret = key_store.get_secret(kid)  # try current and recent previous keys
    for s in secret_candidates(secret):
        expected = hmac_sha256_hex(f"{ts}.{nonce}.{payload}", s)
        if hmac.compare_digest(expected, sig):
            nonce_cache.store(nonce, ttl=max_age)  # record nonce
            return True
    return False

Key rotation:

• Use key IDs. Maintain current key + one or more previous keys for overlap.
• Rotate by publishing new kid and secret; keep old secret for grace period equal to max_age + deployment lag.
• Revoke by removing from key_store and optionally adding to denylist.

Nonce storage/scaling:

• Use distributed cache (Redis) with TTL = max_age.
• For high throughput, store nonce hash and shard cache.

High-security partners (public-key):

• Sender signs canonical string with private key (RSASSA-PSS or ECDSA).
• Header includes X-Signature, X-Timestamp, X-Nonce, X-Kid referencing public key.
• Receiver verifies signature with stored/trusted public key (use certs, OCSP or JWKs endpoint for rotation).
• Validate certificate chain, key id, and signature. Optionally require mutual TLS.

Additional best practices:

• Canonicalize payload deterministically (sorted JSON) to prevent minor formatting differences.
• Use constant-time compare for signatures.
• Log failed attempts and monitor anomaly rates.
• Enforce rate limits and IP allowlists for sensitive endpoints.