Webhooks and Event-Driven Integration Questions

Webhook fundamentals: a webhook is an HTTP callback—your service makes an HTTP POST to a receiver URL when an event occurs. It's push-based, low-latency, and simple to integrate.

Key design concerns:

• Retries & delivery guarantees: assume networks fail; use at-least-once delivery by retrying, but design for duplicates. Persist events and retry until success or expiry.
• Idempotency: include an event ID and type so receivers can dedupe. Receivers should store processed IDs or use idempotency keys.
• Authentication/signature verification: sign payloads (HMAC SHA256) and include timestamp to prevent replay attacks. Receiver verifies signature and timestamp window.
• Delivery semantics: most webhooks provide at-least-once; exactly-once is impractical without coordination.
• Polite retry/backoff: exponential backoff with jitter, capped retries and total time window. Example: retries at 1s, 2s, 4s, 8s,... up to 1 hour, add ±20% jitter, stop after N attempts or 24h, and notify owner on permanent failure.

Sample payload:

{
  "id": "evt_01F...A",
  "type": "order.created",
  "timestamp": 1699999999,
  "data": {
    "order_id": "ord_123",
    "amount": 1999,
    "currency": "USD"
  }
}

How a receiver should validate & acknowledge:

• Validate HTTP: accept only POST, require TLS (https).
• Verify signature: compute HMAC(secret, timestamp + "." + body) and compare to header (e.g., X-Signature). Reject if timestamp outside allowed window (e.g., ±5 minutes).
• Parse and validate schema: check required fields and types.
• Idempotency check: look up event.id; if seen, return 200 OK without reprocessing.
• Acknowledge: on successful validation & processing return 200 OK (or 2xx). For temporary failures return 5xx so sender retries; for invalid/auth failures return 4xx to stop retries.
• Monitoring: log failures, expose dead-letter or webhook dashboard for manual retry.

Approach: compute HMAC SHA256 of the raw request body + timestamp (HubSpot uses header "x-hubspot-signature" and "x-hubspot-signature-timestamp" or similar); compare using constant-time compare; reject if timestamp outside allowed window to prevent replay; support secret rotation by checking multiple current/previous secrets; log invalid attempts with reason and metadata.

// Node.js pseudocode (express-like)
const crypto = require('crypto');
const ALLOWED_DRIFT_MS = 5 * 60 * 1000; // 5 minutes

// secrets: array with primary first, then previous (for rotation)
const HUBSPOT_SECRETS = [process.env.HUBSPOT_SECRET, process.env.HUBSPOT_OLD_SECRET];

function constantTimeEquals(a, b) {
  // both inputs are Buffers
  if (a.length !== b.length) return false;
  // crypto.timingSafeEqual uses constant-time comparison
  return crypto.timingSafeEqual(a, b);
}

function computeSignature(secret, timestamp, rawBody) {
  // HubSpot signs: HMAC_SHA256(secret, timestamp + rawBody) or sometimes body only.
  // Adjust concatenation to match vendor docs. Here we assume timestamp + body.
  const h = crypto.createHmac('sha256', secret);
  h.update(String(timestamp));
  h.update(rawBody);
  return h.digest('hex'); // or base64 depending on header format
}

// Express handler
app.post('/webhook', express.raw({ type: '*/*' }), (req, res) => {
  const rawBody = req.body; // Buffer because of express.raw
  const sigHeader = req.get('x-hubspot-signature'); // hex/base64 string
  const tsHeader = req.get('x-hubspot-signature-timestamp'); // seconds or ms
  const now = Date.now();

  if (!sigHeader || !tsHeader) {
    console.warn('Missing signature or timestamp', { ip: req.ip });
    return res.status(400).send('Invalid request');
  }

  const timestamp = Number(tsHeader);
  if (Number.isNaN(timestamp)) {
    console.warn('Invalid timestamp', { tsHeader, ip: req.ip });
    return res.status(400).send('Invalid timestamp');
  }

  // Check replay window
  const tsMs = (String(timestamp).length <= 10) ? timestamp * 1000 : timestamp;
  if (Math.abs(now - tsMs) > ALLOWED_DRIFT_MS) {
    console.warn('Replay/expired signature', { ip: req.ip, timestamp });
    return res.status(401).send('Signature expired');
  }

  // Try all secrets (supports rotation)
  let valid = false;
  const sigBuffer = Buffer.from(sigHeader, 'hex'); // adjust if base64
  for (const secret of HUBSPOT_SECRETS.filter(Boolean)) {
    const expectedHex = computeSignature(secret, timestamp, rawBody);
    const expectedBuffer = Buffer.from(expectedHex, 'hex');
    if (constantTimeEquals(expectedBuffer, sigBuffer)) {
      valid = true;
      break;
    }
  }

  if (!valid) {
    console.warn('Invalid signature', { ip: req.ip, timestamp, sigHeader });
    return res.status(401).send('Invalid signature');
  }

  // Process webhook
  // ... business logic ...
  return res.status(200).send('OK');
});

Key points:

• Use raw body (Buffer) for exact bytes; do not parse JSON before verifying.
• Use crypto.timingSafeEqual (or equivalent) for constant-time comparison to avoid timing attacks.
• Secret rotation: keep current + recent previous secrets and try each; remove old secrets after safe period.
• Replay protection: enforce timestamp window; optionally store recent signature IDs/nonces if provider sends them to block reuse.
• Log invalid attempts with minimal sensitive data (no secrets or full payload) for incident investigation.

Problem: unreliable third-party webhooks; mobile depends on events.
Server-side patterns:

• Ingestion: accept webhooks via dedicated endpoint that immediately responds 200/202 after basic auth/verification. Persist raw payload with received_at and provider_id.
• Deduplication: require providers to send an idempotency key or signature (e.g., X-Event-Id). Maintain short-lived dedupe store (Redis or DB unique constraint) to ignore duplicates. If no id provided, hash payload and dedupe by hash+timestamp.
• Ordering: timestamp from provider preferred; store sequence numbers when available. Use per-entity event streams with sequence numbers and reconcile out-of-order events by buffering small windows (e.g., few seconds) or applying idempotent handlers.
• Idempotency: design handlers to be idempotent. Use idempotency keys for downstream side effects and store processed_event_ids; replays check this store.
• Processing pipeline: enqueue validated events to durable queue (Kafka/SQS) and have idempotent workers apply business logic and update authoritative state; workers acknowledge after committed state change.
• Exposing API to mobile: provide a stable aggregated endpoint for mobile that returns current derived state and event history: GET /objects/{id}/events?since=seq. Also expose webhook delivery status: GET /webhook-events/{id} returns status (received, processed, failed) and sequence.
• Reconciliation: provide reconciliation API for clients to request full state and lastAppliedEventId so mobile can heal missed or reordered events.
  Why: durable ingestion + dedupe + idempotent handlers insulate mobile from third-party unreliability and offer traceability for debugging.

Requirements (clarify):

• Deliver high-value events reliably with minimal duplicates
• Tolerate long receiver downtime (hours–days)
• Require receiver-side idempotency but also minimize duplicate deliveries
• Support secure automated retries, manual replay, dead-lettering, and dashboard visibility
• Prevent replay/fraud (signing + replay tokens)

High-level architecture:

• Event Producer → Reliable Delivery Service (RDS) → Delivery Queue per Receiver → Outbound Worker Pool → Receiver
• Persistent store: DeliveryRecords (id, event_id, receiver_id, status, attempts, next_attempt_at, backoff_factor, signature_meta, replay_token, last_response)
• Dead Letter Store for permanently failed deliveries

Delivery flow & deduplication:

• Each event has a globally unique event_id and per-receiver delivery_id. RDS writes DeliveryRecord once and enqueues.
• Outbound worker includes DeliveryRecord.id in headers and signs payload with rotating private key + timestamp.
• Receiver must implement idempotency keyed on event_id or idempotency-key header. RDS limits duplicates by tracking successful ack (2xx) and marking delivered atomically.

Backoff & retries:

• Use exponential backoff with jitter: next = now + base * (2^attempts) * (1 + random[-p,p])
• Cap max interval (e.g., 24h) and total retry window (e.g., 14 days).
• Use adaptive backoff: treat transient (5xx, timeouts) vs permanent (4xx except 429) differently. For 429/RateLimit, honor Retry-After header.

Dead-lettering:

• After max attempts or expiry, mark as DEAD_LETTER and move payload to encrypted dead-letter store with metadata. Allow manual inspection and replay. Include reason codes and last_response.

Signing & replay protection:

• Sign payload with HMAC + key-id header and include timestamp and nonce. Receiver verifies signature and timestamp window.
• Generate a replay_token per DeliveryRecord (random, stored hashed server-side). For manual replays, provide one-time replay_token or time-limited replay JWT that authorizes replay and is auditable.
• Maintain key rotation: sign with current key, accept previous keys for short overlap.

Manual retry & secure replay tool:

• Admin UI/API to: 1) view DeliveryRecord, 2) trigger immediate retry (increments attempt with audit), 3) generate time-limited replay token to resend payload without changing original record (or create a new delivery record linked to original event_id).
• All manual actions logged (who, when, why).

Dashboard & developer visibility:

• Per-receiver dashboard showing:
  • Delivery success rate, avg latency, recent failures, histogram of status codes
  • Live queue length, next retry times, dead-letter count
  • Per-event view: timeline (created → attempts timestamps → responses), replay history, replay_token audit
• Provide webhook delivery webhooks (i.e., callbacks) or webhook logs API for integration, with downloadable raw payload and signed headers.

Operational notes & trade-offs:

• Require receiver idempotency but reduce duplication by honoring 2xx only and pausing retries if receiver signals "duplicate (200+body)" — better to have explicit idempotency header acknowledgement.
• Storing payloads encrypted increases storage cost but required for reliable long-term retries.
• Security: use mTLS optional + HMAC signatures to prevent tampering; limit replay token lifetime to reduce risk.
• Scale: shard DeliveryQueue by receiver_id, use rate-limiting per receiver, and backpressure to protect worker pool.

This design balances guaranteed-at-least-once delivery with tooling and cryptographic controls to minimize duplicate processing, support long outages, and give developers clear observability and secure manual recovery.

Requirements:

• Functional: accept events, deliver to subscriber HTTP endpoints with at-least-once semantics, support batching, per-subscriber rate limits, secret-based signing.
• Non-functional: high throughput (millions/day), low latency, multi-region resilience, observability (metrics, logs, tracing), admin + subscriber retry dashboard.

High-level architecture:
API Ingest -> Event Store (Kafka) -> Dispatcher service(s) -> Rate limiter -> Batcher -> Delivery workers -> DLQ store -> Retry controller -> Secret Manager
Monitoring + Dashboard + Alerting

Components:

1. Ingest API: validates, authenticates producers, attaches event metadata, writes to partitioned Kafka topic keyed by subscriber id for ordering guarantees.
2. Dispatcher: reads events per subscriber partition, enforces per-subscriber rate limits (token-bucket via Redis or local leaky-bucket with centralized tokens), groups events into batches (configurable size/time window).
3. Delivery Worker: sends HTTP requests with HMAC signature from Secret Manager (secrets stored in KMS + cached in secure vault). Uses async HTTP client with connection pooling, follows exponential backoff with full jitter (sleep = random_between(0, base * 2^attempt)), caps max backoff, and circuit-breaker per endpoint to avoid cascading failures.
4. Retry Controller & DLQ: delivery failures (network 5xx, connection errors, or non-retriable codes depending on subscriber config) get retried until max attempts or TTL. After exhaustion, events go to per-subscriber DLQ (durable DB or Kafka topic). Subscribers can requeue from DLQ via dashboard or API.
5. Idempotency & At-least-once: include event-id and attempt-id; subscribers should handle dedup. To reduce duplicates, maintain short-lived delivery cache (Redis) to avoid immediate duplicate deliveries across retries.
6. Batching: batch size/time trade-off; use adaptive batching based on endpoint latency to maximize throughput while respecting rate limits.
7. Secret Management: store signing keys in KMS (AWS KMS/GCP KMS) and rotate keys; delivery service fetches keys with least privilege, caches and auto-refreshes.
8. Monitoring & Dashboard: Prometheus metrics (success rate, latencies, retries, DLQ counts), distributed tracing (OpenTelemetry), logs (structured). Subscriber-facing dashboard shows delivery history, retry attempts, failure reasons, allow manual retry/replay, update rate-limit or webhook config.
9. Scalability & Resilience: Dispatcher & workers stateless and autoscaled. Kafka partitions scale with subscribers. Use multiple regions with active-active or active-passive failover and geo-routing.
10. Trade-offs: At-least-once simplifies guarantees but pushes dedup responsibility to subscribers. Strong ordering per-subscriber requires single-partition consumption which limits parallelism per subscriber — mitigate by per-topic-per-subscription sharding for high-volume subscribers.

Edge cases:

• Slow endpoints: circuit-breaker + backpressure to avoid queue buildup; move high-volume endpoints to dedicated partitions.
• Bursts: token-bucket smoothing and DLQ overflow throttling.
• Secret compromise: immediate key rotation and revoke, mark subscriber endpoints disabled.

This design provides resilient at-least-once delivery, robust retries with jitter, manageable DLQ workflows, throughput via batching, per-subscriber rate control, secure secret handling, and full observability plus a subscriber retry dashboard.