Third-Party and SaaS Integration Questions

Recommendation
For unsubscribes, I would centralize the consent workflow and let other systems react to the resulting event. This is a case where correctness matters more than loose independence, because one missed suppression can create a compliance problem.

How I would decide

• Centralize if there is one business rule for consent, one audit trail, and a hard deadline before the next send.
• Use choreography only when systems are loosely coupled and temporary delay is acceptable.

Failure modes I would watch

• A service misses the unsubscribe event
• A retry causes duplicate updates
• A campaign send races ahead of the suppression update
• One system updates from stale cache while another is current

Example
If a user opts out at 9:10 and a campaign is scheduled for 9:15, the central consent service should record the change immediately, publish an event, and block sends until all critical destinations confirm receipt or until the campaign filter reads from the central source of truth.

Control plane
I would add retries, dead-letter handling, a reconciliation job, and a pre-send suppression check so the system stays safe even if one downstream service is slow.

Approach
I would make the consumer idempotent. Idempotent means the same event can be processed twice without creating two results. The core pattern is: persist the raw webhook first, dedupe by a stable event key, then process asynchronously.

Design

• Require an event_id from the sender, or derive a fingerprint from source system, object id, and version.
• Store each event in an inbox table with a unique constraint on that key.
• Ack the webhook only after the raw event is durably stored.
• Use upserts in downstream tools keyed by the business entity, such as lead_email or external_lead_id.
• Track event version or updated_at so older events do not overwrite newer state.

Example
If lead 123 arrives twice at 10:01 with the same event_id, the first insert succeeds and the second is ignored by the unique constraint. If a newer update at 10:03 changes the phone number, the processor applies that one because its version is later.

Failure handling
I would add retries with backoff, a dead-letter queue for poison messages, and periodic reconciliation so partial failures do not silently drift.

Signals to check
I would look at three layers: producer, pipeline, and vendor. Producer signals include send rate, retry rate, and publish latency. Pipeline signals include queue depth, age of oldest message, consumer lag, and dead-letter growth. Vendor signals include 429 responses, timeout rate, and response latency.

How I would interpret it
If the queue grows while the producer rate is flat and vendor 429s rise, the bottleneck is likely the vendor. If queue depth rises but the vendor looks healthy, the consumer or transformation layer may be slow. If the producer rate spikes unexpectedly, the issue may be upstream demand.

Controls

• Apply backpressure, so producers slow down when the queue age crosses a threshold
• Use rate limits and bounded concurrency per destination
• Prioritize fresh or user-facing updates over low-value backlog
• Pause or batch noncritical sends when the vendor is throttling
• Use retries with jitter and a dead-letter queue for repeated failures

Example
If oldest-message age jumps from 2 minutes to 18 minutes and the vendor starts returning many 429s, I would cap concurrency, lower request rate, and alert on freshness rather than just queue size, because old messages are often more harmful than a larger queue.

Trace the issue
I would trace the record at each hop using a correlation ID, which is a unique identifier carried through logs and events. Then I would compare counts at every boundary: source emitted, ingestion received, transform accepted, warehouse loaded, report queried.

What I would inspect

• Structured logs for dropped records or filter rules
• Traces for latency spikes or retries
• Checkpoints or offsets in the queue or stream
• Destination row counts and partition counts
• Quarantine tables for records rejected by validation

Example
If the source sent 1,000 records, ingestion received 1,000, and the warehouse loaded 998, the problem is likely in transformation or load. If the warehouse has 1,000 but the report shows 998, the issue is probably the report filter, not the pipeline.

Automated validation
I would add contract tests for payload shape, replay tests against saved real events, and synthetic canary records that must appear end to end. I would also alert on count drift, not just on API errors, because a healthy API can still lose business data silently.

Design
I would insert an anti-corruption layer between the vendor and the rest of the stack. That means the integration layer translates vendor payloads into a stable internal contract, and the downstream systems only see the internal shape.

Components

• Vendor adapter for each API or webhook version
• Canonical schema with versioning
• Transformer service that maps vendor fields to internal fields
• Contract tests that verify each adapter against sample payloads
• Replay pipeline so old events can be reprocessed through the new adapter

Example
If the vendor changes email_address to primary_email, only the adapter changes. Downstream systems still receive email. If the vendor adds a new field, I can ignore it until a consumer needs it, which avoids a cascade of changes.

How this prevents repeat pain
I would document field ownership, use schema validation at the edge, and run a parallel test environment before cutover. Then when the next vendor change arrives, the blast radius stays inside one adapter instead of spreading across every consumer.