Third-Party and SaaS Integration Questions

Goal
Deliver near-real-time for high-priority leads while obeying a strict 100 req/min vendor limit per account.

Core design

• Prioritized ingest queue (P0 high-priority, P1 normal)
• Per-vendor token-bucket rate limiter that enforces 100 RPM
• Fast-path for P0: maintain reserved low-latency slots (e.g., 20% of capacity) and use batching of 1 where possible
• Batching for P1: aggregate multiple leads into multi-create endpoints if vendor supports it

Queuing & scheduling

• Use prioritized persistent queues (Redis streams or Kafka topics with priority partitioning).
• Scheduler polls queues and fills token-bucket; if capacity remains, send from P0 first. If P0 exceeds reserved capacity, fall back to best-effort with client notification.

Caching & idempotency

• Cache recent lead state and dedupe to avoid unnecessary outbound calls.
• Use idempotency keys on vendor API to safely retry and allow batching without duplicates.

Fallbacks

• If vendor quota exhausted, options:
  • Defer low-priority leads to next window with aggregated batch
  • Use alternate vendor endpoints/accounts if available (multi-account sharding)
  • Persist as pending in DB and surface estimated delivery time to client

Surface delayed updates

• Provide real-time delivery status API/webhook showing state (queued, sent, delayed) and ETA.
• Expose SLA tiers: premium customers get higher reserved slots and lower queuing delay.

Operational & monitoring

• Monitor vendor responses, retry budgets, and queue latency; auto-scale workers.
• Telemetry: per-account request rate, latency, queue depth, and percent of high-priority served within X seconds.

Why this works

• Prioritization + reserved capacity ensures low-latency for critical leads; batching and caching maximize throughput within the hard rate limits; transparency to clients reduces surprises.

Architecture overview
Centralized auth service issuing short-lived tokens, fine-grained scopes, secrets manager for long-lived credentials, centralized audit logging, and a breach-recovery playbook.

Authentication & tokens

• Use OAuth2.0 with client credentials for connectors; issue short-lived access tokens (1–15 minutes) and refresh tokens rotated frequently.
• Mutual TLS for high-security connectors where supported.
• Token rotation: automated rotation of refresh/connector tokens through jobs; revoke previous tokens on rotation.

Authorization & least privilege

• RBAC/ABAC with least-privilege scopes (read:contacts, write:campaigns). Scopes tied to connector registration; connectors request minimal scopes.
• Use policy engine (OPA) to evaluate runtime permissions.

Secrets management

• Central secrets vault (HashiCorp Vault or cloud KMS) with dynamic credentials for downstream systems and audit-enabled access; secrets encrypted with KMS keys per environment/tenant.
• Secrets access logged and rotated automatically; ephemeral dynamic DB creds where possible.

Audit logging & monitoring

• Central immutable audit stream (WORM storage) capturing auth events, token issuance, secret access, connector actions.
• SIEM integration, alerts for abnormal patterns (mass token requests, unusual IPs).

Minimize blast radius

• Per-connector least-privilege principals and tenant-scoped service accounts.
• Network segmentation: restrict outbound egress per connector; use egress allow-lists.
• Rate limits and per-connector quotas; ephemeral session keys with limited privileges.
• If compromise detected: revoke token, rotate secrets, revoke dynamic creds, isolate tenant traffic, and replay recent audit logs to scope damage.

Breach recovery plan

• Containment (revoke keys, quarantine connector), forensic capture (audit logs), rotate affected secrets, notify stakeholders, remediate and harden, and run post-incident review.

Operational controls

• Periodic pen tests, automated secret scanning, and connector onboarding checklist requiring least-privilege justification and periodic attestation.

Signals to check
I would look at three layers: producer, pipeline, and vendor. Producer signals include send rate, retry rate, and publish latency. Pipeline signals include queue depth, age of oldest message, consumer lag, and dead-letter growth. Vendor signals include 429 responses, timeout rate, and response latency.

How I would interpret it
If the queue grows while the producer rate is flat and vendor 429s rise, the bottleneck is likely the vendor. If queue depth rises but the vendor looks healthy, the consumer or transformation layer may be slow. If the producer rate spikes unexpectedly, the issue may be upstream demand.

Controls

• Apply backpressure, so producers slow down when the queue age crosses a threshold
• Use rate limits and bounded concurrency per destination
• Prioritize fresh or user-facing updates over low-value backlog
• Pause or batch noncritical sends when the vendor is throttling
• Use retries with jitter and a dead-letter queue for repeated failures

Example
If oldest-message age jumps from 2 minutes to 18 minutes and the vendor starts returning many 429s, I would cap concurrency, lower request rate, and alert on freshness rather than just queue size, because old messages are often more harmful than a larger queue.

Decision factors
I would compare three options against four needs: speed to launch, complexity, maintainability, and latency. A native connector is simplest when the CRM and marketing tool already support the fields and schedule you need. An integration platform is good when you want faster setup, monitoring, and moderate customization. A custom API integration fits when rules are unique, scale is high, or data freshness matters.

Example
If the team only needs a nightly sync of name, email, and campaign status, a native connector is usually enough. If they later need updates within 5 minutes when a lead converts, I would move toward an API-based or event-driven design, because batch sync may be too slow.

My rule
Start with the least complex option that still meets the business need. If near-real-time updates become required, I would favor a design with webhooks or APIs plus retries and monitoring, because it gives better control over freshness and failure handling than a purely batch connector.

Requirements (clarify): reliable end-to-end flow that (1) decrement inventory, (2) charge payment, (3) notify shipping; services may fail transiently; system must surface status to users/support, avoid double-charges, and support compensating actions.

High-level architecture:

• Orchestrator (Saga orchestrator service) coordinates steps and records saga state in durable store (event log / workflow DB).
• Three participant services (Inventory, Payments, Shipping) expose idempotent APIs.
• Message bus (Kafka/Rabbit) for events and retries; monitoring/observability stack (Prometheus, ELK, tracing).

Flow & error handling:

1. Orchestrator starts saga, marks state PENDING.
2. Call Inventory.reserve(orderId, items) — idempotent: accepts client-generated requestId, returns reservationId. Retry with exponential backoff on transient errors; fail-fast on validation errors.
3. On success, call Payments.charge(orderId, amount, requestId). Use two-phase where Payment supports prepare/commit or simply charge with ability to refund (compensate). Ensure idempotency via requestId and idempotency key.
4. On payment success, call Shipping.createShipment(orderId, details).

Partial failures & compensations:

• If Payments fails after Inventory reserved: orchestrator issues Inventory.release(reservationId) as compensating action.
• If Shipping creation fails after charge: orchestrator initiates Payment.refund(orderId, idempotency key) and Inventory.release if appropriate.
• Compensation operations are retried until confirmed; compensations themselves are idempotent.

Retries, timeouts, idempotency:

• Per-step retry policy: exponential backoff, circuit breaker to avoid cascading failures.
• Timeouts per call (shorter than human-perceivable limits); long-running sagas transition to “requires attention” state after threshold.
• All participant APIs require idempotency keys (client requestId) and return deterministic status; operations must be safe to replay.

Visibility & user-facing status:

• Saga state machine exposes statuses: PENDING → IN_PROGRESS → COMPLETED → FAILED → COMPENSATING → NEEDS_SUPPORT.
• Provide user-facing statuses: “Processing”, “Payment failed — retrying”, “Order placed”, or “Action required”.
• Support UI: support console with full audit trail, event timeline, ability to trigger manual compensation/retry with role-based controls.

Monitoring & alerting:

• Instrument events and spans (distributed tracing) per saga; emit business metrics (saga success/failure rates, avg duration).
• Alerts on rising compensations, repeated refunds, or long-running sagas.
• Dashboards showing failure hotspots and per-step latencies.

Trade-offs:

• Orchestrator (centralized) gives clear visibility and easier compensation logic; choreography reduces coupling but complicates visibility.
• Using real refunds vs. two-phase commit: refunds simpler to implement with external payment gateways; two-phase reduces customer impact but is harder to integrate.

This design ensures safe retries, clear compensations, operational visibility, and predictable user-facing statuses while minimizing double-charges through idempotency.