Integration Patterns and API Design Questions

Focuses on integration concepts, data flow, and API design as the foundation for connecting systems and services. Coverage includes data integration techniques such as ETL and ELT, change data capture, data warehousing, synchronization and eventual consistency challenges, latency and throughput considerations, middleware and messaging solutions, and common integration patterns used in marketing and enterprise stacks. For APIs, topics include what APIs are and why they matter for developer products, REST versus GraphQL trade offs and use cases, HTTP methods and semantics, authentication and authorization patterns, rate limiting and throttling, versioning strategies, idempotency and error handling, documentation and developer experience, monitoring and service level considerations, and how API choices affect product and business decisions.

EasyTechnical

24 practiced

Explain idempotency in the context of HTTP APIs and integrations. Provide two practical implementation patterns for making a non-idempotent operation (such as a payment capture) idempotent, including storage TTL and cleanup considerations for idempotency records.

Sample Answer

Idempotency means that performing the same operation multiple times has the same effect as performing it once. For HTTP APIs and integrations this prevents duplicate side effects (e.g., duplicate charges) when clients retry due to timeouts or network errors. Idempotency is critical for reliability in payments and other non-repeatable operations.

Pattern 1 — Client-generated idempotency key (recommended)- How: Client sends a unique Idempotency-Key header (UUID or composite of user/order/timestamp) with the payment capture request. Server stores a record keyed by (Idempotency-Key, resource) containing request payload, result (success/failure), HTTP status, and timestamp.- Behavior: If a duplicate key arrives, server returns stored response instead of reprocessing.- TTL & cleanup: Keep records for a TTL that covers expected retry windows (commonly 24–72 hours). After TTL, delete or archive records. For compliance/audit, persist a compact audit log elsewhere before deletion.- Trade-offs: Simple, low client complexity; requires server-side storage and careful key validation to avoid accidental reuse.

Pattern 2 — Server-side deduplication via semantic request fingerprinting- How: Server derives a deterministic fingerprint from business identifiers (order_id, payment_method_id, amount). On first request, create a dedup lock/record and proceed with capture; subsequent identical fingerprints short-circuit.- TTL & cleanup: Use short-lived locks during processing (seconds–minutes) plus a longer-lived dedup record (24–90 days) to prevent late retries from recharging. Periodic background job cleans expired records; move older records to long-term audit storage if needed.- Trade-offs: No client changes if fingerprintable; risk of false dedupe if payload legitimately changes (requires strict fingerprint design).

Implementation considerations (architectural):- Atomicity: Use transactional datastore operations (compare-and-set) or DB unique constraints to avoid races.- Persistence choice: Redis for low-latency TTLed keys; SQL/NoSQL for durable records with indexes for queries.- Visibility & monitoring: Expose idempotency status in logs/metrics; surface to clients with headers (e.g., Idempotency-Status).- Security: Validate keys and rate-limit to avoid abuse.- Compliance/audit: Retain sufficient history per regulations; separate short-lived dedup records from long-term audit trails.

These patterns ensure reliable, predictable behavior for payment captures while balancing storage cost, latency, and operational complexity.

HardTechnical

30 practiced

Design an API versioning, compatibility testing, and deprecation strategy for a platform with thousands of dependent clients whose upgrade schedules vary widely. Cover automated compatibility checks, staged rollouts, SDK coordination, deprecation timelines, and how to measure client readiness for deprecation.

Sample Answer

Requirements & goals:- Maintain backward compatibility while enabling evolution.- Minimize client disruption across thousands of clients with varied upgrade cadences.- Provide measurable readiness metrics and a predictable deprecation path.

High-level strategy1. Versioning model- Use semantic versioning for API behavior (MAJOR.MINOR.PATCH) and clear resource-level versioning (e.g., /v1/orders). MAJOR = breaking changes. Support concurrent major versions for a defined overlap period.- Prefer additive, non-breaking changes where possible (new fields, headers, endpoints).

2. Automated compatibility checks- Contract tests: maintain OpenAPI/JSON Schema specs per major version; run automated contract validation in CI against server implementations.- Consumer-driven contract testing: publish stub servers and Pact or similar contracts contributed by key clients; CI validates provider against consumer contracts.- End-to-end regression sandbox: execute representative client flows (captured from production traffic anonymized) in a staging environment on every change.- Static analysis: API diff tooling to flag breaking changes (openapi-diff).

3. Staged rollouts & feature flags- Canary toggles per client segment (by client ID, region, or traffic class). Roll changes to small percentage, observe metrics, then widen.- Shadowing: mirror production traffic to new API implementation to detect runtime issues without affecting clients.- Automated health checks, latency/error budgets and rollback automation.

4. SDK coordination- Maintain official SDKs aligned to API major versions; version SDKs semver and publish migration guides and codemods.- Automate SDK release tests: run SDK integration tests against staging and canary endpoints before general availability.- Communicate SDK deprecations alongside API timelines; provide LTS SDKs for clients that can’t upgrade quickly.

5. Deprecation policy & timelines- Public deprecation policy: e.g., announce deprecation T0, support for 12 months minimum, security patches only during final 6 months, EOL at T0+12m.- Multi-channel notifications: dashboard, email to technical contacts, in-band warning headers in API (e.g., Deprecation, Sunset), and usage reports.- Phased enforcement: warnings → rate-limited degraded responses → 404/410 after EOL.

6. Measuring client readiness- Track per-client usage by API version and endpoint; compute active client cohorts and traffic % on deprecated versions.- Readiness metrics: percentage of clients migrated, % traffic on new version, SDK usage, open migration tickets.- Define thresholds for escalation (e.g., if >95% of traffic moved, proceed to final EOL).- Provide self-service migration reports in portal and offer migration assistance: runbooks, sample code, migration scanners to highlight affected endpoints/fields.

Trade-offs & governance- Maintainability vs. cost: supporting multiple majors increases ops/testing cost—limit overlap windows.- Create an API governance board to approve breaking changes and ensure rollout and migration support.

This plan balances automation (contract tests, diffs, canaries, shadowing) with clear communications, SDK coordination, and measurable deprecation gates to protect both platform stability and client operations.

MediumSystem Design

21 practiced

How would you enforce consistent per-customer rate limits across a multi-region API platform while keeping decision latency low? Discuss local enforcement, global coordination, approximate counters, and trade-offs around strict consistency versus responsiveness.

Sample Answer

Requirements & constraints (clarify): per-customer limits apply globally (e.g., 1000 req/min), multi-region traffic with <10ms decision latency target, tolerate occasional small over- or under-enforcement but must avoid large sustained quota breaches, need for billing/analytics accuracy.

High-level approach: hybrid local enforcement + asynchronous global coordination. Enforce a low-latency local token/leaky-bucket at each region’s API gateway for immediate allow/deny. Maintain a lightweight global view via approximate counters that sync periodically or with probabilistic aggregation to bound global usage.

Architecture components:- Edge gateways (per region): enforce local token-bucket with per-customer local quota = global_quota * (region_weight) + small burst buffer. Immediate decision (sub-ms).- Fast local store: in-process or local Redis for counters to keep latency low.- Global aggregator service: consumes counter deltas from regions (streaming via Kafka) and maintains global approximate counters using CRDTs or windowed sketches.- Global policy store: authoritative config for limits; pushed to regions via config service/consul.- Reconciler & throttling controller: if aggregator detects global exceedance, it issues tighten commands to regions (reduce local quotas) or rejects further requests via control plane.

Approximation techniques:- Use per-region token buckets with periodic (e.g., 1–5s) delta export. Aggregate with additive-only CRDTs (PN-Counters) to get eventual totals.- For lower bandwidth, use probabilistic sketches (e.g., Count–Min) to approximate high-cardinality counters; accept small error rate and one-sided overcount/undercount trade-offs.- Optionally use sliding-window sketches to handle bursty traffic.

Trade-offs / rationale:- Strict global consistency (synchronous global counter) forces cross-region RPC on each request — high latency and availability risk. Only appropriate if absolute per-request correctness is required (rare).- Hybrid model gives fast local decisions and eventual global correctness; you accept small short-term violations (bounded by local burst buffers and sync frequency) but keep responses low-latency and highly available.- Tighter consistency: reduce sync interval and buffer sizes, or use leader-based global token allocation (e.g., lease tokens to regions) — increases complexity and latency for rebalancing.- Accuracy vs cost: more frequent syncs and stronger consistency increase network/coordination cost and possibility of stall under partition.

Operational considerations:- Define SLOs: max allowed global overshoot percentage and window.- Instrument metrics: per-region/local/global counters, sync lag, reconciliation actions.- Handle edge cases: region partitions (pause global sync, increase conservative local limits), customer hot-sharding (one region dominates — move to on-demand global allocation), and backpressure propagation.- Billing/analytics: rely on reconciled global counters for invoicing; keep logs for dispute resolution.

Recommendation: Start with token-bucket per region + 1–5s delta sync using CRDTs or Kafka aggregation, tune local buffer sizes and sync frequency to meet your overshoot SLO. If customers require strict limits, offer an opt-in tier that uses centralized allocation or per-customer regional pinning.

HardTechnical

29 practiced

Architect a CDC-based near-real-time analytics pipeline with an end-to-end latency target under 2 seconds for critical metrics. Include the capture mechanism, streaming stack, transformation layer, sink into an OLAP store, exactly-once or at-least-once semantics choices, schema evolution handling, late-arriving data strategy, and backfill approaches.

Sample Answer

Requirements clarification:- Functional: capture DB changes (CDC) and produce critical metrics with end-to-end latency <2s.- Non-functional: high availability, scalable to millions of events/sec, support schema evolution, provide exactly-once for metrics (preferred), handle late data and backfills.

High-level architecture:Client DBs -> CDC capture (Debezium) -> Kafka (topic per table, compacted changelog) -> Stream processing (Flink for low-latency stateful transforms) -> OLAP sink (ClickHouse/Druid or cloud OLAP like BigQuery with streaming) -> Downstream dashboards/alerts

Capture mechanism:- Debezium connectors capture binlog/WAL and produce Avro/Protobuf messages to Kafka topics. Use connector-level heartbeat and snapshotting for initial state.- Kafka topics partitioned by natural key for ordering; use compacted topics for changelog.

Streaming stack & transformation layer:- Apache Flink (low latency, event-time handling, state backend RocksDB) performs: - Keyed aggregations for metrics (rolling windows, sessionization) in event time. - Joins across streams/tables using stateful keyed state. - Use RocksDB + incremental snapshots for fault tolerance.- Use Kafka as durable message bus; enable record timestamps from CDC source.

Sink into OLAP store:- Use a transactional sink supporting upserts and low-latency ingestion: - ClickHouse (MergeTree with Kafka engine + materialized views) or Druid for sub-second queries. - For cloud: BigQuery streaming inserts + clustered partition tables (note: BigQuery may have slightly higher query latencies).- Implement Flink two-phase-commit sink (exactly-once) to ensure atomic writes per checkpoint.

Exactly-once vs at-least-once:- For critical metrics pick exactly-once end-to-end: enable Flink checkpointing, Kafka consumer offsets committed on checkpoint, use transactional sinks (two-phase commit). This avoids double-counting in aggregations.- If sink cannot support transactions, use idempotent writes (upserts with primary key + last-write metadata) and deduplication on sink side—this yields effectively exactly-once for metrics.

Schema evolution:- Use Confluent Schema Registry (or equivalent) with Avro/Protobuf/JSON Schema; enforce backward/forward compatibility rules per domain.- Include schema id in message; Flink uses registry client to deserialize and adapt. For additive changes, default values used. For incompatible changes, coordinate versioned topics or deploy migration flow: write new topic and map old->new in stream.

Late-arriving data strategy:- Use event-time processing with watermarks in Flink. Set allowed lateness (e.g., 30s) for critical metrics; emit early speculative results while retractions correct aggregates.- Maintain changelog of previous aggregates; when late event arrives, produce correction events (delta) to OLAP sink and dashboards support incremental updates or retraction.- For user-visible SLA (<2s), show provisional metrics and mark as tentative until watermark passes.

Backfill approaches:1. Snapshot + replay: - Take consistent DB snapshot (Debezium snapshot or export), publish to a separate Kafka topic namespace; run a backfill Flink job that consumes snapshot topic and then replays binlog topics to catch up. Merge with live stream using per-record versioning.2. Batch rebuild: - For large historical recompute, run a batch Spark job to compute metric partitions and write to OLAP using partition-level replace (atomic partition swap).3. Dual-write window: - Temporarily dual-write metrics from both stream and batch, then cutover once backfill validated.

Operational considerations:- Monitoring: end-to-end latency SLOs via synthetic events, Kafka lag, Flink checkpoint durations, sink write latency.- Autoscaling: partition count and Flink parallelism tuned to meet <2s latency; prefer more partitions and scaled task slots.- Security: TLS, ACLs, encryption at rest, schema registry auth.- Disaster recovery: cross-region Kafka mirror (MirrorMaker2) and state backup.

Trade-offs:- Exactly-once adds complexity and some throughput overhead; acceptable for critical metrics.- Choosing ClickHouse/Druid yields lower query latency vs BigQuery which excels at scale/cost but may add a few hundred ms.

This design meets <2s E2E for critical metrics by using Debezium+Kafka for reliable low-latency capture, Flink for event-time stateful processing and transactional sinks or idempotent upserts for correctness, plus schema registry, watermarking/late-windowing for late data, and snapshot+replay/backfill strategies for historical recompute.

HardSystem Design

21 practiced

Design a global, distributed rate-limiting system that supports hierarchical quotas (organization > application > user), burst tokens, and fairness across tenants. Requirements: sub-5ms decision latency on the API path, 99.999% availability, and eventual consistency across regions. Discuss algorithms, storage backends, fairness enforcement, and failure handling.

Sample Answer

Requirements clarification:- Functional: enforce hierarchical quotas (org > app > user), support bursts (token buckets), ensure fairness across tenants.- Non-functional: decision latency <5ms on API path, 99.999% availability, eventual consistency across regions.

High-level design:- Edge rate limiter (sidecar or global edge PoP) makes real-time allow/deny decisions (<5ms) using local in-memory token buckets and a lightweight local store. Control plane (global services) manages quota config, long-term state, reconciliation and fairness allocation.

Quota model & algorithms:- Use hierarchical token buckets layered: each request must consume tokens from user, app, and org buckets. Each bucket = token-bucket with refill rate and max burst.- To keep API latency low, pre-allocate regional token leases for each bucket. Global controller periodically (e.g., every 1–10s) issues leases (X tokens/sec for region) so edge can serve requests locally without remote calls.- Fairness: allocate capacity from org-level down using a weighted fair-share allocator (e.g., Dominant Resource Fairness or Deficit Round Robin variant) in control plane to compute regional lease sizes per tenant based on weights/SLA. Edges implement local deficit counters to enforce per-tenant fairness within their leased capacity.- Global counters: use CRDT-based G-Counters or PN-Counters (append-only deltas) to allow eventual convergence of usage across regions without coordination.

Storage backends:- Control/config store: strongly consistent DB (CockroachDB or Spanner) for quota config, SLA, weights.- Global metering/ledger: durable append store (DynamoDB, Bigtable, Cassandra) plus object storage for audit logs; use per-region shards.- Short-term lease/state: Redis (clustered) in each region for fast lease issuance and short TTLs.- CRDT reconciliation store: blob store for deltas; combine per-region deltas into global view.

Decision path & latency:- Edge checks in-memory buckets -> if tokens available, allow and decrement local counters (O(1) in-memory). If local bucket depleted but within grace, edge can: - Borrow from regional Redis (single hop, <2ms typical) using a sized lease request. - If borrow fails, apply soft-deny or queue depending on policy.- No synchronous global consensus on API path — all heavy coordination off-path.

Consistency and correctness:- Eventual consistency across regions: per-region deltas of token consumption published asynchronously to global metering. Global control recomputes allocations and issues adjusted leases.- To avoid global over-usage, control plane keeps conservative global view when issuing leases (safety margin), and edges honor soft-shutdown signals when global budgets near exhaustion.

Failure handling:- Edge failure: sidecar restarts rehydrate from Redis leases; short-lived in-memory tokens may be lost — acceptable (small drift) and corrected during reconciliation.- Control plane failure: use multi-region control plane with leader election (Raft) for lease issuance; if unavailable, edges continue operating on last-valid leases until expiry (fail-open for availability), with configurable fail-open/fail-closed policies per tenant/SLA.- Network partition: edges continue using local leases; to control fairness, leases have short TTLs and conservative sizes to limit runaway usage during long partitions.- Double-spend prevention: leases are unique, time-bounded tokens; reconciliation consumes/credits delta; use monotonic per-lease IDs to ensure idempotency.

Fairness enforcement details:- Control plane runs global periodic allocator (every T seconds) that: - Collects usage deltas (CRDT merges) and backlog. - Computes per-tenant regional quotas via weighted fair share, respecting org-level caps. - Issues regional leases; Redis used to atomically record lease assignments.- Edges enforce per-tenant local fairness by maintaining per-tenant deficit counters and serving requests proportional to assigned weight.

Metrics, monitoring, and debugging:- Emit per-lease, per-bucket metrics (consumed, remaining, refills) to observability pipeline.- Alerts for lease exhaustion, unusual borrowing, or partition-induced overshoot.- Audit trail: immutable global metering logs for billing and for reconciling conflicts.

Trade-offs:- Latency & availability prioritized via pre-allocation and local enforcement; this trades strict global instantaneous consistency for eventual correctness.- Conservative lease sizing prevents global overshoot but may reduce burst capacity across regions; SLA customers may purchase higher safety margins.

Implementation roadmap:1. Build edge sidecar with in-memory buckets + Redis integration.2. Implement control plane allocator and lease API (gRPC).3. Integrate CRDT deltas + global metering pipeline.4. Add fairness weighting, SLA policies, and fail-open controls.5. Staged rollout per region with chaos testing for partitions and failover.

This design meets sub-5ms API decisions by keeping the critical path local, attains 99.999% availability via edge fail-open with short-lived leases, and achieves eventual cross-region consistency using CRDTs and periodic global reconciliation while enforcing hierarchical quotas and tenant fairness.

Unlock Full Question Bank

Get access to hundreds of Integration Patterns and API Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.