Integration Patterns and Messaging Systems Questions

Understand approaches to integrating different systems: synchronous APIs, asynchronous messaging (queues, topics, events), batch processing, and data synchronization. Know when to use message queues vs. publish-subscribe vs. event streams. Understand the reliability patterns (acknowledgments, retries, dead-letter queues) and consistency implications. Know about API gateways, service discovery, and orchestration. Discuss trade-offs between tight coupling and loose coupling, and how integration choices affect system resilience.

MediumTechnical

66 practiced

Describe how you would design a schema evolution policy for protobuf/Avro-based event schemas used across multiple teams. Include compatibility rules, validation, and deployment steps to avoid breaking consumers during rollout.

Sample Answer

Goal: ensure safe, predictable schema evolution so producers can deploy changes without breaking many consumers across teams.

Requirements & constraints:- Multiple teams with independent release cadences- Low-latency event flow, Avro/Protobuf formats- Need automated validation and clear governance

Compatibility rules:- Use semantic compatibility levels in a central schema registry: - Default: Backward compatibility for producers (new producer can write, old consumers can read) - For public/shared topics: Backward and Forward (Full) compatibility required - For internal/private topics: Team can opt-in to relaxed rules with explicit approval- Concrete rules (applies to Avro/Protobuf): - Additive changes allowed: adding optional fields with defaults - Removing fields: mark deprecated first; removal only after two release cycles and explicit consumer acknowledgment - Field renames: use aliases (Avro) or reserved numbers/names in Protobuf; avoid reusing field IDs - Changing types: allowed only if widening (e.g., int -> long) and proven safe - Enforce stable field identifiers for Protobuf (keep tag numbers immutable)

Validation:- Centralized Schema Registry (e.g., Confluent Schema Registry, Apicurio) enforces compatibility checks on push- CI pipeline checks on PRs: - Syntax validation (protoc/avro-tools) - Compatibility check against latest registered schema - Lint rules (naming, documentation-required, reserved ranges)- Automated contract tests: - Consumer-driven contract tests: consumers publish test suites that producers run in CI to ensure compatibility - Integration tests: spin up consumer stubs to validate reading new events- Static analysis: detect unsafe changes (field id reuse, removed defaults)

Deployment & rollout:1. Developer opens schema change PR referencing the topic and change type.2. CI runs validations and compatibility; if failing, block merge.3. Merge to main triggers canary deployment: - Deploy producer version capable of writing both old and new fields (use feature flag to enable new fields) - Run producer in shadow mode writing new schema to a test topic; run consumers against shadowed events4. Gradual traffic shift: - Enable new fields for small percentage of messages or specific partitions - Monitor consumer errors, deserialization metrics, consumer lag5. Consumer upgrade window: - Notify consumers of upcoming change (schema registry alert + changelog + deprecation timeline) - Consumers upgrade to read new fields; run integration tests against staging data6. Finalize: - After all consumers confirm, remove deprecated fields following policy schedule - Update schema registry and documentation; increment major version if breaking change unavoidable

Governance & tooling:- Schema owners and approvers per domain; automated approvals possible for safe additive changes- Standard templates for change requests (impact, consumers, rollback plan)- Dashboards for compatibility status, consumers behind on versions, recent schema changes- Regular audit and quarterly review of deprecated fields

Rollback & incident handling:- If consumers break: disable producer feature flag to stop new fields, revert producer artifact, or route to old schema topic.- Run postmortem and update rules to prevent recurrence.

Trade-offs:- Strict full compatibility reduces risk but slows innovation; balance via approval workflows and private topics policy.- Investing in consumer-driven contracts and a schema registry increases upfront cost but prevents large-scale outages.

This policy offers clear, automated checks, staged rollouts, and organizational processes so teams can evolve Avro/Protobuf schemas safely at scale.

MediumTechnical

80 practiced

Explain how API gateways can implement rate limiting and quota enforcement for tenants in a multi-tenant SaaS platform. Describe technical options for per-tenant quotas, burst handling, and enforcement points (edge vs upstream).

Sample Answer

High-level approach: an API gateway enforces per-tenant rate limits and quotas by identifying the tenant (API key, JWT claim, client id), applying policy rules (requests/sec, concurrent connections, monthly quota), and tracking usage in a consistent store. Key concerns: accuracy, latency, cost, burst tolerance, and failure modes.

Per-tenant quota implementation options:- In-memory token bucket per gateway instance: very low latency, supports bursts, but requires careful sync for multi-instance deployments (local counters lead to overages).- Distributed counters (Redis/Etcd/Cassandra): strong consistency or eventual consistency modes; use atomic INCR with TTL for fixed windows, or Lua scripts for sliding windows/token bucket. Balanced choice for correctness and scale.- Hybrid: local token bucket for immediate bursts + periodic reconciliation with global store to prevent long-term drift.

Burst handling strategies:- Token bucket / leaky bucket: allows short bursts while enforcing long-term average. Configure burst size relative to tenant SLAs.- Separate burst pool per tenant or priority classes: high-tier tenants get larger burst tokens.- Grace/queueing: short queuing with retry-after headers, or soft-throttle (return 429 with Retry-After) when bursts exceed allowance.

Enforcement points — edge vs upstream:- Edge (gateway/edge CDN): pros — low latency, prevents wasted upstream work, easier to present consistent 429/Retry-After; cons — needs distributed state or eventual consistency, potential complexity during scale/outages.- Upstream (service mesh / backend): pros — authoritative usage accounting, richer context for decisions; cons — wasted upstream resources during attacks, increased latency before rejecting.- Recommended hybrid: enforce conservative fast checks at edge (simple per-second limits, local tokens) to drop abusive traffic early, and authoritative quota accounting upstream using distributed counters for monthly/longer quotas and reconciliation. For critical tenants, centralize enforcement with strong consistency.

Operational considerations:- Monitoring/alerts for quota exhaustion, misconfiguration, or cache failures.- Metering and billing integration: export usage to billing pipelines, include idempotency and reconciliation windows.- Fail-open vs fail-closed policy: choose per SLA (fail-open with logging for non-critical; fail-closed for paid tiers).- Security: protect quota store from DoS (rate-limit write patterns), and authenticate tenant identity.

Trade-offs summary:- Low-latency local enforcement vs global accuracy — hybrid patterns usually best.- Redis-based token bucket with Lua scripts gives atomic, fast operations; combine with local caching to support bursts.- Clear SLAs and tenant classes simplify policy complexity and capacity planning.

MediumSystem Design

67 practiced

Design an error handling and dead-letter strategy for a multi-tenant messaging system where some tenants produce malformed messages intermittently. Specify how you isolate tenant failures, provide visibility, and allow for message replay after fixes.

Sample Answer

Requirements & constraints:- Multi-tenant messaging pipeline must isolate tenant failures, surface metrics/alerts per tenant, store malformed messages for inspection, and support replay after fixes without impacting healthy tenants. Low-latency for normal flow; scalable to many tenants.

High-level architecture:- Ingest → Validation & Router → Tenant-specific Processing Queues → DLQ store (per-tenant) → Monitoring & Replay Service

Components & responsibilities:- Ingest layer (API/Gateway or broker like Kafka): accepts messages with tenant-id header.- Validation & Router: lightweight schema/format checks (JSON/schema, size, auth). Valid messages forwarded to tenant processing topics/queues; invalid ones forwarded to a per-tenant dead-letter topic (DL-topic).- Tenant-specific processing queues/topics: logical separation (either separate topics per tenant or partitioning with tenant key + quotas) to prevent noisy neighbors.- DLQ store: immutable storage (object store + metadata DB) holding raw payload, validation errors, timestamps, offsets, and tenant-id.- Monitoring & Observability: per-tenant dashboards (error rate, DLQ count, volume), alerts on thresholds; sample messages surfaced in an audit UI.- Replay Service: authenticated UI/CLI to filter DL entries, apply fixes (manual edit or transformation), and re-inject into validation stage or into specific offsets for ordered systems. Support bulk replay and provenance tagging.- Quotas & Backpressure: circuit-breaker per tenant to throttle producers that exceed error/volume thresholds; move into quarantined mode with notification.

Data flow:1. Producer sends message → broker/gateway attaches tenant-id.2. Validation fails → message written to tenant DL-topic and DL store; validation metadata recorded.3. Alerts/metrics updated; ops/tenant notified.4. After fix, operator uses Replay Service to revalidate and re-enqueue messages for processing.

Isolation strategies:- Logical tenant topics or partitioning + broker quotas to prevent cross-tenant impact.- Rate limits and circuit breakers per tenant.- Separate DL storage namespaces per tenant to enforce retention and access controls.

Visibility & audit:- Structured error records: schema violations, parse errors, downstream consumer errors, with stack traces, original payload, and contextual metadata.- Dashboards with drill-downs, sample messages, and SLA impact.- RBAC for tenant vs ops views.

Replay considerations & guarantees:- Preserve original offsets/sequence where ordering matters; support ordered replay by replaying into tenant-specific ordered queue.- Idempotency: consumers should support idempotent processing or replay service should mark messages with replay-id to avoid duplicates.- Atomicity: track replay jobs with status and provenance; allow partial retries.

Trade-offs:- Per-tenant topics increases management overhead but gives strongest isolation; partitioning with quotas reduces operational burden but needs careful keying to avoid hotspotting.- Eager validation at ingress reduces downstream errors but adds latency; balance by offloading heavier schema checks asynchronously while catching basic issues upfront.

This design isolates noisy tenants, provides clear visibility and audit trails, and enables safe replays while protecting healthy tenants and maintaining operational control.

HardSystem Design

74 practiced

Design a multi-region messaging topology for a global e-commerce platform to reduce cross-region latency while ensuring eventual consistency. Explain where you would place producers, consumers, replication, and how you would handle conflicts and failover.

Sample Answer

Requirements:- Low cross-region latency for reads/writes; eventual consistency across regions; global scale; high availability and automated failover; reasonable conflict resolution.

High-level architecture:- Per-region message broker cluster (e.g., Kafka/MQ/Managed PubSub) colocated with regional app servers and edge CDN.- Global control plane for metadata, partition routing, and health (centralized service like Conductor).- Asynchronous cross-region replication between regional clusters (replicate topics/partitions) using log-based replication.

Placement:- Producers: Run in the user’s nearest region (edge/API layer) to minimize write latency. They write to the local regional broker.- Consumers: Prefer regional consumers for region-specific processing (e.g., order confirmation). Global consumers (analytics, billing) subscribe to replicated global topics.- Replication: Each regional broker appends messages to a local log; an async log-replication pipeline (mirror-maker style or CDC/stream-replication) ships those logs to other regions' brokers or to a central ingest where they are re-published. Use partitioning by logical key (userID, orderID) to keep related events collocated.

Consistency & Conflict Handling:- Model: Eventual consistency with strong per-key ordering (single partition per key).- For commands that can conflict (concurrent updates to same cart/profile), use: - Primary-authority per key: elect region owner for a key (hash-based or sticky session) to serialize writes when strong ordering required. - CRDTs for mergeable state (shopping cart, counters) so merges are commutative and conflict-free. - For non-mergeable state (payment status), use vector clocks + application-level conflict resolution: present conflicts to a workflow or use last-writer-with-lease backed by monotonic timestamps (with NTP/Hybrid Logical Clocks) and compensating transactions.- Idempotency: Require producers to include idempotency keys to safely retry.

Failover:- Regional broker fail: route producers to nearest healthy region via global DNS + health checks; producers should buffer locally (durable queue) and resume replication once reachable.- Partition leader fail: elect new leader within cluster; for cross-region leader election (if primary-authority used) fallback to pre-defined hash-based owner or run raft/consensus among region controllers for stable ownership.- Full-region outage: Consumers in other regions will process replicated logs; on recovery, use checkpointed offsets and rebootstrap replication with verification.

Operational considerations:- Monitoring/observability for replication lag, per-partition conflicts, consumer lag.- Backpressure and throttling when replication lag grows.- Security: mutual TLS between regions, encryption-at-rest, and access policies.- Trade-offs: Async replication reduces latency but allows windows for conflicts; primary-authority + partitions increase complexity but reduce conflicts. Use hybrid approach: CRDTs for common mergeable types, authoritative region for critical operations.

This topology balances low-latency local operations with eventual global convergence, predictable conflict semantics, and resilient failover.

EasyTechnical

126 practiced

List and briefly describe three reliability patterns used with messaging systems: acknowledgments, retries, and dead-letter queues. For each, give a practical example of how a Solutions Architect would apply them in a payment-processing pipeline.

Unlock Full Question Bank

Get access to hundreds of Integration Patterns and Messaging Systems interview questions and detailed answers.

Join thousands of developers preparing for their dream job.