Kafka, Message Queuing, and Event Sourcing Questions

Design Kafka-based architectures for event streaming. Understand topics, partitions, consumer groups, and offset management. Consider Kafka as an event store and how to build systems on event sourcing principles.

MediumTechnical

22 practiced

Describe an incident you led where a Kafka-based data pipeline produced incorrect analytics results for a day. Explain your incident management steps: triage, stakeholder communication, technical remediation, root-cause analysis, and how you prioritized permanent fixes versus quick mitigations.

Sample Answer

Situation: One morning our analytics dashboards showed a 25% drop in active-users metrics for the previous day. I was the on-call data engineer responsible for our Kafka-based ingestion pipeline (Producers → Kafka topics w/ Schema Registry → Spark streaming jobs → Redshift).

Task: Lead triage, communicate to stakeholders, implement a quick mitigation so reports were accurate, then perform RCA and deploy permanent fixes.

Action:- Triage: I checked pipeline health (Kafka broker metrics, consumer lag, schema evolution logs). Found a producer had rolled out a client that inadvertently changed the Avro schema (added a non-nullable field) without bumping the schema version; consumers silently failed and a subset of events were dropped into a DLQ. Spark jobs had partial output causing the 25% shortfall.- Stakeholder communication: Within 30 minutes I sent an incident Slack to analytics, product, and SRE with impact, scope, ETA, and hourly updates. I escalated to the analytics lead and scheduled a 1:1 post-incident briefing.- Quick mitigation: Reverted producer to previous version via feature-flag rollback to stop further invalid events. Replayed DLQ messages after transforming them with a temporary script to populate the missing field with a safe default so analytics could re-run for the day. This restored dashboards within 6 hours.- Technical remediation & RCA: Ran a postmortem. Root cause: lack of producer-side schema validation and missing enforcement of non-nullable field rules at ingestion. Contributing factors: no automated consumer schema compatibility checks and no alert on DLQ volume.- Prioritization (permanent fixes vs quick mitigations): I prioritized (1) immediate rollback and DLQ replay to restore data (low-risk, high-impact), (2) medium-term fixes within two weeks: add producer CI step to validate schema compatibility with Schema Registry, enforce non-nullable fields as backward-incompatible in governance, and add alerts for DLQ growth and consumer errors, (3) long-term: adopt an automated contract-testing pipeline and add idempotent producers. I created tickets, estimated effort, and coordinated with platform and dev teams; the schema CI and alerts went into the next sprint, and contract testing was scheduled for the quarter.- Outcome & learning: Dashboards were corrected; stakeholders regained trust. Postmortem published with action owners and SLAs. We reduced similar incidents by adding schema checks and DLQ monitoring; within three months DLQ-related incidents dropped to zero.

Result: Rapid mitigation minimized business impact; structured remediation eliminated the root cause and improved pipeline resilience and cross-team processes.

HardTechnical

22 practiced

Case study: A fintech requires tamper-evident, replayable audit logs for regulatory review. Design an event-sourcing pipeline using Kafka that supports immutable event storage, encryption at rest, access controls, efficient replay, and cryptographic proof (e.g., hash chaining). Describe storage, retention, and queryability for auditors.

Sample Answer

Requirements clarification:- Append-only, tamper-evident, replayable events; encryption-at-rest; strict access controls; auditor-friendly queries; regulatory retention (e.g., 7+ years).

High-level architecture:- Producers -> Kafka (append-only topics) -> Kafka Connect sink -> Immutable object store (S3 with Object Lock/WORM, SSE-KMS) -> Audit Query API + Athena/Presto for ad-hoc queries. A small “ledger” service stores periodic signed roots/hashes.

Core components & responsibilities:1. Kafka - Topics configured as append-only (no deletions). Use retention window small for hot reads (e.g., 30d) and rely on immutable S3 for long-term. - Include event metadata in headers: event_id, timestamp, prev_hash, stream_sequence. - Enforce ACLs via Kafka ACLs + TLS client auth. Brokers use disk encryption and OS-level protections.2. Event immutability & cryptographic proof - Each event payload includes its SHA-256 hash and prev_hash (hash chaining). Example: event.hash = SHA256(event.payload || prev_hash). - Periodically (e.g., per-hour/segment) compute a Merkle root over that segment and sign it with a KMS-held private key; write signed root to the ledger service and as an immutable object in S3. - Ledger service exposes signed roots for auditor verification.3. Long-term immutable storage - Kafka Connect sinks to S3 in Parquet/AVRO partitioned by stream/date, using S3 Object Lock in Compliance mode (WORM) and SSE-KMS. - Store original Kafka metadata (offset, headers) with each object.4. Encryption & keys - Broker disks encrypted; S3 SSE-KMS with CMKs; KMS for signing Merkle roots. Rotate keys per policy; keep old keys for verification.5. Access control & auditing - IAM + fine-grained S3 policies for roles. Read-only "auditor" IAM role, VPC endpoints, no delete permissions. - Kafka ACLs restrict produce/consume. All accesses logged to CloudTrail/connector logs.6. Replayability - For short-term: consumers can rewind offsets in Kafka. - For long-term: a Replay Connector can stream historical Parquet back into a Kafka replay topic (or to a sandbox cluster) preserving original offsets and headers. Use partitioning and parallelism for speed.7. Queryability for auditors - Expose Athena/Presto over the Parquet lake with views: by event_id, account_id, timeframe; include original headers and proofs. - Provide an Audit API that returns event(s) + chain proof: event payload, event.hash, prev_hash chain up to nearest signed Merkle root and its signature.8. Integrity verification process - Scheduled verification job: re-compute hashes for objects, recompute Merkle roots and verify signatures against ledger; report anomalies.9. Retention & governance - Kafka: hot window 7–30 days. - S3: retention per regulation (e.g., 7+ years) enforced via Object Lock; lifecycle to move older data to Glacier Deep Archive but keep WORM semantics. - Policy for key retention to allow future verification.

Trade-offs:- Keeping Kafka forever is expensive; offloading to S3 with strong WORM is standard. Hash-chaining per event increases payload size but gives per-event tamper evidence. Merkle roots provide efficient proof and compact signed checkpoints.

Edge cases & operational concerns:- Clock skew: use monotonic sequence + timestamp sources (NTP).- Partition rebalancing: include sequence and prev_hash per partition to preserve chain per-partition.- Key compromise: rotate/revoke keys and maintain old keys for verification; store signed roots externally.

This design yields append-only audit trails, cryptographic proof of integrity, encrypted-at-rest storage, controlled auditor access, and efficient replay from cold storage when needed.

MediumTechnical

35 practiced

Outline the design of a Kafka Connect pipeline to stream CDC from PostgreSQL to Kafka, including configuration for Debezium, single message transforms for filtering/PII masking, schema evolution handling, and performance tuning for high-throughput tables.

Sample Answer

Requirements:- Capture CDC from Postgres to Kafka reliably (at-least-once), preserve schema, support schema evolution, filter rows/fields and mask PII, and sustain high throughput for large tables.

High-level architecture:Postgres WAL -> Debezium PostgreSQL connector (Kafka Connect) -> Kafka topics (Avro/JSON+Schema) -> Consumers/streaming jobs.

Debezium connector (important settings):

json

{
  "name":"pg-debezium",
  "config":{
    "connector.class":"io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname":"db.host",
    "database.port":"5432",
    "database.user":"replication_user",
    "database.password":"secret",
    "database.dbname":"appdb",
    "database.server.name":"prod-db",
    "plugin.name":"pgoutput",
    "slot.name":"debezium_slot",
    "snapshot.mode":"initial",
    "tombstones.on.delete":"false",
    "key.converter":"io.confluent.connect.avro.AvroConverter",
    "value.converter":"io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url":"http://sr:8081"
  }
}

SMTs for filtering and PII masking:- Use org.apache.kafka.connect.transforms.Filter to drop undesired tables/ops.- Use MaskField or custom SMT to redact fields, or RegexRouter + ReplaceField.Example to mask email and ssn:

json

"transforms":"mask,route",
"transforms.mask.type":"org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.mask.blacklist":"email,ssn",
"transforms.route.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex":"(.*)","transforms.route.replacement":"${topic}"

For partial masking, implement a custom SMT or use Single Message Transform plugins that support scripting (e.g., JoltTransform or ScriptTransformer) to replace characters.

Schema evolution handling:- Use Schema Registry (Avro/Protobuf) with compatibility = BACKWARD or FORWARD depending on consumers.- Debezium emits before/after; enable "schema.history.internal.kafka.bootstrap.servers" and topic to persist DDL history.- Configure converters to use schema registry so changes register automatically. Test migrations: add nullable columns first, backfill if needed, and avoid destructive renames—use deprecation and new columns.

Performance tuning for high-throughput tables:- Increase connector tasks? Debezium Postgres is single-task per database; scale by logical DB partitioning: replicate by publication/slot per subset or use multiple connectors each subscribing limited tables via "table.include.list".- Tune Postgres: increase wal_sender, wal_buffers, max_wal_senders, and check wal_compression.- Kafka producers: configure batch.size, linger.ms, compression.type (lz4/snappy/zstd), and acks=all for durability trade-offs.- Connect worker tuning: increase heap, configure worker.max.request.size, producer/consumer configs via "producer.override.*" and "consumer.override.*".- For large tables, avoid full snapshot where possible: use snapshot.mode=never if logical decoding already in place, or snapshot.select.statement.overrides to minimize snapshot size.- Monitor lag, commit rates, and increase Kafka partitioning for target topics to allow downstream parallel consumers.

Observability & safety:- Enable metrics (JMX), audit connectors, set up alerts on connector status and lag.- Use end-to-end tests for schema changes and maintain migration runbooks.

This design provides reliable CDC, masked sensitive fields, managed schema evolution, and tuning options for high throughput.

MediumTechnical

31 practiced

Explain schema evolution strategies for Kafka messages using Avro or Protobuf and Schema Registry. Define backward, forward, and full compatibility, explain compatibility checks, and propose consumer/upcaster strategies when schema changes are required in production.

Sample Answer

Start by clarifying compatibility types (how new/old schemas interoperate):- Backward compatible: New schema can read data produced with the previous schema (consumers using new code can read old messages). Typical for adding optional fields with defaults.- Forward compatible: Old schema can read data produced with the new schema (consumers using old code can read new messages). Achieved by only adding fields that have sensible defaults and not removing required fields.- Full compatible: Both backward and forward: any producer/consumer version can read each other's messages.

How Schema Registry enforces checks:- Schema Registry stores schemas per subject and can be configured (NONE, BACKWARD, FORWARD, FULL). On producer registration it compares the new schema against the latest (or a compatibility level) using Avro/Protobuf rules and rejects incompatible changes.- For Avro: adding a field with a default is backward-compatible; removing a field is forward-compatible if the reader provides a default; changing field type requires specific safe promotions.- For Protobuf: reserved/removed tags and default values matter; Schema Registry applies analogous compatibility logic.

Practical evolution strategies for production:1. Additive changes first: prefer adding optional fields with defaults or new messages versions (v2 subject). This preserves backward compatibility.2. Use semantic versioning for subjects: topic-key/value-v1, -v2 to isolate consumers.3. Dual-writing: producers write both old and new formats (or write new format and a transformed legacy message) while consumers migrate.4. Upcaster/adapter layer on consume path: - Consumer-side upcasting: when a consumer reads older-format messages, run an upcaster that transforms payload to the newest in-memory model (stateless transformer, implemented in streaming job or client library). - Broker-side transformation: use Kafka Streams/KSQ/CDC jobs to produce a normalized topic with the latest schema.5. Consumer compatibility patterns: - Tolerant readers: code that ignores unknown fields and uses defaults. - Version-aware consumers: switch behavior based on schema id from registry (Schema Registry embeds id in the message), enabling safe branching logic.

Operational best practices:- Enforce compatibility level per subject in Schema Registry, run schema-check in CI to prevent accidental breaks.- Document allowed changes and provide migration playbooks (dual-write, transform jobs).- Test end-to-end with contracts and schema evolution tests (producer + consumer integration tests).- Monitor consumer errors and deserialization exceptions; maintain backward compatibility where downtime is unacceptable.

Example quick rules:- Safe: add field with default (Avro), add optional field (Protobuf).- Unsafe: removing required fields, renaming fields without aliases, changing field tags/types incompatible with promotion.

These patterns keep pipelines resilient while allowing iterative schema changes with minimal consumer disruption.

HardSystem Design

25 practiced

Design a large-scale event replay system to rebuild downstream materialized views from Kafka topics at petabyte scale. Include orchestration, parallelism, incremental checkpointing, snapshot bootstrapping, and strategies to limit costs and user-visible downtime during rebuilds.

Sample Answer

Requirements & constraints:- Rebuild materialized views from Kafka at petabyte scale with minimal user-visible downtime, bounded cost, support incremental restarts, and ability to bootstrap from snapshots.

High-level architecture:- Orchestrator (Kubernetes + Airflow/Argo Workflows) coordinates replay jobs.- Distributed workers using a scalable stream/batch engine (Spark Structured Streaming / Flink) that read Kafka partitions and write to target stores (OLAP DBs / key-value stores).- Snapshot store (object storage: S3/GS) for bootstrapped checkpoints and state snapshots.- Metadata service to track progress, partition ranges, checkpoints, and job idempotency.

Key components & flow:1. Snapshot bootstrapping: create consistent point-in-time snapshots of source materialized view state (CDC-enabled, or periodic export). Store snapshot manifest (offsets per topic partition).2. Partitioned replay plan: split Kafka offsets into many shards by (topic, partition, offset range, key-range). Planner balances shard size for parallelism and hotspot avoidance.3. Orchestration: orchestrator enqueues shard tasks with affinity to worker pools; uses autoscaling based on backlog.4. Worker processing: each task: - Loads snapshot state if applicable (from S3) for its key-range. - Reads Kafka from start_offset to end_offset using consumer groups with static assignment. - Applies deterministic idempotent transformations and writes to a staging sink (append-only files / partitioned tables). - Periodically commits incremental checkpoints (offsets + output manifests) to snapshot store and metadata service.5. Commit & swap: after all shards for a view finish, perform an atomic swap (rename pointers) or apply delta merge to minimize downtime. Use read-copy-update: build new view in parallel, then switch traffic.

Parallelism & scaling:- Horizontal shard count >> partitions. Use key-range splitting to increase parallelism beyond Kafka partition count.- Worker-local buffering to aggregate writes and use bulk writes to backend.- Backpressure via rate limiting per shard; prioritize hot keys.

Incremental checkpointing & fault tolerance:- Checkpoints contain last processed offset per partition and output artifact references. Store in S3 + durable metadata DB (DynamoDB / Spanner).- On failure, restart shard from last checkpoint; ensure exactly-once or idempotent sinks (use write-idempotency keys, upserts, or versioned records).

Snapshot bootstrapping details:- For initial state, export materialized view or upstream DB consistent snapshot with corresponding Kafka offsets (use transactions or CDC cutover).- For large snapshots, support parallel snapshot upload (range-based exports), and progressive loading by workers.

Cost & downtime mitigation:- Cost limits: spot/ preemptible workers for non-critical shards; rate-limit throughput; tiered replay (high-priority recent window first).- Minimize user-visible downtime by: - Build view in parallel in separate namespace and swap atomically. - Serve read requests with hybrid strategy: continue serving from old view + merge-on-read from in-progress deltas for changed keys (read-through cache or shadow reads).- Prioritize recent windows to make view "near-current" quickly.

Operational considerations:- Monitoring: per-shard metrics, lag, checkpoint frequency, cost estimates.- Safety: dry-run mode, canary shards, throttling.- Trade-offs: more shards => faster but higher cost; snapshot freshness vs replay size.

This design supports petabyte-scale replays with bounded downtime, incremental recovery, and cost controls by combining snapshot bootstrapping, fine-grained sharding, robust checkpointing, and orchestrated parallel rebuilds.

Unlock Full Question Bank

Get access to hundreds of Kafka, Message Queuing, and Event Sourcing interview questions and detailed answers.

Join thousands of developers preparing for their dream job.