**Context & goal**As a Technical PM I’d define a clear API contract and schema-evolution policy so producers and consumers using Avro/Protobuf over Kafka can evolve safely with predictable migrations and SLAs.**API contract (summary)**- Transport: Kafka topics per logical event.- Schema: Avro or Protobuf stored in Schema Registry. Each event has a canonical subject: <namespace>.<eventName>-value.- Contract fields: name, version, owner, lifecycle (deprecated/removed), required consumers, SLAs for changes (e.g., 2-week deprecation notice), test harness link.**Compatibility modes (definitions)**- Backward: new schema can read data produced with older schema (consumers using new code can read old events).- Forward: old consumers can read new data (new producers are compatible with old consumers).- Full: both backward and forward (strongest).**Additive vs breaking changes**- Additive (safe): adding optional field with default, adding enum values (Avro: with default), adding new message. Allowed under backward compatibility.- Breaking: removing field, changing required -> optional vice versa without defaults, renaming, changing field type (e.g., int -> string) — require migration.**Schema Registry usage**- Enforce chosen compatibility mode per subject.- Validate on schema push; block breaking changes.- Store schema metadata, owners, and description.- CI step: register schema in feature branch, run compatibility check, create migration ticket.Example Avro additive change:json
{
"type":"record","name":"OrderCreated","fields":[
{"name":"orderId","type":"string"},
{"name":"amount","type":"double"},
{"name":"currency","type":["null","string"],"default":"USD"}
]
}
**Migration plan for breaking change (change field type: int -> string)**1. Stakeholders: notify consumers + owners; create migration playbook and timeline.2. Dual-write path: - Phase A: Producer writes both old and new fields (keep old int field as legacy_field and add new string_field). Update Avro to include both (backward-compatible).3. Consumer updates: - Consumers update to handle string_field if present; deploy and verify on canary (10% of traffic).4. Data migration: - Run streaming converter job (Kafka Streams/Flink) that reads old events and produces normalized events with string_field for historical topics or into a new topic.5. Switch producer: after all consumers in canary/majority support new field, stop writing legacy_field; set compatibility to backward to allow removal.6. Cleanup: after 30 days and metrics OK, remove legacy_field and set subject to full compatibility if desired.**Monitoring**- Metrics: consumer error rates, schema registry rejection rate, consumer lag, deserialization exceptions, end-to-end event counts.- Dashboards + alerts: spike in deserialization errors -> page on-call.- Tracing: correlate producer schema version to consumer failures.- Canary logs: observe canary consumer health for 24–72 hours.**Rollback**- If canary fails: revert producer to previous schema (registry supports previous versions), switch traffic back via feature flag, use consumers’ fallback to legacy_field.- If data corruption: pause producers via orchestrated feature-flag and run corrective converter to reprocess events from checkpoint.As TPM I’d own timelines, cross-team comms, runbooks, and ensure Schema Registry policies are enforced by CI to prevent accidental breaking changes.