Safe protobuf schema evolution has rules and operational practices. Key principles:
- Never reuse numeric tags once assigned; use reserved for removed tags.
- Adding fields with new tags is safe (back/forward compatible) if they are optional (proto3 defaults) or use wrapper types.
- Removing fields: mark tags as reserved to prevent reuse. Consumers should tolerate unknown fields.
- Enums: add values only; mark removed values as reserved.
- Avoid changing types or semantics of fields (breaking).
Example .proto showing safe changes:
proto
syntax = "proto3";
package user.v1;
message User {
int64 id = 1;
string name = 2;
// added later — safe: consumers that don't know it will ignore it
string email = 3;
// deprecated field; reserve tag to prevent reuse
// deprecated: phone - use contact_numbers
repeated string contact_numbers = 4;
reserved 5; // previously used by phone
}
Go marshaling/unmarshaling (preserve unknown fields when proxying):
go
import (
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
)
// Unmarshal bytes -> message, preserving unknown fields in proto.Message
var u userpb.User
if err := proto.Unmarshal(inBytes, &u); err != nil { ... }
// When accepting JSON from clients, discard unknown JSON fields to avoid failures
unmarshaler := protojson.UnmarshalOptions{DiscardUnknown: true}
if err := unmarshaler.Unmarshal(jsonBytes, &u); err != nil { ... }
Handling defaults and optional semantics:
- In proto3, primitive fields default to zero-values; use google.protobuf.* wrapper types (e.g., StringValue) if you need presence detection.
- Example: google.protobuf.StringValue email = 3; // lets server know if client set email vs default ""
Unknown fields and proxies:
- Protobuf binary encoding preserves unknown fields in the message binary; if your service proxies messages, avoid re-parsing+re-serializing into structures that drop unknowns. Use raw proto bytes or preserve UnknownFieldSet when possible.
Breaking-change prevention (CI + tooling):
- Use buf (or protoc-gen-validate) to enforce rules and run breaking-change checks in CI:
yaml
# .buf.yaml
version: v1
breaking:
use:
- FILE
Run: buf breaking --against 'git:origin/main#branch' to detect incompatible changes.
Operational practices:
- Run backward/forward compatibility checks in PR gate.
- Add integration tests: older client binary -> new server and vice-versa using recorded protobuf bytes.
- Deploy with canary/feature flags; monitor error rates and deserialize failures.
- Maintain changelog of proto revisions and reserved tags.
Handling real breaking needs:
- Introduce new message (v2) and run dual-write/read support; migrate consumers gradually.
- Use API versioning at RPC boundary (service v1/v2) for explicit cutover.
Summary checklist:
- Add fields with new tags; mark removed tags as reserved.
- Use wrapper types for presence.
- Preserve unknown fields when proxying.
- Use buf and CI checks to prevent accidental breaking changes.
- Canary deploy and monitor deserialization/semantic errors.
