API Versioning, Deprecation and Backward Compatibility Questions

Safe protobuf schema evolution has rules and operational practices. Key principles:

• Never reuse numeric tags once assigned; use reserved for removed tags.
• Adding fields with new tags is safe (back/forward compatible) if they are optional (proto3 defaults) or use wrapper types.
• Removing fields: mark tags as reserved to prevent reuse. Consumers should tolerate unknown fields.
• Enums: add values only; mark removed values as reserved.
• Avoid changing types or semantics of fields (breaking).

Example .proto showing safe changes:

syntax = "proto3";

package user.v1;

message User {
  int64 id = 1;
  string name = 2;
  // added later — safe: consumers that don't know it will ignore it
  string email = 3;
  // deprecated field; reserve tag to prevent reuse
  // deprecated: phone - use contact_numbers
  repeated string contact_numbers = 4;
  reserved 5; // previously used by phone
}

Go marshaling/unmarshaling (preserve unknown fields when proxying):

import (
  "google.golang.org/protobuf/encoding/protojson"
  "google.golang.org/protobuf/proto"
)

// Unmarshal bytes -> message, preserving unknown fields in proto.Message
var u userpb.User
if err := proto.Unmarshal(inBytes, &u); err != nil { ... }

// When accepting JSON from clients, discard unknown JSON fields to avoid failures
unmarshaler := protojson.UnmarshalOptions{DiscardUnknown: true}
if err := unmarshaler.Unmarshal(jsonBytes, &u); err != nil { ... }

Handling defaults and optional semantics:

• In proto3, primitive fields default to zero-values; use google.protobuf.* wrapper types (e.g., StringValue) if you need presence detection.
• Example: google.protobuf.StringValue email = 3; // lets server know if client set email vs default ""

Unknown fields and proxies:

• Protobuf binary encoding preserves unknown fields in the message binary; if your service proxies messages, avoid re-parsing+re-serializing into structures that drop unknowns. Use raw proto bytes or preserve UnknownFieldSet when possible.

Breaking-change prevention (CI + tooling):

• Use buf (or protoc-gen-validate) to enforce rules and run breaking-change checks in CI:

# .buf.yaml
version: v1
breaking:
  use:
    - FILE

Run: buf breaking --against 'git:origin/main#branch' to detect incompatible changes.

Operational practices:

• Run backward/forward compatibility checks in PR gate.
• Add integration tests: older client binary -> new server and vice-versa using recorded protobuf bytes.
• Deploy with canary/feature flags; monitor error rates and deserialize failures.
• Maintain changelog of proto revisions and reserved tags.

Handling real breaking needs:

• Introduce new message (v2) and run dual-write/read support; migrate consumers gradually.
• Use API versioning at RPC boundary (service v1/v2) for explicit cutover.

Summary checklist:

• Add fields with new tags; mark removed tags as reserved.
• Use wrapper types for presence.
• Preserve unknown fields when proxying.
• Use buf and CI checks to prevent accidental breaking changes.
• Canary deploy and monitor deserialization/semantic errors.