Overview (goal)
Design a cross‑cloud KMS that supports BYOK, HSM-backed keys, automated rotation, cross‑region access and resilient failover while preserving strong audit and least‑privilege controls.
High‑level approach
- Use envelope encryption: protect data with a Data Encryption Key (DEK) and store only encrypted DEKs (ciphertext blobs) protected by a Key Encryption Key (KEK) in KMS. This minimizes KMS calls and eases cross‑cloud transfer.
- DEK generation/encryption lives with the application or a sidecar; KEK operations stay in HSM/KMS.
Centralized KMS vs Per‑cloud KMS (comparison)
- Centralized KMS (single tenant appliance or HSM cluster you control)
- Pros: single policy plane, uniform audit, easier BYOK control and key lifecycle management.
- Cons: network latency, cross‑cloud egress, single point of failure unless geo‑distributed; higher integration effort.
- Per‑cloud KMS (use native KMS per provider + BYOK imports)
- Pros: low latency, native integrations, regional redundancy, lower egress cost.
- Cons: multiple policy planes, inconsistent audit formats, more complex rotation/coordinated governance.
A pragmatic model: hybrid—central control plane for policy/audit and per‑cloud HSM/KMS for in‑region crypto. Use cross‑cloud key replication or wrap/unwrap APIs to synchronize KEKs.
BYOK & HSM details
- BYOK flow: generate key material in customer hardware or approved vault, export in supported wrapped format, import into target cloud KMS/HSM with attestation. Prefer HSM import APIs with FIPS 140‑2/3 attestation.
- For hardware‑backed keys: keep private key material in HSM (CloudHSM, Azure Dedicated HSM, external HSMs). Use a PKCS#11 or KMIP bridge when needed.
Key rotation & versioning
- Rotate KEKs on a schedule with automated re‑wrapping of active DEKs:
- New KEK = generate in HSM; re‑encrypt (re‑wrap) existing DEKs asynchronously.
- Keep old KEK versions for decryption until all ciphertexts move; support key version metadata and TTL.
- Use rolling rotation and maintain a “decrypt hierarchy” to avoid downtime.
Cross‑region access, replication & failover
- Replication:
- Do not replicate raw key material unless HSM permits and security/controls allow. Prefer HSM clustering or secure export/import with wrap keys.
- Replicate ciphertext blobs and coordinate DEK re‑wrapping per region.
- Failover:
- Active‑active: replicate KEK via HSM cluster across regions (supported by some vendors); apps use nearest KMS endpoint.
- Active‑passive: promote regional KMS if primary fails; use cached encrypted DEKs and fallback to secondary KEK unwrap.
- Ensure documented RTO/RPO and test failover runbooks.
Access control & least privilege
- Enforce IAM + resource policies per cloud: role separation for key administrators, operators, and services.
- Use short‑lived service identities (OIDC, STS) and per‑service DEK access only via approved KMS operations.
- Apply attribute/ABAC for dev/prod separation; require HSM attestation for sensitive ops.
Auditability & monitoring
- Centralize logs: forward CloudTrail, Azure Activity Logs, GCP audit logs to SIEM (e.g., Splunk, Sentinel). Normalize events: key usage, grant/revoke, import/export, rotation, unwrap/reencrypt.
- Alert on anomalous patterns: bulk exports, unexpected key policy changes, or unusual unwrap rates.
Operational considerations & tradeoffs
- Latency vs control: per‑cloud improves latency; centralized increases control. Hybrid is often best for enterprises.
- Regulatory: keep keys in specific geographic boundaries (do not replicate across legal boundaries if prohibited).
- Testing: regularly run key rotation, failover, and restore drills. Maintain key custodianship and automated key escrow for recovery.
This design uses envelope encryption for efficiency, HSMs for strong key protection, automated rotation and versioning to avoid downtime, per‑cloud KMS for performance, and a centralized governance/audit plane to maintain consistency and compliance.
