Cryptographic Implementation Security Questions

Requirements:

• Encrypt customer data at rest and in transit; support multi-service storage, low-latency access, auditability, key rotation, and emergency key revocation/recovery.

High-level approach:

• Use envelope encryption: data encrypted with Data Encryption Keys (DEKs); DEKs encrypted by Key Encryption Keys (KEKs) managed by KMS. TLS with mutual auth for in-transit.

Envelope encryption pattern:

• Each object/blob/DB row gets its own DEK (or per-file/segment). DEKs generated with secure RNG (AES-GCM 256). Store ciphertext + encrypted DEK (DEK_C) and DEK metadata (key id, algorithm, nonce).
• On access: service requests DEK unwrap from KMS (decrypt DEK_C) then uses DEK in memory to decrypt payload; zeroize DEK after use.

KMS key hierarchy & HSM:

• Root Master Key (RMK) stored in FIPS 140-2/3 HSM (offline or tightly controlled). RMK used only to wrap/unwrap intermediate keys or to bootstrap KMS. Daily operations use Cluster KEKs held in HSM-backed KMS instances.
• KEKs: KMS-managed asymmetric or AES keys inside HSMs; DEKs never stored in HSM, only wrapped/unwrapped to minimize HSM ops and latency.

Key rotation policy:

• DEKs: rotate by re-encrypting new writes with new DEKs; lazy re-encryption for existing data, prioritize high-value items.
• KEKs: scheduled automated rotation (e.g., 90 days) via KMS: rotate by creating new KEK and re-wrapping DEKs (can be batched). Maintain previous KEKs for unwrap until all DEK rewraps complete.
• RMK: long-lived, rotated rarely (years) with strict procedures (multi-party approval) and key-ceremony.

Access controls & auditing:

• Principle of least privilege with IAM roles for services; KMS grants limited-purpose wrap/unwrap permissions, with conditions (time window, IP, service identity).
• Use mutual TLS + mTLS client certs and short-lived credentials (STS) to call KMS.
• Audit every KMS operation (who, when, key id, purpose) to an append-only log (WORM) stored separately and protected.

Recovery & revocation:

• Key revocation: mark KEK as revoked in KMS; prevent further unwraps. For emergency, have a "decryption-only" recovery KEK split via Shamir Secret Sharing across trustees for emergency unwrapping (offline mechanism). Maintain encrypted key backups sealed to RMK and stored in secure backup locations.
• Recovery process: documented playbook with multi-person authorization, offline HSM procedures for RMK use, and replay of audit logs to validate actions.
• Compromise handling: if a KEK suspected compromised, revoke, create replacement KEK, rewrap DEKs (or rotate RMK if RMK compromised). For data encrypted under compromised keys, re-encrypt ciphertexts after secure unwrap via recovery process.

Operational best practices:

• Use AEAD (AES-GCM) for integrity, include associated data (object id, version) in encryption.
• Keep DEKs in memory only; use secure enclaves or process memory protections where possible.
• Limit HSM operations to key wrapping/unwrapping; cache unwrapped DEKs in secure caches with TTL to reduce latency.
• Test rotation and recovery with drills; automate rewrap jobs and monitor metrics (failed unwraps, latency, key usage).
  Trade-offs:
• HSM and strict RMK procedures increase cost/operational complexity but greatly reduce key compromise risk. Lazy re-encryption reduces downtime at cost of longer exposure window for old ciphertexts.

Layer‑7 load balancers (e.g., Envoy, HAProxy, NGINX) terminate TLS by acting as the TLS endpoint: they present a certificate to clients, perform the TLS handshake, decrypt HTTP(s) traffic, and then forward requests to backends. For SNI-based routing the LB reads the TLS ClientHello’s SNI field (server_name) and selects the proper certificate and virtual host. Two common modes:

• SNI routing with TLS termination: LB selects cert based on SNI, completes handshake, then routes HTTP to backends.
• SNI passthrough: LB inspects TCP SNI and forwards the raw TLS stream to backend (no decryption).

Security vs performance trade-offs:

• Termination at LB (decrypt at edge): Pros — enables HTTP routing, WAF, detailed logging, compression and L7 features; faster CPU offload if LB has hardware/optimized TLS. Cons — plaintext inside network unless re-encrypted; increases attack surface at LB and centralizes key material.
• End-to-end TLS (passthrough or re‑encrypt to backend): Pros — backend confidentiality preserved, smaller trust boundary. Cons — limits L7 features, higher backend CPU for TLS, harder to inspect/block threats.

Preserving strong guarantees:

1. Backend TLS verification (re‑encrypt): Configure LB to establish TLS to backend using mutual verification or server cert validation. LB holds client‑facing private keys separately from backend keys.
2. Mutual TLS (mTLS): Both LB and backend present certificates and verify each other (CA pinned). This prevents MITM inside network and ensures only authorized LBs reach backends.
3. Short‑lived certs / mTLS with automanaged PKI (e.g., SPIRE/Cert‑Manager) to rotate keys.
4. Harden LB: restrict admin access, use hardware security modules (HSM) or cloud KMS for private keys, enable strict cipher suites and TLS 1.2+/1.3 only.
5. Network controls: use private subnets, mutual auth in service mesh, and certificate revocation/CRL/OCSP.

Example patterns:

• Edge TLS terminate + LB→backend TLS with server cert validation (common balance of features + security).
• Passthrough or mTLS for highest confidentiality.

Operational notes: monitor CPU on LB/backends, ensure health checks over TLS, log SNI for routing audits, and test cert rotation/rollback.