Applied Cryptography and Key Management Questions

High-level goal
Design a resilient PKI/key-management service that scales to 10M constrained IoT endpoints with intermittent connectivity and mixed hardware trust.

Requirements & constraints

• Strong device identity, offline-friendly auth, low-power/latency tolerant, survivable revocation, OTA-secure updates, minimal TCB on device.

Architecture (overview)

• Hierarchical PKI: Root CA (air-gapped, offline), several intermediate Enrollment CAs (ECAs) per region, Operational CAs (OCAs) scoped by fleet/function. Central KMS for operator keys; device-facing services (Enrollment Service, Provisioning Gateway, Revocation Service, OTA Signing Service).

Provisioning & bootstrap/root-of-trust

• Factory: burn unique device ID + manufacturer-installed default credential (ECC keypair) into device. If TPM/secure element present, generate and store keys in hardware; otherwise use software-protected keystore with secure boot measurement and monotonic counter. Root-of-trust = offline Root CA certificate anchored in device at manufacture; alternatively use manufacturer-issued device certificate tied to device serial + immutable public key.

Secure provisioning flow

1. Device presents manufacturer credential to Enrollment Service via authenticated one-shot handshake (mutual TLS with CSR or ECDSA-signed enrollment request).
2. Enrollment Service validates via ECA and issues device certificate from an Operational CA with short lifetime (e.g., 90 days) and scoped capabilities. Use EST or SCEP variants adapted for intermittent devices (store-and-forward via Provisioning Gateway).

Key storage choices

• Prefer TPM/SE/TEE for private keys and attestation.
• Where absent: use hardware-backed key derivation (unique per-device root KDF seeded from factory secret), encrypt private key with device-specific KEK stored in secure boot area, protect with obfuscation + HSM-wrapped symmetric keys in cloud.

Rotation & lifetimes

• Short-lived leaf certificates (30–90 days) with automated re-enrollment; CA/intermediate keys rotated less often and signed by Root CA offline.
• Key rotation on device triggered by expiry, compromise signal, or OTA command; use graceful overlap: new cert issued before old expires.

Revocation strategy

• For intermittent devices, CRLs are large and impractical. Use combination:
  • Short lifetimes to minimize window of exposure.
  • Revocation stream / push via provisioning gateway + delta updates (compact bloom-filter based revocation lists) cached on device.
  • For online validation, provide OCSP stapling from gateways or OCSP-over-HTTP with soft-fail policy adjusted per context.
  • For high-risk revocations, push immediate device-level kill via gateway and mark in cloud KMS to refuse auth.

OTA updates & code signing

• OTA images signed by OTA Signing Service (HSM-protected keys). Devices verify signature using pinned OTA public key rooted at factory. Use signed metadata including version, nonce, and rollback protection (monotonic counter or secure boot revocation of older images). Support delta updates and A/B partitions for safe rollback.

Scalability & availability

• Partition CAs by region/tenant; use stateless Enrollment Gateways behind autoscaling load balancers; store certificates and revocation state in highly available databases (sharded). Use HSM clusters for master key operations with geo-replication.

Monitoring, auditing & incident response

• Audit all issuance/revocation events in immutable logs (paper-trail + SIEM). Support rapid emergency revocation and re-issuance playbooks.

Trade-offs & rationale

• Short cert lifetimes reduce reliance on immediate revocation but increase enrollment load — mitigated by scalable gateways and batched re-enrollment. Hardware-backed keys are ideal; compensating controls (KDF, encrypted keystores) used where hardware lacks. Bloom/delta revocation balances bandwidth vs. false positives; combine with policy for critical services.

This design balances security, scalability, intermittent connectivity, and constrained devices while keeping the Root CA minimal and offline for maximum assurance.

Overview (short)
As a Security Architect I evaluate secrets stores by threat model, lifecycle, scale, and compliance: Kubernetes Secrets, Cloud-managed secret stores (AWS KMS/Secrets Manager, GCP Secret Manager, Azure Key Vault), HashiCorp Vault, and HSMs.

Kubernetes Secrets

• Brief: Namespace-scoped base64 objects; integrated with k8s RBAC.
• Ideal use-case: Non-critical app config in single-cluster dev/stage.
• Risk: Stored unencrypted on etcd by default; wide pod access if RBAC misconfigured.
• Mitigation: Enable encryption at rest for etcd, use RBAC + PodSecurityPolicies and CSI drivers to mount ephemeral secrets.

Cloud-managed Secret Stores

• Brief: Provider-native secrets + KMS; strong IAM integration and audit.
• Ideal use-case: Cloud-first services with compliance needs (PCI, HIPAA).
• Risk: Cloud account compromise or excessive IAM scope.
• Mitigation: Use least-privilege, MFA, dedicated service accounts, and logging+alerts.

HashiCorp Vault

• Brief: Centralized secret broker with dynamic secrets, lease/rotation.
• Ideal use-case: Multi-cloud/hybrid environments requiring dynamic DB creds.
• Risk: Single control-plane becomes critical availability/security dependency.
• Mitigation: HA deployment, sealed/unseal procedures (auto-unseal w/ KMS), strong ACL policies and monitoring.

Hardware Security Modules (HSMs)

• Brief: Tamper-resistant key storage for root keys and crypto ops.
• Ideal use-case: Storing root-of-trust keys for signing, CAs, payment systems.
• Risk: High cost and operational complexity; single vendor lock-in.
• Mitigation: Use HSM clusters with key replication, documented key ceremonies, and integrate with backup/DR and vendor-agnostic standards (PKCS#11).

Definition & core difference

• Symmetric: single shared secret key for encrypt/decrypt (AES). Fast, suitable for bulk data.
• Asymmetric: key pair (public/private) (RSA, ECDSA). Enables secure key exchange, authentication, digital signatures; slower.

Where to use in enterprise

• Database at-rest: symmetric AES with keys stored in KMS/HSM. Efficient for large volumes.
• TLS: hybrid — asymmetric (RSA/ECDHE) for handshake/authentication and symmetric (AES/GCM) for session traffic.
• Signing: asymmetric (ECDSA/RSA) for code signing, JWTs, certificate chains.

Key distribution & management

• Use PKI for issuing/trusting public keys; private keys protected in HSMs or KMS. Symmetric keys distributed via secure channels (wrapped with asymmetric keys) or derived using key exchange (ECDH). Rotate keys regularly, apply least privilege and audit.

Performance impacts

• Symmetric: low CPU, suitable for high throughput. Asymmetric: CPU-intensive; limit use to small payloads (handshakes, signatures).
• Offload crypto to HSMs or dedicated accelerators where latency/throughput matters.

When to use HSMs

• Protect root keys, CA private keys, payment keys (PCI), signing keys, KMS root of trust. Use HSMs for secure key generation, non-exportability, attestation, and regulatory compliance.

High-level approach
Use envelope encryption with a centralized KMS backed by HSMs. KMS holds Customer Master Keys (CMKs) used to wrap/unwrap short-lived Data Encryption Keys (DEKs). Applications and storage systems use DEKs for high-throughput symmetric crypto; CMKs are used only to encrypt/decrypt DEKs.

Key hierarchy

• Root HSM Key (offline / rarely used) — protects KMS master material (split‑keystore, multi‑party)
• CMKs (KMS-managed, HSM-backed, versioned) — asymmetric or AES-KW for wrapping
• DEKs (per-object or per-database-page, AES-256-GCM) — ephemeral, stored encrypted (wrapped) alongside data
• Optional per-tenant or per-environment CMKs to support isolation & multi-tenancy

HSM and KMS

• Use FIPS 140-2/3 validated HSMs (cloud HSM or on-prem) for CMKs.
• KMS provides wrap/unwrap, key rotation, RBAC, audit logging, and key policy enforcement.
• Enforce separation of duties: HSM admin ≠ key custodian ≠ app owner.

Rotation policies

• CMKs: rotate annually or when required by compliance; create new key version in KMS rather than deleting old keys.
• DEKs: short TTL (days or per-file). For databases/object stores use per-file or per-table DEKs; for backups use per-backup DEKs.
• Implement automatic key-version tagging; keep old CMK versions to decrypt existing wrapped DEKs.

Encrypting databases

• Use DEKs at page or column level depending on needs:
  • Transparent Data Encryption (TDE) pattern: database engine uses a DEK per DB instance, DEK wrapped by CMK.
  • For granular access control, encrypt sensitive columns with per-column DEKs stored in a secure vault.
• Cache unwrapped DEKs in memory with strict TTL and W^X protections; never persist plaintext DEKs.

Encrypting object stores

• Use per-object DEKs; store wrapped DEK in object metadata.
• For high-throughput, generate DEKs client-side, upload object + wrapped DEK. Use multipart uploads with DEK reuse for parts if safe.
• Enforce server-side encryption using KMS for objects created by cloud services.

Backups

• Each backup uses its own DEK, wrapped by current CMK version; store wrapped DEK with the backup manifest.
• Keep CMK history so old backups remain decryptable until policy allows secure rekeying.

Zero-downtime key rotation techniques

• Key-versioning: KMS supports multiple CMK versions; unwrap using key version listed in wrapped DEK metadata.
• Dual‑write (rolling rewrap): new writes wrap DEKs with new CMK version. Background rewrap job scans existing objects/backups and rewraps DEKs to new CMK; use throttling to avoid load spikes.
• Lazy re-encryption: on-read rewrap — when object is read, unwrap with old CMK, rewrap with new CMK, atomic replace metadata. Useful for huge datasets.
• Stateless services: cache wrapped-DEK and version only; when rotating, invalidate caches via pub/sub to avoid stale keys.
• For DBs: use DB engine features that support TDE key rotation (create new DEK, update engine pointer, rekey pages in background).
• Avoid decrypting bulk data in memory at once; perform streaming rewrap with checkpointing and idempotency.

Operational controls

• Strict IAM for KMS APIs, audit logs to SIEM, periodic key usage review.
• Emergency key disable/rotation playbook and backup CMK escrow where allowed.
• Test rotations in staging, measure latency and throughput, monitor error rates.

This strategy balances security (HSM-backed CMKs, short-lived DEKs, separation of duties) with operational needs (versioning, background rewrap, zero-downtime rotation) suitable for enterprise high-throughput environments.

Overview — core requirements

• Centralized, encrypted secrets store with strong crypto (AES-256, KDFs) and automatic key wrapping.
• Least privilege access model, MFA and short-lived credentials where possible.
• End-to-end auditability and tamper-evident logs retained to meet compliance SLAs.

Secrets storage

• Secrets encrypted at-rest and in-transit; separate crypto keys from stored secrets (key‑encrypting keys).
• Support versioning, metadata, and secure backup/DR procedures.

Rotation policies

• Automated rotation for credentials, API keys, certificates with configurable TTLs.
• Emergency rotation playbooks and dependency mapping to avoid outages.

Access controls & auditing

• Role-based and attribute-based access control (RBAC/ABAC); just-in-time access and approval workflows.
• Immutable audit logs, alerting on anomalous access patterns, periodic access reviews.

Integration points

• Native SDKs and secrets injection for applications, service meshes, and container runtimes.
• CI/CD integration: ephemeral secrets via pipeline agents, no hardcoded creds; secret scanning for repos.

Decision criteria: managed KMS vs self-hosted vault vs HSM

• Managed Cloud KMS: choose if you want low ops, cloud-native integration, and acceptable vendor trust — good for most app keys.
• Self-hosted Vault (e.g., HashiCorp): choose for richer secret lifecycle, dynamic secrets, complex policy needs, multi-cloud portability.
• HSMs: choose when you need FIPS 140-2/3 Level 3/4, regulatory key custody, or on-prem root-of-trust — higher cost/operational burden.

Always evaluate threat model, compliance requirements, availability/RTO, performance, and operational maturity before selecting.