Cryptographic Key Management and Infrastructure Questions

Designing, implementing, and operating systems that manage cryptographic keys and associated cryptographic infrastructure across the full lifecycle of keys and certificates. This includes secure key generation using validated entropy sources and randomness validation, key hierarchies and key derivation strategies, master key protection, algorithm selection and algorithm agility planning, and key migration strategies. It covers secure storage options and protections such as hardware security modules, cloud key management services and key vaults, encrypted and sealed storage patterns, and practical deployment considerations for both on premise and cloud environments. Access control and authorization patterns such as role based access control, separation of duties, and least privilege enforcement are essential, along with automated provisioning, rotation, retirement, and deprovisioning workflows. Operational topics include secure key distribution to services and devices, secure archival and destruction procedures, key escrow and recovery mechanisms, backup and disaster recovery for key material, incident response and handling of compromised keys, and audit logging and monitoring of key operations. Public key infrastructure and certificate lifecycle management are included, covering trust models, certificate issuance and renewal, revocation mechanisms and online status checking, and integration with identity and access management systems. Candidates should also address testing and validation approaches, cryptographic module attestation and tamper resistance, threat modeling and key compromise drills, standards and compliance considerations including guidance from the National Institute of Standards and Technology and other frameworks, scaling and performance trade offs for enterprise and internet scale deployments, and the balance between operational convenience, availability, and cryptographic assurance.

EasyTechnical

40 practiced

Design role-based access control and separation-of-duties for a corporate key management system. Define at least four roles (e.g., key-admin, operator, auditor, developer), the minimum permissions each needs for key creation, use, rotation, archival, and destruction, and describe technical controls to enforce least privilege and approval workflows.

Sample Answer

**Overview**I would implement RBAC with separation-of-duties (SoD) to ensure least privilege for a corporate Key Management System (KMS). Define clear roles, map minimum permissions, and enforce via technical controls and approval workflows.

**Roles & Minimum Permissions**- Key-Admin - Create key metadata, set policies, assign roles - Initiate rotation and schedule archival - Cannot sign/encrypt data or destroy keys- Operator - Perform key rotation operations, view key usage metrics - Activate/deactivate keys - Cannot create keys, change policies, or destroy keys- Developer - Use keys for cryptographic operations (encrypt/decrypt/sign/verify) via scoped service identities - Read-only access to key metadata necessary for integration - No policy, rotation, archival, or destruction rights- Auditor - Read-only access to full key metadata, audit logs, and config snapshots - Cannot perform operational or administrative actions

**SoD Rules**- Key creation and destruction require two roles: Key-Admin + approvals from a separate Approver (could be senior Key-Admin or Security Manager)- Rotation must be initiated by Operator and approved by Key-Admin for sensitive keys- Developers never have destructive or policy-change privileges

**Technical Controls**- Enforce RBAC via KMS-native IAM + centralized IAM (e.g., cloud IAM + LDAP/SSO)- Use attribute-based access (ABAC) for scoping by environment, application, and sensitivity tag- Require cryptographic operations through service accounts with short-lived credentials (OIDC, workload identity)- Hardware Security Modules (HSM) enforce separation of key roles (admin/operator vs crypto user)- Immutable, append-only audit logs (WORM) shipped to SIEM for auditor access- Multi-factor and just-in-time (JIT) elevation for privileged tasks with time-bound access

**Approval Workflows**- Integrate workflow engine (e.g., Jira/CAMUNDA or cloud KMS approval) requiring: request → approver 1 (Key-Admin) → approver 2 (Security Manager) for creation/destruction- Store approvals as signed artifacts linking to key UUIDs; automated policy engine enforces block until signatures present- Require MFA + approval TTL; record all actions with signed audit entries

**Enforcement & Monitoring**- Policy-as-code for RBAC rules in CI/CD; automated tests validate SoD constraints- Periodic access reviews, automated alerts for anomalous key usage, and quarterly SoD compliance reports- Automated key lifecycle policies (retain/auto-archive) to reduce human intervention

This design enforces least privilege, prevents single-person destructive actions, and provides an auditable, automated approval path suitable for enterprise KMS.

MediumTechnical

35 practiced

Define a secure backup and disaster-recovery plan for master keys that protect data encryption keys. Specify RPO/RTO targets, storage protections (encryption of backups and key-wrapping), geographic distribution, access controls for retrieval, and test procedures to validate recovery in a safe way.

Sample Answer

**Situation & objective (brief)** Design a secure backup & disaster-recovery plan for master keys (KMKs) that protect data encryption keys (DEKs), preserving confidentiality, integrity, availability and providing safe, auditable recovery.

**RPO / RTO targets** - RPO: <= 15 minutes for active-use KMs (nearline replication); 24 hours for archival key sets. - RTO: < 1 hour for critical services (failover HSMs); < 8 hours for full-site recovery. Targets chosen to match business SLAs and minimize data re-encryption work.

**Storage protections** - Store KMKs only inside certified HSMs (FIPS 140-2/3). Export backups as wrapped key blobs (envelope encryption): DEKs wrapped by KMK inside HSM, backup KMK wrapped with a backup KEK held in a separate HSM. - Backups encrypted at rest with separate KEKs and use authenticated encryption (AES-GCM). - Use immutable/append-only storage (WORM) and retention policies to defend against ransomware.

**Geographic distribution & separation** - Maintain at least 3 copies: primary HSM cluster, secondary region HSM (cross-region replication), and an offline key-escrow in an air-gapped facility (or secure cloud KMS dedicated account). - Geographical separation across different AZs/regions and different legal jurisdictions where required.

**Access controls for retrieval** - Use M-of-N threshold scheme (e.g., Shamir + HSM-backed unwrapping) combined with multi-party approval workflows. - Require strong MFA, hardware tokens (YubiKey/PIV), signed retrieval request, and break-glass escalation with time-limited tokens. - Enforce least privilege via IAM roles, time-bound privileged access, and just-in-time (JIT) access with session recording. - All retrievals require dual control and are logged to immutable audit systems and SIEM.

**Test procedures (safe validation)** - Maintain a separate test environment and test key hierarchy mirroring production. - Quarterly table-top DR exercises and semi-annual live recovery drills using non-production keys to validate processes end-to-end: restore backup blobs into test HSMs, perform unwrap/DEK recovery, decrypt sample data. - Validate M-of-N workflows, approvals, and logging. Perform negative tests (missing quorum, revoked creds) to ensure correct denials. - After each test, produce a signed lessons-learned report and remediate gaps.

**Governance & monitoring** - Policy for key rotation, revocation, and secure destruction. Periodic audits, automated integrity checks (hashes), and continuous monitoring/alerting for anomalous access. - Keep playbooks, contact lists, and legal considerations updated.

This plan balances availability with strict cryptographic controls and auditable, multi-party recovery to reduce misuse risk while meeting RPO/RTO targets.

MediumSystem Design

33 practiced

Design an internal PKI to issue and manage TLS certificates for 100k microservices across multiple clusters and cloud providers. Address CA hierarchy, certificate templates, automated enrollment/renewal (ACME or internal protocols), revocation at scale, trust distribution, and monitoring of certificate health.

Sample Answer

**Clarify goals & constraints**- Issue/manage TLS for ~100k microservices across clouds; low-latency issuance, automated renewals, cross-cluster trust, high-availability, minimize blast radius.

**CA hierarchy**- Root CA (offline, long-lived, e.g., 10–20y) -> Multiple regional/subordinate Intermediate CAs (cloud/cluster scoped, 1–3y) -> Short-lived leaf issuing CAs per environment (dev/stage/prod) or per service mesh (90 days).- Benefits: offline root reduces compromise risk; intermediates limit blast radius and map to trust boundaries.

**Certificate templates & lifetimes**- Define templates by use-case: mTLS (service-to-service), ingress TLS, admin. Enforce key usage, EKU, SAN rules, minimal lifetimes (30–90 days), key algorithm (ECDSA P-256).- Automate profile validation via CSR policy engine.

**Enrollment & renewal**- Primary: internal ACME (RFC 8555) service endpoints deployed per region with centralized CA back-end. Use ACME for automated issuance and short-lived certs.- Fallback: mTLS-based internal enrollment (SCEP-like) for bootstrapping nodes that can't reach ACME.- Integrate with service mesh sidecar (e.g., SPIRE/SPIFFE) for identity issuance and rotation.

**Revocation at scale**- Prefer short lifetimes + automated rotation to reduce reliance on revocation.- Implement OCSP stapling at proxies/ingress and an OCSP responder cluster with caching and CDN fronting for scale.- Maintain signed CRLs for rare cases; publish differential CRLs and use delta or OCSP for real-time checks.- Use certificate transparency-like internal logging (append-only audit) and automated detection of anomalous issuance.

**Trust distribution**- Distribute intermediate CA certs and trust bundles via secure configuration management (immutable images, cloud secret managers) and mesh discovery. Use SPIFFE trust bundles for workload APIs.- Implement automated bundle rotation with overlap period.

**Monitoring & telemetry**- Central certificate inventory service with discovery (k8s API, cloud metadata, sidecar reports) storing expiry, issuer, SANs.- Alerts: 30/14/7/1 days before expiry; circuit-breaker alerts for OCSP failures, high issuance error rates, anomalous revocations.- Audit logs, issuance metrics, SLOs for issuance latency, OCSP response times, and periodic pentest/CA compromise drills.

**Trade-offs**- Short lifetimes reduce revocation needs but increase issuance load—mitigate with ACME scale-out and caching.- Offline root increases security but complicates rotation—automate subordinate re-signing workflows.

I would implement a pilot: one cloud region + service mesh integration, validate ACME scale, OCSP caching, and monitoring before full rollout.

MediumTechnical

34 practiced

Write a Terraform configuration snippet that provisions an AWS KMS symmetric key with key rotation enabled, a dedicated admin role with restricted permissions, and a grant allowing a Lambda function to encrypt/decrypt without exposing key material. Include the relevant IAM policy and explain how this enforces separation of duties.

Sample Answer

**Approach (brief)**Provide a Terraform snippet that creates a KMS symmetric key with rotation, an admin IAM role with narrow management permissions, and a grant that allows a Lambda function's role to use the key for Encrypt/Decrypt without exposing key material. Explain how IAM + KMS grants enforce separation of duties.

**Terraform snippet**

hcl

provider "aws" { region = "us-east-1" }

resource "aws_kms_key" "data_key" {
  description             = "Symmetric key for application data; rotation enabled"
  deletion_window_in_days = 30
  enable_key_rotation     = true
  key_usage               = "ENCRYPT_DECRYPT"
  customer_master_key_spec = "SYMMETRIC_DEFAULT"
  tags = { Owner = "security-team" }
}

# Dedicated admin role: can manage key metadata and policies but NOT use for encrypt/decrypt
resource "aws_iam_role" "kms_admin" {
  name = "kms-admin-role"
  assume_role_policy = data.aws_iam_policy_document.kms_admin_assume.json
}

data "aws_iam_policy_document" "kms_admin_assume" {
  statement { effect = "Allow" principals { type = "AWS" identifiers = ["arn:aws:iam::123456789012:role/SecurityTeam"] } actions = ["sts:AssumeRole"] }
}

resource "aws_iam_policy" "kms_admin_policy" {
  name = "kms-admin-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid = "ManageKeyMetadata"
        Effect = "Allow"
        Action = [
          "kms:DescribeKey",
          "kms:GetKeyPolicy",
          "kms:PutKeyPolicy",
          "kms:CreateAlias",
          "kms:DeleteAlias",
          "kms:EnableKeyRotation",
          "kms:DisableKeyRotation"
        ]
        Resource = aws_kms_key.data_key.arn
      },
      {
        Sid = "DenyUseOfKey"
        Effect = "Deny"
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:Sign",
          "kms:Verify"
        ]
        Resource = aws_kms_key.data_key.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "attach_admin_policy" {
  role       = aws_iam_role.kms_admin.name
  policy_arn = aws_iam_policy.kms_admin_policy.arn
}

# Lambda role (example) that will use the key
resource "aws_iam_role" "lambda_role" {
  name = "lambda-data-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

data "aws_iam_policy_document" "lambda_assume" {
  statement { effect = "Allow" principals { type = "Service" identifiers = ["lambda.amazonaws.com"] } actions = ["sts:AssumeRole"] }
}

# Attach basic lambda execution policy (logs) omitted for brevity

# KMS grant to allow Lambda role to use key for cryptographic operations without exposing key material
resource "aws_kms_grant" "lambda_use" {
  name            = "lambda-encrypt-decrypt-grant"
  key_id          = aws_kms_key.data_key.key_id
  grantee_principal = aws_iam_role.lambda_role.arn
  operations      = ["Encrypt", "Decrypt", "ReEncryptFrom", "GenerateDataKey", "GenerateDataKeyWithoutPlaintext"]
  # Retain grant only while lambda role exists
  retire_on_delete = true
}

**Why this enforces Separation of Duties**- Admin role has management privileges (Describe, PutKeyPolicy, rotation) but explicit Deny for cryptographic actions prevents admins from decrypting data — prevents key misuse by ops/security admins.- Lambda role cannot change key policy or rotation state (no management actions) but has a KMS grant that permits only cryptographic operations. Grants are evaluated by KMS independently of IAM policy, enabling fine-grained runtime permissions without exposing key material or key material export.- KMS never exposes plaintext key material; GenerateDataKey returns plaintext only when permitted and by design can be audited. Combining IAM deny for admins and grants for services enforces least privilege and clear separation between key custodians and data consumers.

**Operational notes**- Enable CloudTrail logging for kms:* events and monitor grants/retirements.- Use key policy to explicitly allow AWS account root and restrict PutKeyPolicy to the admin role as needed.

MediumTechnical

66 practiced

Write a Python script (boto3) that rotates an AES-256 data encryption key wrapped by an AWS KMS CMK. The script should: 1) generate a new data key, 2) re-encrypt (re-wrap) existing envelope keys in batches, 3) atomically switch pointers (e.g., database metadata) to the new key version, and 4) log actions for auditing. Provide pseudocode or runnable code and discuss rollback and idempotency.

Sample Answer

**Approach (brief)**- Use envelope encryption: KMS CMK creates plaintext data key and ciphertext data key (encrypted under CMK). For rotation: generate new data key, re-wrap stored ciphertext keys (decrypt then encrypt with new CMK or use KMS ReEncrypt), update metadata atomically (DB transaction / versioned pointer), log every step.

**Runnable Python (boto3) — batch rewrap using ReEncrypt**

python

import boto3, logging, uuid
from botocore.exceptions import ClientError

kms = boto3.client('kms')
logger = logging.getLogger(); logging.basicConfig(level=logging.INFO)

def rotate_keys(cmk_new_id, envelope_key_records, batch_size=100):
    """
    envelope_key_records: iterable of dicts {'id':..., 'ciphertext_key': b64}
    """
    tx_id = str(uuid.uuid4())
    logger.info("START rotate %s -> %s", tx_id, cmk_new_id)
    for i in range(0, len(envelope_key_records), batch_size):
        batch = envelope_key_records[i:i+batch_size]
        rewrapped = []
        for rec in batch:
            try:
                resp = kms.re_encrypt(
                    CiphertextBlob=rec['ciphertext_key'],
                    DestinationKeyId=cmk_new_id
                )
                new_ct = resp['CiphertextBlob']
                rewrapped.append((rec['id'], new_ct))
                logger.info("Rewrapped id=%s", rec['id'])
            except ClientError as e:
                logger.error("Rewrap failed id=%s: %s", rec['id'], e)
                raise
        # atomic metadata update (pseudo DB transaction)
        try:
            with db_transaction() as t:
                for id_, ct in rewrapped:
                    t.update("envelope_keys", id_, {"ciphertext_key": ct, "key_version": cmk_new_id})
                t.commit()
            logger.info("Committed batch %d-%d", i, i+len(batch)-1)
        except Exception as e:
            logger.error("DB commit failed, aborting: %s", e)
            raise
    logger.info("COMPLETE rotate %s", tx_id)

**Key concepts & reasoning**- Use KMS ReEncrypt to avoid exposing plaintext keys; this lets KMS decrypt under source CMK and re-encrypt under destination CMK inside KMS.- Batch processing limits memory and reduces blast radius.- DB transaction per batch ensures atomic pointer swap for that batch.

**Rollback & Idempotency**- Idempotency: include key_version field and check before rewrapping; skip records already marked with cmk_new_id.- Rollback: if a batch commit fails, records already committed remain using new CMK; maintain audit log with tx_id so a compensating job can rewrap remaining records or rewrap committed ones back to prior CMK if necessary (requires storing previous ciphertext or version history).- Safe pattern: write new ciphertext to a new column and atomically flip pointer; keep old ciphertext until verification and retention period pass.

**Logging & Audit**- Log tx_id, record ids, timestamps, KMS response metadata, and DB commits. Preserve logs in immutable storage (CloudWatch Logs with retention + S3/Glacier) for forensic traceability.

**Edge cases**- CMK permissions, cross-account CMKs (use Grants), KMS limits (throttling), partial failures — implement retries with exponential backoff.

Unlock Full Question Bank

Get access to hundreds of Cryptographic Key Management and Infrastructure interview questions and detailed answers.

Join thousands of developers preparing for their dream job.