Data Protection and Encryption in Practice Questions

Situation & attacker capabilities
An attacker with filesystem+env access on a CI runner can read any files, environment variables, mounted secrets, credentials cached by the runner, and execute network calls from the runner. Concrete abuses in cloud context:

• Long‑lived IAM keys, service account JSONs, kubeconfigs, Docker registry creds, SSH keys, Terraform state files with secrets.
• Short‑lived tokens if still cached/refreshable (refresh token or metadata endpoint access).
• Ability to push malicious artifacts, backdoor images, exfiltrate secrets to external endpoints, pivot to cloud APIs.

Threat surface & priority assets

• Cloud provider credentials (IAM user keys, service account keys)
• OIDC tokens / refresh tokens
• Terraform state / encrypted secrets store keys
• Container images/artifacts and registry credentials
• Build artifacts and deploy pipelines (ability to trigger prod deploy)

Mitigations (preventive)

• Ephemeral runners: run jobs on one-time VMs/containers destroyed after job.
• Workspace isolation: per-job ephemeral workspaces, mount only required paths; avoid host volumes.
• Least privilege & short‑lived credentials: use OIDC or workload identity to mint tokens with minimal scopes and short TTLs; avoid long‑lived keys.
• No secret files: inject secrets via runtime ephemeral providers (secrets manager) or environment injection that avoids writing to disk; use kms envelope encryption for any persisted secrets.
• Secret scanning & artifact signing: scan repos/build outputs for leaked keys; sign artifacts and verify in deploy.
• Network egress controls: restrict runner egress to trusted endpoints, use private endpoints for cloud metadata with token binding.
• Immutable infrastructure: builds produce artifacts, deploy from artifact registry, not from runner filesystem.

Detection

• Log collection: forward runner system logs, CI audit logs to SIEM.
• Cloud audit trails: monitor CloudTrail/GCP Audit logs for unusual token use, CreateAccessKey, ServiceAccountKey creation, role assumption events, unusual API calls.
• Anomaly detection: spikes in artifact uploads, large outbound transfers, new images pushed, authentication from runner identity to unexpected regions.
• Host telemetry: EDR on runners to detect suspicious processes, reverse shells, or exfil attempts.
• Secret scanning alerting for commits/build outputs.

Remediation

1. Isolate: immediately remove the runner from CI fleet and block its network.
2. Revoke credentials: rotate/revoke any credentials that could be used (revoke sessions, delete service account keys, rotate IAM keys, revoke refresh tokens). For OIDC-issued tokens, revoke relying party trust if needed.
3. Rebuild & redeploy: destroy compromised runner image and rebuild CI pool from known-good base images; redeploy artifacts after verifying integrity and scanning for backdoors.
4. Forensics: capture disk image, memory (if available), collect logs and network captures for investigation; use retained audit logs to trace actions.
5. Rotate secrets: rotate secrets that could have been exposed (DB credentials, API keys, certificates) following prioritized inventory.
6. Improve controls: enforce ephemeral runners, OIDC, tighter IAM policies, egress controls, and automated secret scanning.

Why this approach
Combines least privilege, ephemeral credentials, and isolation to reduce blast radius, plus detection and quick credential revocation to limit attacker dwell time. As a Cloud Engineer I implement OIDC-based short-lived role assumption (AWS STS, GCP Workload Identity), automated runner lifecycle (immutable images), and centralized logging/alerting to detect and respond rapidly.

Clarify goals & constraints

• Provide short-lived credentials to workloads across AWS, Azure, GCP
• No long-lived static secrets; support cross-account/project access; minimize blast radius; auditability

High-level architecture

• Each workload uses platform-native identity to exchange for cloud credentials:
  • AWS: STS AssumeRoleWithWebIdentity (OIDC token) or AssumeRole via IRSA on EKS
  • Azure: Managed Identity endpoint -> Azure AD token -> Azure RBAC
  • GCP: Workload Identity Federation / metadata server -> access token for service account

Example flow (AWS):

1. Workload obtains OIDC JWT from K8s service account
2. Calls STS AssumeRoleWithWebIdentity with audience; STS returns short-lived creds
3. Use creds, cache locally with TTL < expiration

Trust relationships

• Define minimal-trust IAM/AD/Service Account bindings:
  • AWS role trust policy allows OIDC provider + specific subject claim (namespace/serviceaccount)
  • Azure assign role to principalId of managed identity scoped to resource/subscription
  • GCP create Workload Identity Pool, map external principal to GCP service account with limited IAM roles
• For cross-account/project: create roles/service accounts in target account and allow source identities to assume via explicit trust and condition checks (e.g., aws:SourceAccount, principal tags, GCP attribute conditions)

Token issuance & refresh

• Issue only short-lived tokens (minutes to hours). Platform defaults (AWS STS: up to 1 hour, Azure tokens ~1 hour, GCP tokens ~1 hour).
• Refresh strategy:
  • Client obtains token on start, refresh when remaining TTL < refresh_threshold (e.g., 20% of lifetime)
  • Use background refresh with jitter to avoid stampedes
  • On refresh failure, exponential backoff and degrade gracefully

Caching & local store

• Cache tokens in-memory or ephemeral filesystem with strict ACLs
• Cache keys indexed by (identity, scope/audience)
• TTL enforcement: never use token beyond expiry; proactively refresh
• Use single refresh leader per host/pod to prevent thundering herd (leader election or file/lock)

Security & best practices

• Principle of least privilege: narrow IAM roles, scope resource-level permissions
• Use conditional policies: source IP, resource tags, OIDC subject claims
• Shorten token lifetime for higher-risk flows; rotate mappings and audit keys
• Enforce mTLS between workloads and identity agent if using a sidecar
• Audit: enable CloudTrail, Azure AD logs, GCP audit logs for Assume/Token events
• Protect metadata endpoints: disable IMDSv1, require IMDSv2, restrict container access

Cross-account/project access

• Cross-account: create role in target account with trust: specific source identity (AWS ARN or OIDC sub) and require external ID or session tags
• Cross-project (GCP): use Workload Identity Federation with attribute mapping to target service account; grant minimal roles
• Central broker (optional): run a hardened token broker service that validates incoming native tokens, exchanges for target-cloud creds, logs requests, enforces policy. Keep broker minimal and highly audited.

This design yields secretless, short-lived credentials, centralized auditability, and safe cross-account access while minimizing blast radius.

Situation & goal
I’d treat a leaked production API key as an active compromise requiring immediate containment, safe rotation, and follow-up to prevent recurrence. Below is a concrete, cloud-focused step‑by‑step runbook.

1) Containment (first 0–15 min)

• Disable the exposed key immediately via cloud console/CLI (e.g., AWS IAM/GCP Service Account key disable) or mark as inactive in Secrets Manager.
• If the key is used by a service, switch traffic to a maintenance mode or feature flag to prevent unauthorized actions while minimizing user impact.

2) Validation

• Confirm which resources/roles the key could access (IAM policies, resource logs).
• Query audit logs (CloudTrail/Cloud Audit Logs) for anomalous activity since the leak timestamp.
• Capture forensics: save logs, paste snapshots, and metadata for investigation.

3) Rotation
(creating new creds safely)

• Create new credentials with least-privilege IAM role.
• Generate keys via Secrets Manager / KMS or create new service account keys in GCP; avoid long-lived keys where possible.
• Store new secret in vault (AWS Secrets Manager / Parameter Store or HashiCorp Vault) with strict access policies and encryption.

4) Rolling updates (deployment)

• Trigger CI/CD pipeline to inject new secret into services (use encrypted secrets, not hard-coded).
• Perform gradual rollout (canary -> 50% -> 100%) and validate functionality and telemetry.
• Use health checks and circuit breakers; be ready to rollback to a safe state that uses alternate safe credentials.

5) Secrets revocation & cleanup

• Revoke and delete the compromised key once new keys are active and verified.
• Revoke any associated tokens/sessions (JWTs, OAuth tokens) and rotate dependent credentials (DB creds, downstream API keys) if tied to same trust boundary.
• Rotate related keys (KMS data keys) only if there’s evidence of wider compromise.

6) Communication

• Immediately notify security, on‑call cloud engineer, product owner, and legal/compliance per incident policy.
• Provide concise status updates: actions taken, user impact, next steps.
• If required by law/regulation, prepare external disclosure with timeline and mitigation.

7) Post‑incident actions

• Conduct full postmortem within 48–72 hours: root cause, timeline, impact, and lessons learned.
• Implement preventive controls: automated secret scanning in CI, block paste sites via DLP or egress controls, enforce short-lived credentials, mandatory vault use, MFA for console/API, and alerting on public leaks.
• Add tests to CI to prevent commits with secrets, enforce branching policies, and schedule periodic key rotation and audits.
• Update runbook and run a tabletop exercise.

Why this works: immediate disable limits attacker window; vault-based rotation + CI/CD rolling deploys avoid downtime; revocation and forensic validation ensure scope is contained; communication and policy changes reduce future risk.

Brief comparison

• RBAC: assigns permissions to roles (role->principal->resource). Simple, auditable, good for static teams.
• ABAC: uses attributes (principal, resource, environment, tags) evaluated at request time. Flexible for dynamic, multi-tenant clouds.

Example RBAC policy (Kubernetes Role + RoleBinding for microservice "orders" in namespace "prod"):

# Role: read secrets only in namespace prod
kind: Role
metadata: { name: secret-reader, namespace: prod }
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get","list"]
---
kind: RoleBinding
metadata: { name: orders-secret-reader, namespace: prod }
subjects: [{ kind: ServiceAccount, name: orders-sa, namespace: prod }]
roleRef: { kind: Role, name: secret-reader, apiGroup: rbac.authorization.k8s.io }

Example ABAC policy (AWS IAM with conditions using tags/environments):

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Action":["secretsmanager:GetSecretValue"],
      "Resource":"arn:aws:secretsmanager:us-east-1:123456789012:secret:*",
      "Condition":{
        "StringEquals":{
          "aws:ResourceTag:env":"prod",
          "aws:PrincipalTag:service":"orders"
        }
      }
    }
  ]
}

Strengths & weaknesses for large, dynamic clouds

• RBAC strengths: easy to reason about, simple audits, well-supported by platforms (K8s, IAM). Weaknesses: role explosion in many namespaces/environments; brittle for ephemeral workloads.
• ABAC strengths: scale with attributes—teams, environments, or ephemeral services auto-adopt policies via tags; fine-grained context-aware controls. Weaknesses: relies on correct tagging and attribute propagation; policy complexity and harder to audit/debug.

Recommendation: use hybrid—RBAC for coarse boundaries and ABAC conditions (tags, environment, network) for dynamic, fine-grained control; enforce tagging via automation and guardrails.

Encryption at rest vs in transit

• At rest: data (including secrets) is encrypted while stored on disk or in a cloud service. Protects against storage compromise (e.g., stolen disks, snapshot access).
• In transit: data is encrypted while moving between systems (TLS, mTLS, VPN). Protects against network eavesdropping and MITM.

Envelope encryption with KMS (cloud engineer view)

• Process: generate a data encryption key (DEK) locally or by KMS; use DEK to encrypt the secret; encrypt (wrap) the DEK with a KMS-managed customer master key (CMK); store ciphertext(secret) + wrapped DEK.
• At read: request KMS to unwrap the DEK (requires IAM permissions), use DEK to decrypt secret.
• Example: AWS KMS CMK wraps DEKs; secrets stored in S3 or Secrets Manager hold wrapped DEKs.

Why rotate keys separately from secrets

• Rotating a secret (e.g., database password) changes credential value; rotating keys (CMKs/DEKs) re-writes encryption envelope without changing secret semantics.
• Key rotation limits blast radius if a key is compromised; older ciphertexts can be re-encrypted with new DEKs/CMKs.
• Operational reasons: different lifecycles, compliance requirements, and access controls. Implement automated rewrapping, audit KMS use, and ensure backward compatibility/retention policies.