Container and Kubernetes Security Questions

Situation: Suspected container breakout (container process escaping to host) on a production node. Goal: contain attacker, preserve evidence, restore safe state, and remediate with minimal customer impact.

Immediate containment (first 0–15 min)

• Isolate node from cluster network and scheduler: cordon + drain (but do NOT evict suspicious pods until evidence collected).
  • kubectl cordon <node>; kubectl rollout pause; iptables/SG rule to cut external egress.
• Quarantine suspicious pods: snapshot runtimes but don’t restart/kill until captured.
• Remove node from LB/traffic pool to stop customer impact.

Evidence collection (preserve volatile data)

• Record timeline and C2: timestamps, alerts, who acted.
• Host-level:
  • dmesg > /tmp/dmesg.log; journalctl -k > /tmp/journal_kernel.log
  • ps auxf > /tmp/ps.log; netstat -tunap or ss -tunap
  • capture mounts: mount, ls -l /proc/*/fd
• Container runtime:
  • containerd/CRI logs: /var/log/containerd.log, /var/log/pods/*; docker/daemon.json if applicable
  • kubectl get pod -o yaml for suspicious pods
• Memory & image artifacts:
  • Use LiME or fmem to capture host memory: lime.ko -> host.mem
  • docker/containerd checkpoint or cri-tools to export container filesystem: crictl inspect --output json
• Hash and transfer: sha256 of logs and images; copy to secure forensic server (read-only).

Forensic analysis (offline / controlled)

• Triage memory image for injected kernel modules, rootkits, suspicious processes, open sockets; use Volatility/rekall.
• Analyze kernel logs for OOPS, new modules, unusual syscall patterns.
• Inspect container images and layers for modified binaries, suid files, cronjobs.
• Correlate network connections to external IPs, DNS queries, and time series from monitoring.

Remediation

• Treat node as compromised: rebuild from trusted image; do NOT re-image in-place.
• Rotate all credentials/secrets that could have been exposed: cloud keys, service account tokens, TLS certs.
• Patch kernel and container runtime; upgrade to latest stable; enable security features: kernel lockdown, seccomp, AppArmor, user namespaces, cgroups v2.
• Re-scan images, implement immutable registries and image signing (Notary/Cosign).
• Reintroduce node only after validation and hardening.

Communications

• Notify incident response, security, product owners, and on-call SRE lead with status, impact, and next steps.
• Post internal incident timeline updates every 30–60 minutes until resolved.
• Prepare customer-safe statement if outage/customer data affected; coordinate with legal/PR.

Audit & post-incident

• Document chain of custody for evidence, actions taken, commands run, and timestamps.
• Create incident report: root cause, scope, remediation, mitigations, and TTR.
• Implement lessons: tighten runtime policies, improve detection rules, automate memory/log collection on suspicious alerts, schedule tabletop exercises.

Why: Immediate isolation stops attacker movement; preserving volatile evidence enables root-cause and supports legal; rebuilding nodes prevents persistent compromise; rotating credentials removes stolen secrets; communication maintains coordination and trust.

Situation: Our CI/CD was breached and a malicious container image, signed with stolen signing keys, was pushed and may have been deployed across many Kubernetes clusters. Immediate goals: stop further trust in the compromised keys, detect which clusters/nodes pulled the malicious image, and remediate/rollback safely with minimal disruption.

Action — revoke trust and block the image

1. Revoke signing keys quickly:

   • Revoke keys in the key management system (KMS/HSM) and remove private key access.
   • Update any key lists (TUF metadata, Sigstore/fulcio revocation if used) and publish revocation statements.
   • Rotate signing keys: generate new keys in HSM/KMS, publish new public keys and update CI to use them.
   • Revoke any CI service account tokens/credentials that had access to signing keys.
     Reasoning: Immediate key revocation prevents further signing, rotation prevents future trust of compromised key.

2. Make the specific image untrusted:

   • In the registry, mark the image digest as revoked/untrusted; if registry supports immutable tags, delete the malicious digest and apply denylist policies.
   • Update attestation logs (e.g., Rekor) or Notary/TUF metadata to reflect revocation.
   • Update admission controls to reject images signed by the old key or images matching the revoked digest.

Identify which clusters/pods/nodes pulled the image

1. Query the container registry:

   • Use registry access logs to list pulls of the malicious digest (by digest or tag) and collect client IPs, timestamps, and auth identities.
   • Map registry client IPs/auths to clusters/nodes/accounts.

2. Use centralized observability:

   • Search centralized logs (ELK/CloudWatch/GCP Stackdriver) for imagePull events, kubelet/containerd logs, and kube-apiserver events referencing that image digest or tag.
   • Query cluster-level event APIs via automation: kubectl get pods --all-namespaces -o jsonpath='{..image}' | grep <digest>.
   • Example (across contexts):
     kubectl --context=CLUSTER-A get pods --all-namespaces -o json | jq -r '..|.image? // empty' | grep <digest>
   • Use metrics (registry pull metrics, Prometheus) to find clusters with elevated pulls for the digest.

3. Use runtime detection:

   • Run Falco/OSQuery on nodes to detect if the container process is running.
   • Check node filesystem/containerd image store for presence of the digest.

Remediate and roll back safely

1. Triage and scope:

   • Classify clusters as: Not pulled, Pulled but not running, Running.
   • Prioritize critical clusters/services and production workloads.

2. Containment:

   • For clusters where image is running: cordon affected nodes, isolate network egress for those pods, or apply NetworkPolicy to restrict their connectivity.
   • Scale down Deployments/ReplicaSets running the malicious image (kubectl scale --replicas=0) to stop activity without deleting metadata.

3. Rollback to known-good images:

   • Identify last-known-good digest from image manifests or release manifests.
   • Update Deployment manifests or use kubectl rollout undo if history is reliable.
   • Force redeploy with trusted image digest signed by the new key:
     kubectl set image deploy/myapp mycontainer=registry.example.com/myapp@sha256:<good-digest>
   • Use admission controller to enforce only images signed by new keys are allowed to start; apply mutation/validation webhook to verify signature before restarting.

4. Clean up image cache:

   • On affected nodes, remove the malicious image from container runtimes (ctr images rm / docker rmi) and restart kubelet/container runtime to ensure new pulls use good image.

5. Rotate credentials and secrets:

   • Rotate registry credentials, CI/CD service accounts, and any tokens that could be used to push images.
   • Revoke any compromised service accounts and audit RBAC.

6. Validate and harden:

   • Run vulnerability and behavioral scans on rolled-back workloads.
   • Confirm enforcement: attempt to deploy the revoked digest in a staging cluster to ensure admission rejects it.
   • Deploy monitoring rules (alerts for unsigned images, unexpected new digests, and registry pull spikes).

7. Communication & post-incident:

   • Notify stakeholders and compliance teams with scope and remediation plan.
   • Preserve artifacts: registry logs, node forensic data, images for analysis.
   • Conduct a postmortem, update runbooks: tighten key storage (HSM-only), shift to short-lived keys/CI ephemeral signing, enable reproducible builds, enable image provenance (Sigstore/cosign, supply chain attestations), and implement immutable infrastructure patterns.

Key trade-offs and reasoning:

• Immediate revocation and containment may cause brief downtime; prioritize critical services first and use rolling restarts where possible.
• Deleting images from registries is not sufficient — enforce admission/validation to prevent re-deployment using different credentials or reintroduced images.
• Centralized logging and registry auditing are critical — invest in observable telemetry to accelerate detection in future incidents.

Result: The approach stops further signing with compromised keys, identifies impacted clusters via registry and centralized logs, and safely rolls back or isolates affected workloads while restoring trust via rotated keys and strengthened admission controls.

Requirements & constraints:

• Dozens of customers on shared clusters, need isolation (security, performance), cost efficiency, per-tenant billing and operational boundaries, some tenants may require stronger isolation.

High-level strategy:

1. Namespaces per tenant (or per tenant+environment)

• Create one namespace per tenant (prod/dev) to scope resources, RBAC, quotas, and network policies.
• Use consistent naming and labels (tenant=acme, env=prod).

2. Resource isolation: ResourceQuotas + LimitRanges + Requests

• Enforce ResourceQuota per namespace (cpu/memory/pods/services) to cap tenant usage.
• Require CPU/memory requests and limits via LimitRange; use admission controller to reject pods that lack requests.
• Use Vertical Pod Autoscaler or HPA for autoscaling within quotas.
• For noisy-neighbor mitigation, use Guaranteed QoS for critical pods (set requests==limits) and eviction thresholds.

3. Node pools & scheduling

• Create node pools (instance types) per critical characteristics: general-purpose, high-memory, GPU, and a "private" pool for high-security tenants.
• Use taints on node pools + tenant-specific tolerations to control placement and reduce cross-tenant interference.
• Use topologySpreadConstraints and pod anti-affinity for distributing tenant workloads.
• Consider Fenced nodes (dedicated nodes) for tenants with compliance needs (effectively per-tenant nodes).

4. Network isolation: NetworkPolicies + CNI

• Default deny ingress/egress per namespace; require explicit NetworkPolicies for allowed flows.
• Implement layered policies: namespace-level policies to block cross-tenant traffic, plus podSelector policies for intra-namespace rules.
• Use a CNI that enforces network policies robustly (Calico, Cilium). Consider service mesh with mTLS for stronger identity-based isolation if needed.

5. RBAC boundaries

• Grant tenant teams access only to their namespace via Roles and RoleBindings (avoid ClusterRole for tenant admins).
• Use groups (e.g., tenants:acme-admins) from identity provider (OIDC/AD) and bind to namespace Roles.
• Control who can create ClusterRoles/ClusterRoleBindings via admin approval.
• Use automation (GitOps) for role management and auditing.

6. Control plane & multi-tenancy trade-offs

• For most tenants: shared control plane with strict namespace-level controls.
• For high-security tenants: offer dedicated clusters or dedicated control planes (managed Kubernetes offer or self-hosted) to eliminate shared etcd/control-plane risk.
• Use admission controllers (OPA/Gatekeeper) to enforce policies (image registries, seccomp, disallow hostPath, require Scan/Signing).

7. Observability, billing, auditing

• Tag/label resources with tenant metadata; aggregate metrics by namespace label for chargeback.
• Collect per-namespace CPU/memory/persistent volume usage via Metrics Server, Prometheus, or cloud billing APIs.
• Centralized logging with tenant labels and RBAC-restricted views (e.g., Kibana spaces).
• Enable and retain audit logs; route sensitive audit streams for compliance tenants to separate storage.

8. Operational practices & SLOs

• Define per-tenant SLOs and capacity guardrails; enforce via quotas and alerts.
• Run periodic chaos/scale tests to validate isolation.
• Automate onboarding/offboarding (namespace creation, quotas, network policies, billing tags).

Trade-offs & risks:

• Shared clusters are cost-efficient but increase blast radius; use node pools and strict policies to reduce risk.
• Dedicated clusters increase cost but simplify compliance.
• Rely on correct request/limit settings; enforce via admission to prevent evictions/noisy neighbors.

This combination (namespaces + quotas + network policies + RBAC + node-pools + admission enforcement + selective dedicated clusters) gives layered isolation balancing security, performance, and cost.

High-level answer / recommendation

• Use a hybrid approach: push fast, deterministic, developer-facing checks into CI (fail-fast) and keep a smaller, high-confidence runtime admission layer for last-mile enforcement and drift detection. CI reduces developer friction by catching problems early; admission protects the cluster from bypasses and supply-chain changes.

Why each layer matters

• CI/blocking builds: good for static checks (linting, container image build-time scanning, SBOM generation, policy-as-code tests, signature creation). Fast feedback loop => higher dev productivity because issues are caught before merge.
• Admission/runtime (mutating/validating webhooks, OPA Gatekeeper, Kyverno): enforces runtime invariants (node selectors, PSP/PSA, network policies), verifies image digests/signatures, and prevents manual/rogue changes that bypass CI.

Handling false positives

• Triage pipeline: start with non-blocking/dry-run mode in both CI and admission. Collect violations with metadata (who/what/commit) and measure false-positive rate.
• Progressive enforcement: promote specific rules from monitor -> warn -> block based on metrics (false positive < threshold, adoption rate).
• Allow exceptions: temporary allowlists with automatic expiry and issuing tickets for justification; require attestations for permanent exceptions to be audited.
• Provide fast, actionable feedback: include remediation steps and links to docs in failure messages; add pre-commit and IDE checks to reduce noisy failures.

Balancing developer productivity

• Put fast, cheap checks in pre-commit/CI (linters, container SBOM, cosign signing) so most failures never reach cluster-level gates.
• Keep admission layer minimal and targeted to high-risk policies (image signature verification, immutable digest enforcement, secrets in env vars).
• Provide self-service: tools to locally run the same policies (policy CLI, preflight) and clear documentation + chat ops integration to rapidly resolve blocked builds or pods.
• Measure impact: track mean time to fix, build queue time, and dev satisfaction; tune thresholds.

Guaranteeing runtime equals CI-approved artifacts

• Immutable artifacts: CI must produce and publish artifacts tagged by digest; deployment manifests must reference image digests, not mutable tags.
• Signed provenance & attestation: sign images (cosign/Sigstore) and attach provenance from CI (build ID, commit SHA). Admission validates signatures and expected provenance.
• GitOps reconciliation: make Git the source of truth; controllers (ArgoCD/Flux) reconcile manifests and revert manual changes. Admission denies direct kubectl apply for protected namespaces or requires annotations indicating Git-managed.
• Runtime attestations and periodic verification: continuous scanners (runtime scanners, kube-bench, policy audits) and periodic "recon" jobs that compare deployed pod specs and image digests against CI attestations; alert and quarantine mismatches.
• Immutable infra and RBAC: restrict who can push changes, require pipelines to run with limited service accounts, and audit all mutating events.

Operational practices & metrics

• Start with dry-run and collect: violation count, false-positive rate, blocked builds/pods, MTTR to remediation.
• Enforce SLAs for rule promotion and exception expiration.
• Maintain an escalation playbook and runbooks for developers to resolve blocks quickly.
• Regularly review policies with product/security/dev representatives.

Example flow

1. CI builds image, runs tests, generates SBOM, signs image and produces attestation with build-id and git commit.
2. CI pushes image by digest, creates deployment manifest in Git referencing digest, and opens PR.
3. Admission controller validates pod image signature and attestation; rejects pods without valid CI attestation or with mutable tags.
4. GitOps reconciler ensures any manual changes are reverted; reconciliation failure alerts SREs.

Trade-offs summary

• Blocking in CI: better developer experience, earlier fixes, but can be bypassed if someone pushes images manually or edits live manifests.
• Admission blocking: final protection but risk of developer friction and false positives harming deploy velocity.
• Hybrid + instrumentation minimizes friction while providing robust runtime guarantees: guardrails in CI, authoritative checks in admission, immutable artifacts, signatures, and GitOps reconciliation to ensure runtime state matches approved CI outputs.

Approach (brief): apply least privilege by designing narrow Roles (namespaced) / ClusterRoles only when needed, avoid wildcard verbs/resources, bind to specific ServiceAccounts, and enforce via OPA Gatekeeper admission policies that (a) block privilege escalation patterns and (b) audit existing bindings before enforcing. Rollout incrementally with discovery, audit, remediation, and gradual enforcement.

Example Role and safe binding:

# namespaced role granting only read access to ConfigMaps
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: payments
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-cm-reader-binding
  namespace: payments
subjects:
- kind: ServiceAccount
  name: payments-sa
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io

Key design principles:

• Principle of least privilege: only required verbs/resources and minimal scope (namespace vs cluster).
• Prefer Roles over ClusterRoles unless cross-namespace or cluster-wide required.
• Use dedicated ServiceAccounts per component, avoid reusing default SA.
• Set automountServiceAccountToken: false for pods that don’t need it.
• Use namespacing and naming conventions (svc-<app>-sa) to track ownership.

Gatekeeper / OPA policies (examples):

• Deny Role/ClusterRole rules with "*" verbs/resources.
• Deny RoleBinding/ClusterRoleBinding that bind ClusterRole "cluster-admin" or wildcard subjects.
• Prevent creation of RoleBindings that elevate namespace SAs to cluster-wide roles.
• Ensure automountServiceAccountToken is false for pods without RBAC needs.

Sample Rego constraint (deny wildcard permissions & cluster-admin bindings):

package rbac.restrictions

deny[msg] {
  input.review.kind.kind == "Role" 
  some i
  r := input.review.object.rules[i]
  any_wildcard(r)
  msg := sprintf("Role %v contains wildcard verbs/resources", [input.review.object.metadata.name])
}

any_wildcard(r) {
  r.verbs[_] == "*"
}
any_wildcard(r) {
  r.resources[_] == "*"
}

deny[msg] {
  input.review.kind.kind == "ClusterRoleBinding"
  input.review.object.roleRef.name == "cluster-admin"
  msg := "Binding to cluster-admin is forbidden"
}

Gatekeeper setup: create ConstraintTemplate wrapping above Rego and a Constraint that targets Role/ClusterRoleBinding. Start with enforcementAction: dryrun (audit) then deny.

Prevent privilege escalation patterns to catch:

• RoleBindings that reference ClusterRoles (binding namespace SA to cluster role).
• ClusterRoleBindings to service accounts in namespaces (unless approved).
• Roles with escalation verbs (bind, impersonate, create clusterroles).
• Use policy to deny verbs: "bind", "escalate", "impersonate" on sensitive resources.

Rollout plan (minimal disruption):

1. Discovery (1–2 weeks)

   • Audit current RBAC: list Roles/ClusterRoles, RoleBindings/ClusterRoleBindings, and ServiceAccounts. Map which SAs are used by pods.
   • Run risk scoring: identify high-risk bindings (cluster-admin, wildcards, impersonate/bind verbs).
   • Use kube-rbac-proxy or custom scripts to produce dependency maps.

2. Audit phase with Gatekeeper dry-run (2 weeks)

   • Deploy Gatekeeper with constraints in dryrun to collect violations.
   • Notify owners (via labels/annotations recorded in audit) and create tickets for remediation.

3. Remediation & least-privilege design (2–4 weeks, iterative)

   • For each app, define required actions and create narrow Roles.
   • Replace broad bindings: create new Roles/RoleBindings, update pod specs to use dedicated SAs.
   • Use feature flags / rollout windows for teams; allow exceptions with timeboxed approval via an exception CRD.

4. Enforcement (gradual)

   • Flip constraints from dryrun to deny for low-risk/green areas (non-prod namespaces first).
   • Monitor audit logs, kube-apiserver denies, and app behavior.
   • Progress to production namespaces after verification.

5. Harden defaults & automation

   • Enforce automountServiceAccountToken: false via PodSecurityPolicy/PodSecurity admission (or Gatekeeper).
   • Add CI checks: prevent merging manifests that create wildcard RBAC or cluster-admin bindings.
   • Periodic audits and alerting for new violations.

Rollback & safety:

• Start in dry-run; use short windows for converting to deny and have runbooks to revert constraints.
• Keep emergency 'rbac-exception' process: a short-lived ClusterRoleBinding allowed by Ops with strict TTL and audit.
• Use feature flags and health probes: if denials cause failures, temporarily allow the binding and create remediation ticket.

Monitoring & SLOs:

• Track number of denied admissions, number of wildcard/broad bindings, time-to-remediate violations.
• Alert if a new cluster-admin binding is created.
• Incorporate in incident runbooks to check RBAC denials when pods fail to start or API calls fail.

Why this works:

• Narrow roles and scoped bindings reduce blast radius.
• Gatekeeper provides centralized, auditable policy enforcement and a safe dry-run path.
• Incremental rollout with discovery and owner involvement minimizes disruption while raising cluster security posture.