Container and Kubernetes Security Questions

Security for containerized applications and Kubernetes platforms across the full lifecycle: secure image creation and supply chain, image scanning and vulnerability management, secure base images, image signing, runtime protection and intrusion detection, container isolation and least privilege at the container level, secrets management, pod security policies and admission controllers, network policies and microsegmentation, role based access control for cluster access, cluster hardening and configuration management, secure cluster bootstrapping and upgrades, and compliance considerations and audit logging for container environments. Candidates should be able to discuss tooling, threat models specific to cloud native workloads, and operational practices for preventing and responding to container and orchestration security incidents.

MediumTechnical

78 practiced

You notice multiple newly deployed pods showing high CPU and excessive outbound connections shortly after a deploy. Outline an incident response play-by-play to determine if this is a misconfiguration or a malicious cryptomining/command-and-control activity. Include commands and data sources you would consult.

Sample Answer

Situation: Multiple newly deployed pods show sudden high CPU and many outbound connections.

Play-by-play response — triage, contain, investigate, remediate, then recover.

1) Rapid detection & scope- Query cluster for affected pods and nodes:

bash

kubectl get pods --all-namespaces -o wide | grep -E "CrashLoopBackOff|Running"
kubectl top pods --all-namespaces | sort -k3 -nr | head
kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp

- Identify rollout that created them:

bash

kubectl rollout history deploy/<service> -n <ns>
kubectl get rs -n <ns> --show-labels

2) Contain (non-destructive first)- Throttle/limit egress at network policy or node firewall:

bash

# apply restrictive NetworkPolicy to namespace
kubectl apply -f deny-egress.yaml -n <ns>

- Scale down or cordon node if necessary:

bash

kubectl scale deployment/<svc> --replicas=0 -n <ns>
kubectl cordon <node>

3) Live forensics on an affected pod- Exec and capture process/network view:

bash

kubectl exec -it pod -n <ns> -- ps auxww
kubectl exec -it pod -n <ns> -- ss -tunaep
kubectl exec -it pod -n <ns> -- lsof -i
kubectl exec -it pod -n <ns> -- cat /proc/1/cmdline
kubectl logs pod -n <ns> --since=1h

- Dump suspicious binary, hashes, timestamps:

bash

kubectl cp <ns>/pod:/path/to/binary ./binary
sha256sum binary
strings binary | head

4) Network and IOC analysis- Capture pcap on node or using privileged pod:

bash

tcpdump -nn -i any host <suspicious_ip> -w /tmp/pod_traffic.pcap
tshark -r /tmp/pod_traffic.pcap -q -z conv,ip

- DNS and external endpoints:

bash

kubectl exec -it pod -n <ns> -- cat /etc/resolv.conf
kubectl exec -it pod -n <ns> -- dig +short suspicious.domain
whois <ip>

- Check for known C2/miner indicators: domains, ports, stratum/p2p patterns.

5) Artifact & deployment inspection- Verify image provenance and image digest:

bash

kubectl get pod pod -o jsonpath='{.spec.containers[*].image}'
crictl inspecti <image-id>   # or query registry for digest
trivy image <image>

- Diff deployment YAML vs previous:

bash

kubectl get deploy <svc> -n <ns> -o yaml > current.yaml
git show <previous-commit>:deploy/<svc>.yaml

6) Logs & metrics correlation- Query Prometheus/Grafana for CPU, netbytes, pod creation time windows.- Check kube-apiserver audit logs for who created the deployment:

bash

# search audit logs for user/api call
grep -R "deploy/<svc>" /var/log/kube-apiserver-audit.log

7) Decide: misconfig vs compromise- Signs of misconfiguration: new image tag matches CI artifacts, start command expected, no unknown binaries, outbound to known service endpoints, recent config change in repo.- Signs of compromise: unknown binaries, obfuscated strings, many external connections to IPs with bad reputation, persistence mechanisms, container images not matching signed digests, unexpected RBAC/activity in audit logs.

8) Remediation & recovery- If compromise: isolate cluster, rotate credentials, remove malicious images, rollback to known-good revision, redeploy with enforced imagePolicy and PSP/OPA Gatekeeper, block discovered IPs at network level, perform full incident response with security team.- If misconfig: rollback, fix resource limits, egress rules, CI pipeline.

9) Post-incident- Preserve evidence (pcaps, disk images, logs), timeline, root cause, and actions.- Implement prevention: image signing, egress network policies, runtime policies (Falco), scanning (Trivy), stricter RBAC, circuit-breaker alerts for unusual outbound rates.

This sequence balances rapid containment (egress/scale down) with forensic data collection (ps/ss/tcpdump, logs, image digests, audit logs) so you can determine misconfiguration vs malicious activity and remediate safely.

HardTechnical

75 practiced

Compare the security benefits and operational costs of using a service mesh (e.g., Istio) to enforce mTLS and policies versus relying on Kubernetes NetworkPolicies and mTLS at application level. When is a service mesh justified for security?

Sample Answer

High-level summary: a service mesh (Istio) centralizes and automates per-service mTLS, identity, and L7 policy enforcement, plus observability and traffic control. Kubernetes NetworkPolicies + app-level mTLS is simpler and lower-cost operationally but provides coarser, L3/L4 enforcement and places certification/rotation burden on each app.

Security benefits of a service mesh:- Automatic, mutual TLS between workloads with short-lived certificates and centralized rotation -> stronger identity, reduces developer error.- Fine-grained L7 policies (path, header, method) and per-service RBAC for zero-trust east-west controls.- Centralized authN/authZ, telemetry and auditing (who called whom, denied requests) for incident investigation.- Consistent policy across heterogeneous runtimes (legacy apps, VMs, multiple languages).

Operational costs and risks of a service mesh:- Complexity: control plane, sidecars per pod, upgrade surface area, nontrivial configuration (CRDs, gateways).- Performance: CPU/memory overhead per sidecar, added latency; needs benchmarking (e.g., Istio sidecar ~50–200ms p95 depending on workload).- Failure modes: misconfigured policies can cause widespread outage; more things to monitor and secure.- Operational load: managing mesh configs, mTLS CA, Istio upgrades, RBAC rules, and training teams.- Observability noise: large volume of telemetry requiring storage/aggregation.

Kubernetes NetworkPolicies + app mTLS trade-offs:- Lower operational overhead; native primitives, simpler debugging.- NetworkPolicies enforce pod-to-pod L3/L4 isolation but cannot enforce L7 intent or per-path auth.- App-level mTLS gives end-to-end encryption but inconsistently implemented across services; certificate rotation and identity mapping are developer responsibilities.- Better for small clusters, performance-sensitive workloads, or teams unwilling to adopt sidecar model.

When a service mesh is justified for security (criteria):- Many microservices (dozens+) with frequent service-to-service calls where consistent identity, L7 policies, and telemetry are needed.- Need for centralized certificate management, short-lived certs, and automated rotation to meet compliance.- Requirement for fine-grained L7 access controls, rate-limiting, or request-level telemetry for debugging/security audits.- Multi-cluster, hybrid, or multi-tenant environments where uniform policy enforcement and identity federation matter.- Teams can commit to operational investment: SRE+platform ownership to run, secure, and observe the mesh.

Practical recommendation:- Start with NetworkPolicies + standardized app mTLS for early stages. Evaluate gaps: if you need centralized identity, L7 policy, or consistent telemetry at scale, pilot a service mesh on non-critical services, measure latency/CPU, build automated rollout/rollback, and limit blast radius via namespaces or mesh-per-team before cluster-wide adoption.

HardTechnical

98 practiced

Integrate SLOs and error budgets with security operations: propose a policy that allows emergency patching that temporarily consumes error budget, explain how to trigger such an intervention, who approves it, and how to report and compensate for the consumed budget afterward.

Sample Answer

Situation: We needed a way to rapidly patch critical security vulnerabilities without undermining our SLO-driven reliability governance. The risk: emergency patches may degrade availability for short windows and consume error budget.

Task: Define a policy that permits emergency security interventions that may temporarily consume error budget, with clear triggers, approvers, controls, and post-action compensation and reporting.

Action:- Policy summary: Emergency Security Patching Exception (ESPE). ESPE permits immediate patches that may exceed SLO-derived change budgets when a verified security threat meets trigger criteria (see below). Any ESPE invocation temporarily marks the service as “under security intervention” and records expected availability impact and rollback plan.- Triggers (must be met): 1. Vulnerability scored ≥ 7.0 CVSSv3 OR confirmed active exploit affecting our production artifact. 2. Threat assessment by Security on-call confirming exploitability within 24–72 hours. 3. No non-disruptive mitigations available or mitigation window unacceptable.- Who approves: 1. Security on-call performs initial triage and proposes patch plan. 2. SRE on-call (or SRE manager after-hours) approves the operational plan (risk, rollback, monitoring). 3. For changes expected to consume >25% of the current error budget or impact >5% availability for >1 hour, require asynchronous approval from Product SLA owner (email/Slack + “approve” button in change tool). If not reachable, SRE manager can approve with post-facto review.- Controls and execution: - Use an emergency change ticket in the change management tool with required fields: CVE, CVSS, expected impact, rollback steps, monitoring dashboard links, and a timestamped approval chain. - Run change behind feature-flag or phased rollout (canary) wherever possible. - Automated safety nets: run automated health checks, circuit-breakers, and automated rollback on error-rate/latency thresholds tied to SLO alarms.- Reporting & compensation: - Immediate: change ticket updates and a Slack incident channel; dashboards display real-time SLO impact and consumed error budget. - Post-action (within 72 hours): SRE + Security produce a short report: root cause, justification, actual SLO impact (minutes of SLO lost / percent error budget consumed), rollback outcome, lessons, and remediation to prevent reoccurrence. - Compensation plan: to replenish consumed error budget, the service will: 1. Enact a release freeze for non-critical feature rollouts equal to the percentage of error budget consumed (min 24 hours). 2. Allocate 1–2 sprints of engineering focus on reliability fixes proportional to impact (e.g., minor consumption → bugfixes; major consumption → capacity or automation work). 3. Adjust sprint priorities: Product agrees to defer feature work equal to consumed budget impact until budget is restored. - Governance: ESPE incidents are reviewed in the next SLO burn-down and monthly reliability review; repeated ESPEs trigger an elevated remediation plan (technical debt, architecture changes).- Metrics and transparency: - Track count of ESPEs, mean time to patch, actual vs. predicted availability impact, and time to restore error budget. - Publish a short public postmortem for customers if availability SLO breached materially.

Result: This policy balances urgency of security response with SRE governance: it enables fast, auditable interventions while limiting unchecked SLO erosion through approvals, automation safeguards, explicit compensation, and transparent post-incident review.

EasyTechnical

90 practiced

Describe the purpose of Kubernetes RBAC and Pod Security Standards (PSS). How do they differ in scope and enforcement, and how would you combine them to enforce least privilege and secure pod configuration in a cluster?

EasyTechnical

96 practiced

What is an SBOM (Software Bill of Materials) for a container image? Describe the key elements an SBOM should contain and explain how SRE teams can use SBOMs to reduce supply chain risk.

Unlock Full Question Bank

Get access to hundreds of Container and Kubernetes Security interview questions and detailed answers.

Join thousands of developers preparing for their dream job.