Kubernetes Storage and Persistence Questions

Covers the Kubernetes storage model and how to manage persistent state for applications running on Kubernetes. Topics include ephemeral volumes versus persistent volumes, persistent volume claims, storage classes, dynamic and static provisioning, container storage interface drivers, volume access modes and volume modes, reclaim policies, and topology aware provisioning. Candidates should know how to provision storage for stateful workloads such as databases and message queues, manage data persistence across pod restarts, and handle resizing, mounting options, and file system considerations. Also includes cloud and network storage options such as local node volumes, cloud provider block storage like Amazon Elastic Block Store or Google Persistent Disk, network file systems, and specialized storage backends. Assessment should cover backup and restore strategies, snapshots, disaster recovery planning, data migration, consistency and replication approaches, operator based solutions for stateful services, security considerations such as encryption and access control, and common troubleshooting and performance tuning techniques for Kubernetes storage.

EasyTechnical

54 practiced

When should you prefer running a stateful application under a StatefulSet versus a Deployment? Cover features of StatefulSets relevant to persistent storage such as stable network IDs, ordinal indexing, and PVC templates, and give a short example (e.g., single-node Redis vs replicated database).

EasyTechnical

53 practiced

Explain the difference between a snapshot and a backup in Kubernetes storage contexts. Cover Consistency (application vs crash-consistent), typical tools (VolumeSnapshots vs backup operators), and a simple scenario where snapshots are insufficient and a full backup is required.

Sample Answer

Snapshot vs Backup — short, practical SRE view:

- Definition: - Snapshot: Point-in-time copy of a volume (storage-level). Fast, incremental, usually stored by the CSI driver or cloud provider. - Backup: Logical or physical copy of application data + metadata (volumes, manifests, secrets, CRDs) stored outside the cluster or in a dedicated backup store; often includes orchestration to quiesce apps and verify restores.

- Consistency: - Snapshots are typically crash-consistent: they capture the block state at that instant. If applications had in-flight transactions, you may get partial writes and need replay/repair on restore. - Backups can be application-consistent: backup operators coordinate with apps (e.g., pre-freeze, flush, use DB transactions, WAL archiving) to ensure transactional integrity and correct restore order.

- Typical tools: - VolumeSnapshots (Kubernetes CRD + CSI): create snapshots of PVCs; good for quick clones, fast rollbacks, test/dev restores. - Backup operators: Velero, Kasten, Stash, Cohesity — capture resources, PVC data (often using snapshots or copy), and cluster metadata; provide schedule, retention, cross-cluster restore, and application hooks.

- When snapshots are insufficient (simple scenario): - You run a stateful app using a primary DB (Postgres) with streaming replicas plus application state in multiple PVCs and Kubernetes secrets/configmaps. A node failure corrupts the primary’s in-memory transactions. VolumeSnapshots capture block state per-PVC but not: - The sequence of WAL segments stored off-volume, - Kubernetes objects (Secrets, PV/PVC bindings, CRDs), - Cluster-level config required for restore. - Restoring only snapshots may leave the DB inconsistent or missing WAL needed to recover to a consistent point. A full backup that includes WALs, coordinated DB checkpoint (application-consistent snapshot), and cluster metadata is required to guarantee correct recovery.

Practical SRE takeaway: use snapshots for fast recoveries, clones, and short-term rollback; use backup operators for durable, application-consistent, policy-driven protection and cross-cluster restores. Combine both: snapshots for speed, backups for completeness and compliance.

HardTechnical

49 practiced

Design an encryption-at-rest strategy for Kubernetes PVs using provider-managed keys (e.g., AWS KMS) and provider-agnostic application-level encryption. Explain key rotation impacts, backup encryption, and how SREs should verify encrypted data integrity after restores.

Sample Answer

Requirements:- Ensure PV data is encrypted at rest using cloud provider keys (AWS KMS) and also by application-level, provider-agnostic encryption so data remains secure if storage or provider access is compromised.- Support key rotation without long downtime, backups encrypted, and clear SRE verification steps after restores.

Design:1) Storage-layer (provider-managed) encryption- Use cloud provider server-side encryption for volumes (EBS with SSE-KMS) plus the Kubernetes CSI driver configured to use a specific KMS CMK per environment. This provides transparent encryption of blocks and integrates with snapshot APIs.

2) Application-level (envelope) encryption — provider-agnostic- Apps perform envelope encryption: generate a random data encryption key (DEK) per object/volume, encrypt payload with a vetted AEAD cipher (AES-GCM or XChaCha20-Poly1305), store ciphertext + metadata (DEK ID, nonce, version, HMAC/tag).- DEKs are wrapped (encrypted) by a key-encryption-key (KEK) managed in KMS (Encrypt API) before persisting alongside metadata. The wrapped DEK is provider-agnostic because apps can be modified to support another KMS or local HSM by changing KEK unwrap logic.

Key rotation impacts and strategy:- KMS key rotation (KEK rotation): rotation creates new CMK versions. Do not re-encrypt all data immediately. Instead: - On write: generate DEK, wrap with current KEK version, store KEK version ID in metadata. - On read: unwrap using KEK version indicated. KMS supports decrypt by previous keys if enabled (or use rewrap/rotate API). - Background rewrap job: SRE-run batch process that unwraps DEKs (using old KEK) and rewraps with new KEK for hot data to reduce future latency. - Full re-encryption (data re-encrypt with new DEKs) only for compliance windows; schedule and rollout with throttling to avoid I/O spikes.- Provider-managed rotation (SSE-KMS): rotating the CMK used by EBS/S3 changes only service-side; existing volumes remain decryptable if policy permits. Confirm provider behavior (AWS re-encrypts snapshots on copy if requested).

Backup encryption:- Ensure snapshot/backups are created with provider-side encryption enabled (e.g., EBS snapshots encrypted with KMS).- For application-level encryption, backups will contain ciphertext + wrapped DEKs — thus backups are safe even if provider KMS compromised (unless KEK unwrapped).- Backup tooling (Velero) must be configured with cloud credentials to use KMS for provider-side snapshot encryption and must not store plaintext DEKs. Store backup metadata (KEK version) and ensure backup manifests include integrity hashes.

SRE verification after restores:1) Automated restore pipeline:- Restore a test namespace/cluster in an isolated environment (nightly smoke restores).- Validate provider-side: verify volume/snapshot metadata shows correct KMS CMK and encryption flag.2) Application-level integrity checks:- Use stored AEAD tags/HMACs and checksums: after restore, attempt to decrypt sample files using KMS unwrap + DEK decrypt; verify AEAD tag matches.- Run application end-to-end smoke tests that exercise read/write of encrypted data.3) Canary rewrap/rotation test:- After a restore, run a rewrap operation for a subset and verify unwrap/rewrap succeeds with active KEK.4) Monitoring and alerts:- Instrument KMS API errors, unwrap failures, high latency, and rewrap-job failures. Alert on mismatches between metadata KEK versions and available KEKs.5) Audit logs:- Validate CloudTrail/KMS Audit for unwrap/encrypt operations during restore to detect unauthorized unwraps.6) Rollback plan:- If integrity checks fail, retain original snapshots, revoke access keys, and escalate.

Operational notes and trade-offs:- Envelope encryption adds CPU and complexity but decouples data confidentiality from provider.- Rewrap jobs must be rate-limited and monitored to avoid I/O/CPU storms.- Keep minimal IAM for KMS (least privilege) and rotate service principals.- Test restores regularly and include cryptographic integrity checks in CI to ensure recoverability.

HardTechnical

40 practiced

You need to build automation that detects low inode availability on nodes serving many small-file workloads (e.g., logs) and automatically quarantines or migrates offending pods. Outline the detection mechanism (metrics, thresholds), migration strategy, and safety checks to avoid cascading failures while remediating inode exhaustion.

Sample Answer

Requirements & goals:- Detect nodes approaching inode exhaustion from small-file workloads and remediate automatically without causing cascading disruption or violating SLOs.

Detection mechanism (metrics & thresholds):- Metrics to collect per node: - node_files_used, node_files_total (from node-exporter or kubelet /proc/sys/fs/inode-nr) - inode_utilization = node_files_used / node_files_total - per-pod filesystem usage stats (via cAdvisor or exec du in container) and file-creation rate (files/sec) - disk I/O and avail bytes to correlate- Thresholds (example, tunable by service): - WARN at 70% inode_utilization - ACTION at 85% inode_utilization - CRITICAL at 95% (manual ops)- Rate-based guard: only act if inode utilization trend > +5% over 5m to avoid flapping for transient spikes.

Detection implementation:- Use Prometheus rules + Alertmanager to emit alerts with labels: node, namespace, offending_pods (from recording rules + a small analyzer job).- A Kubernetes operator (or controller) subscribes to alerts and runs remediation logic.

Migration / quarantine strategy:1. Quarantine: Mark node with taint inode/low=true:NoSchedule and add annotation inode.remediation=time to avoid repeated actions.2. Identify offending pods: sort by highest per-pod inode usage and file churn rate; prefer non-critical, lower-importance pods (use pod priority, disruptionBudget, and service labels).3. Migrate: - Respect PodDisruptionBudgets (PDBs): only evict if PDB allows. - Evict pods via the eviction API (grace period configurable). For stateful workloads, prefer eviction to next available node; for dataful pods, skip and notify owner. - If eviction fails or insufficient capacity exists, scale up capacity (cluster autoscaler) or move to a node pool sized for many small files (with larger inode count).4. Post-eviction: re-check inode utilization; if stable <70% remove taint after a cooldown (e.g., 10m).

Safety checks to avoid cascading failures:- Global throttling: limit concurrent remediations per cluster (e.g., max 2 nodes or N% of nodes).- Respect PDBs and Pod priority: never evict critical system pods (kube-system with high priority).- Capacity check: before eviction, verify target nodes have >20% inode headroom and sufficient CPU/memory; otherwise trigger scale-up instead of mass migration.- Backoff & cooldown: exponential backoff per node for repeated actions; require sustained trend before re-acting.- Observability & human-in-loop: send a runbook + incident to on-call when action exceeds a threshold (e.g., evicted >10 pods or cluster autoscale triggered).- Dry-run mode and canary rollout: run in dry-run for new clusters; pilot in low-risk namespaces.

Example operator flow (pseudo):- On Action alert: - if concurrent_remediations >= limit -> queue - query per-pod inodes; filter by priority & PDB - simulate evictions, ensure target capacity pass - apply taint; evict best candidates one-by-one, monitoring inode change - if inode drops below safe threshold -> stop and remove taint

Operational notes & hardening:- Maintain a dedicated nodepool with XFS/ext4 tuned for high-inode counts for log-heavy workloads.- Educate devs: recommend log aggregation (e.g., sidecar to push to central logging) to reduce pod-local files.- Run chaos tests simulating rapid file creation to validate thresholds and scaling behavior.

MediumTechnical

46 practiced

Describe security best practices for Kubernetes storage: encrypt-at-rest options (cloud-provider vs filesystem), secrets and key management, RBAC and PodSecurity for volume mounts, and how to prevent unauthorized hostPath or block device access in a multi-tenant cluster.

Sample Answer

Brief summary: Secure Kubernetes storage requires defense in depth: encrypt data at rest, protect secrets with strong KMS integration, tighten RBAC and Pod-level policies for volume mounts, and enforce admission controls to stop hostPath/block device abuse in multi‑tenant clusters.

Encrypt-at-rest- Cloud-provider: use provider-managed disk encryption (AWS EBS + KMS, GCP CMEK, Azure Disk Encryption) for simplicity, durability, key rotation and auditing. Prefer CMEK/CSEK where you need customer-managed keys and separation of duties.- Filesystem-level: layer LUKS/FS-level encryption (dm-crypt) when you manage nodes or need additional control. Combine with cloud disk encryption for layered security.- Kubernetes-level: enable encryption providers for etcd (encryption at rest for Secret objects) and ensure etcd data disks are encrypted.

Secrets and key management- Don’t store long-term keys in plain Secrets. Use an external KMS/secret store (HashiCorp Vault, AWS KMS + Secrets Manager, GCP KMS/Secret Manager) and inject secrets at runtime via CSI Secrets Store Driver or Vault Injector.- Use envelope encryption: Kubernetes encrypts data with a DEK that is itself encrypted by a KMS-managed KEK.- Principal of least privilege for KMS access: grant IAM roles narrowly (nodes or service accounts) and rotate keys regularly. Enable KMS audit logs.

RBAC and PodSecurity for volume mounts- Use least-privilege RBAC: restrict who can create StorageClasses, PersistentVolumeClaims (PVCs), and CRDs that control volume mounts.- Limit who can create privileged Pods, use the PodSecurity admission (or OPA/Gatekeeper) to enforce PodSecurity Levels: restrict privileged, hostPath, block device, and allowPrivilegeEscalation.- Use Pod Security Policies (if still used) or Pod Security Admission and disallow hostPath, allow readOnly rootFilesystem, deny allowedCapabilities that permit mounting devices.

Prevent unauthorized hostPath or block device access (multi-tenant)- Disable hostPath by default via admission controller policies (PodSecurity, OPA/Gatekeeper) — explicitly allow only curated namespaces.- Prevent CSI driver abuse: restrict which StorageClasses / CSIDriver CRs can be used via RBAC and validating webhooks. Require StorageClasses be created by admins.- Node-level protections: enable NodeRestriction to avoid pods creating fake node identities. Use seccomp, AppArmor, and SELinux where possible.- Use runtime/container isolation: gVisor/Kata Containers for strong tenant isolation if untrusted tenants exist.- Audit and monitoring: enable kube-audit, monitor volume-related events, alert on creation of hostPath PVs or privileged pods.- Network and IAM isolation: ensure tenants cannot access cloud provider APIs for KMS or attach volumes unless allowed.

Operational best practices- Automate policy enforcement (Gatekeeper), CI checks for manifests, and periodic audits of StorageClasses/PVs/PVCs.- Test key rotation and backup/restore (etcd with encrypted backups).- Document incident playbooks for compromised keys or misconfigured volumes.

This approach balances cloud convenience (managed KMS/disk encryption) with cluster-level controls (PodSecurity, RBAC, admission webhooks) and node/runtime hardening to keep multi‑tenant storage safe.

Unlock Full Question Bank

Get access to hundreds of Kubernetes Storage and Persistence interview questions and detailed answers.

Join thousands of developers preparing for their dream job.