Container Orchestration and Kubernetes Operations Questions

This topic covers the design, deployment, operation, and scaling of containerized applications and Kubernetes clusters in production environments. Candidates should understand application level constructs such as pods, replica sets, deployments and controllers; rolling updates and canary and blue green deployment strategies; horizontal pod autoscaling and cluster autoscaling; resource requests and limits; scheduling, node and pod affinity and taints. It also includes service discovery, internal and external load balancing, ingress and traffic management, service mesh patterns, persistent storage including persistent volumes and storage classes, and storage provisioning. Candidates should demonstrate knowledge of container networking models, network policies, security and role based access control, secrets management, and observability including logging, metrics and distributed tracing for both cluster and application health. Operational responsibilities include cluster provisioning and upgrades, control plane and etcd considerations, high availability and multi zone topologies, multi cluster strategies, backup and disaster recovery, capacity planning, cost and reliability trade offs, managed versus self managed Kubernetes services, continuous integration and continuous deployment integration, operational runbooks, incident response, and debugging and troubleshooting approaches at production scale. Senior level candidates should be able to articulate cluster architecture and design trade offs, extensibility and automation strategies, maintenance and upgrade strategies, and long term operational governance.

MediumTechnical

73 practiced

Draft an upgrade plan to upgrade Kubernetes control plane and node versions in a multi-zone cluster with three control plane replicas across zones. Include steps for etcd snapshots, order of control plane and worker upgrades, draining, health checks, and commands or checks you'd run before and after each step to ensure quorum and minimal disruption.

Sample Answer

Requirements & constraints:- 3 control-plane replicas across 3 zones (must maintain quorum: etcd majority = 2)- Minimal disruption, rolling upgrade, full rollback plan, backups (etcd)- Using kubeadm-managed cluster (adjust if provider-managed)

High-level plan:1) Pre-checks and snapshot2) Upgrade control-plane replicas one-at-a-time (zone-by-zone)3) Upgrade worker nodes in batches with drains4) Post-upgrade validation and cleanup

Detailed steps with commands and checks:

Pre-checks & etcd snapshot- Verify cluster healthy:

bash

kubectl get nodes
kubectl get pods -n kube-system -l component=kube-apiserver
kubectl get cs   # if available; otherwise check component pods

- Take etcd snapshot on a control-plane host (use etcdctl v3; run on leader or any member with certs):

bash

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key snapshot save /var/backups/etcd-snap-$(date +%F_%T).db

- Verify snapshot:

bash

ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-snap-*.db

- Confirm etcd cluster health and quorum:

bash

ETCDCTL_API=3 etcdctl --endpoints=$(echo MEMBER_ENDPOINTS) endpoint status --write-out=table
ETCDCTL_API=3 etcdctl --endpoints=$(echo MEMBER_ENDPOINTS) endpoint health

Control-plane rolling upgrade (one replica at a time)For each control-plane node (order: non-leader first ideally; avoid rebooting >1 member):- Pick node A.- Cordon node (optional for control-plane pods managed by static pods):

bash

kubectl cordon <node-A>

- SSH to node, drain only workloads if any (ignore DaemonSets, but don't evict control-plane static pods):

bash

kubectl drain <node-A> --ignore-daemonsets --delete-local-data --force

- Upgrade kubeadm and kubelet to target version:

bash

# On node
apt-get update && apt-get install -y kubeadm=<VERSION>
kubeadm upgrade node
systemctl restart kubelet

- Upgrade control-plane components (if using kubeadm):

bash

kubeadm upgrade apply <VERSION>   # run from a control-plane node; but only perform for that node if using patch approach

(If control-plane is static pods: replace manifests under /etc/kubernetes/manifests or use kubeadm; follow distro-specific instructions.)- After restart, verify:

bash

kubectl get pods -n kube-system -o wide
kubectl get cs
ETCDCTL_API=3 etcdctl endpoint status --write-out=table
kubectl get nodes

Ensure etcd member rejoined and quorum maintained (at least 2/3 healthy).

Repeat for other control-plane nodes one-by-one, verifying quorum and API availability between each.

Worker node upgrade (batched to preserve capacity)- Define batch size (e.g., 25-33% nodes per zone).For each batch:- Cordon & drain node:

bash

kubectl cordon <worker>
kubectl drain <worker> --ignore-daemonsets --delete-local-data

- Upgrade kubeadm, kubelet, kubectl on node:

bash

apt-get install -y kubeadm=<VERSION>
kubeadm upgrade node
systemctl restart kubelet

- Uncordon after checks:

bash

kubectl uncordon <worker>
kubectl get pods -o wide --field-selector spec.nodeName=<worker>

- Health checks per node: - kubelet status: `systemctl status kubelet` - Node Ready: `kubectl get node <worker> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'` - Application pod readiness and service endpoints: `kubectl get pods -A -o wide`, `kubectl get endpoints`

Post-upgrade validations- All nodes Ready:

bash

kubectl get nodes

- API responsiveness and control-plane pods healthy:

bash

kubectl get pods -n kube-system
kubectl get cs

- etcd health & member list:

bash

ETCDCTL_API=3 etcdctl endpoint status --write-out=table
ETCDCTL_API=3 etcdctl member list

- Run a smoke test of applications, check metrics/alerts (Prometheus, Grafana), and run conformance e2e if available.

Rollback considerations- If etcd corrupted: restore snapshot on new cluster or on nodes:

bash

ETCDCTL_API=3 etcdctl snapshot restore /path/to/snap.db --data-dir=/var/lib/etcd-from-backup
# then adjust systemd unit or manifests to point to restored data-dir, start etcd

- If control-plane binary downgrade needed, follow kubeadm downgrade docs and don't mix versions that violate skew policy.

Key operational notes / checks- Maintain etcd quorum: never reboot more than one control-plane at a time.- Monitor API latency and error rates (alert if SLO thresholds exceeded).- Keep Kubernetes version skew rules in mind: kubelet can be at most +1 minor ahead of control plane.- Schedule maintenance in low-traffic windows and inform stakeholders.

This plan balances safety (snapshots, quorum checks), minimal disruption (single-member control-plane upgrades), and observability (explicit commands/health checks before and after each step).

MediumTechnical

59 practiced

Evaluate multi-tenancy approaches for Kubernetes: shared cluster with namespaces + RBAC + resource quotas, cluster-per-team, and virtual clusters (vCluster). Compare operational cost, security isolation, noisy neighbor effects, and upgrade complexity, and recommend when each approach is appropriate.

Sample Answer

Situation: We're evaluating three Kubernetes multi-tenancy models—shared cluster (namespaces + RBAC + quotas), cluster-per-team, and virtual clusters (vCluster)—to decide what's appropriate for our SRE environment.

Comparison (operational cost / security isolation / noisy-neighbor / upgrade complexity):- Shared cluster (namespaces + RBAC + resource quotas) - Operational cost: Low — single control plane, fewer clusters to maintain. - Security isolation: Low–medium — namespaces + RBAC provide logical separation, but control-plane and node-level access can leak; PodSecurity, NetworkPolicies required. - Noisy neighbor: Medium — resource quotas & limits mitigate but noisy workloads on shared nodes still impact others unless strict node/pod isolation used (taints, dedicated node pools). - Upgrade complexity: Low — one upgrade path; coordination needed but single window.

- Cluster-per-team - Operational cost: High — many control planes, more infra, cluster lifecycle overhead. - Security isolation: High — strong isolation at control plane and node level; faults less likely to cross team boundaries. - Noisy neighbor: Low — resource contention limited to that cluster; teams can size their clusters independently. - Upgrade complexity: High — each cluster must be upgraded/tested individually; can stagger but overhead is large.

- Virtual clusters (vCluster) - Operational cost: Medium — single physical cluster but separate virtual control planes per tenant; management tooling required. - Security isolation: Medium–high — virtual control planes limit kube-apiserver access per tenant, but underlying nodes are still shared so node-level isolation needed. - Noisy neighbor: Medium — improved API isolation but pods still share nodes; can combine with node pools for better isolation. - Upgrade complexity: Medium — virtual control planes can be upgraded per tenant independently, while underlying physical cluster upgrades remain a concern.

Recommendation (when to use each):- Shared cluster: Use for many small teams or dev/test environments where cost and simplicity matter, and security/risk tolerance is moderate. Enforce strict NetworkPolicies, PodSecurity, quotas, admission controllers.- Cluster-per-team: Use for high-security, high-compliance, or production workloads where isolation and blast-radius reduction are top priorities and you can afford operational overhead.- vCluster: Use as a middle ground when you want per-team API isolation and developer-friendly self-service without full cluster-per-team costs—good for large orgs with many dev/test environments, or when teams need different control-plane configs but can tolerate shared nodes.

Final note: Combine approaches—e.g., production on cluster-per-team, staging/dev on shared or vCluster—and automate cluster creation/upgrades, observability, and cost monitoring to manage operational burden.

HardTechnical

54 practiced

Define service level objectives for a public API hosted on Kubernetes receiving 100k requests per minute. Choose SLOs and SLIs (latency percentiles, success rate), compute allowed error budget, propose alert thresholds and escalation policies, and explain how SLOs should influence release cadence, canary policies, and error budget governance.

Sample Answer

Situation & approach: For a public API at 100k requests/min, define measurable SLIs (success + latency percentiles), concrete SLO targets, compute error budgets over a rolling 30-day window, set alert thresholds tied to burn rate and remaining budget, and define how SLOs control releases, canaries and governance.

SLIs (what we measure)- Success rate: fraction of HTTP requests with 2xx (and acceptable 3xx) responses within expected business semantics.- Latency percentiles (observed at ingress/load‑balancer or service mesh): p50, p95, p99 response time.- Request throughput and CPU/memory error metrics (500s, timeouts, OOMs) as supporting SLIs.

SLOs (targets, rolling 30-day window)- Availability / success rate: 99.95% (four 9s- half) over 30 days.- Latency: p50 < 50ms for 99% of requests; p95 < 200ms for 99.9% of requests; p99 < 500ms for 99.99% of requests.(You may relax tightening depending on user tolerance and cost.)

Compute error budget (30-day window)- Total requests/month ≈ 100,000/min * 60 * 24 * 30 = 4,320,000,000 requests.- Allowed failure fraction = 1 - SLO = 0.0005 (0.05%).- Allowed failed requests = 0.0005 * 4,320,000,000 = 2,160,000 failed requests per 30 days.- For latency SLOs, compute allowed violations similarly: e.g., p95 target 99.9% ⇒ 0.1% of requests may exceed 200ms => 4,320,000 requests/month allowed to be >200ms.

Alerting and escalation policy- Two classes of alerts: - SLI violation alerts (customer-impacting): trigger when rolling 5m or 1h error-rate or latency percentile breaches an SLO threshold. - Error budget burn alerts: monitor burn rate over short (6h) and medium (24h) windows.- Concrete thresholds: - Warning: remaining error budget < 50% (on 30d window) OR 6h burn rate > 2x expected. Pager to on-call (P2), Slack notification to SRE + dev owning service. - Critical: remaining error budget < 20% OR 6h burn rate > 5x expected OR sustained SLI violation > 15 minutes. Immediate page (P1) to SRE on-call + dev lead, conference bridge. - Emergency: error budget exhausted (0% remaining) OR actual end-user impact increasing (e.g., availability < 99.9% sustained 30m). All-hands, release freeze, rollbacks allowed.- Escalation: if on-call ack >5 minutes escalate to secondary; if remedial action not started in 15 minutes, notify engineering manager and incident commander.

Canary and release cadence governed by SLOs- Canary policy: - Start with small canary (1% traffic) for at least 15–30 minutes or until canary SLIs stable. - Define objective pass criteria: no SLI violations during canary; burn rate within baseline; no new 5xx or increased p99 > 2x baseline. - If canary passes, ramp to 10% for 30–60 minutes, then to 50% for health window, then full. - Automate rollback on predefined SLI thresholds (e.g., error rate increase by >3x or p99 increase > 2x).- Release cadence & error budget governance: - If error budget healthy (>50% remaining): normal release cadence allowed (weekly or as planned). - If error budget low (20–50% remaining): only low-risk releases; require engineering lead sign-off and extended canary windows. - If error budget exhausted (<20% remaining or 0): freeze feature releases and high-risk changes until budget is recovered; only emergency/security fixes allowed after risk review and SRE approval.

Implementation considerations on Kubernetes- Measure SLIs at edge (API Gateway / ingress-controller / service-mesh e.g., Istio) to capture real user latency and status codes, not only pod-level.- Use high-cardinality telemetry (prometheus histograms for latency, counters for success/fail). Export p50/p95/p99 via histogram_quantile or use summary metrics with caution (summary vs histogram trade-offs).- Correlate SLI spikes with pod events (OOM, restarts), HPA scaling, node pressures, kube-dns etc.- Automate canary traffic routing with service mesh (virtual services) or ingress + weighted routing; integrate with CI/CD to gate progressive rollouts by SLI checks.- Alerting: implement burn rate calculations in Prometheus (compare current error ratio to allowed ratio scaled by window) and create synthetic transactions for end-to-end SLI validation.

Governance & process- Error budget policy documented: owners, decision rights, rollback criteria, exception process for security/urgent fixes.- Blameless postmortems for any exhausted budget or critical incidents; feed learnings into release risk assessment and capacity planning.- Regular SLO reviews: re-evaluate SLOs quarterly against customer needs, cost, and observed telemetry.

Why this matters- Tying releases to SLO-driven error budgets balances velocity vs reliability. Canary automation + clear burn alerts let teams move fast while limiting user impact and giving structured escalation when risk thresholds are crossed.

EasyTechnical

72 practiced

Describe the role of an Ingress resource versus an Ingress Controller. For TLS termination, path-based routing, and basic rate limiting requirements, which parts do you configure in the Ingress versus controller annotations or an upstream proxy?

MediumTechnical

69 practiced

Explain how PodDisruptionBudget (PDB) interacts with voluntary evictions such as node drain and involuntary evictions like node failure. How does Cluster Autoscaler respect PDBs and what happens when the number of available pods is at or close to the PDB threshold during maintenance?

Sample Answer

PodDisruptionBudget (PDB) declares how many replicas must remain available during disruptions (minAvailable) or how many can be unavailable (maxUnavailable). It is enforced by the Kubernetes eviction API for voluntary disruptions, but it cannot prevent involuntary loss.

Behavior summary- Voluntary evictions (node drain, kubectl drain, eviction API): The drain/eviction call goes through the API and the PDB controller checks whether the eviction would violate any PDB for the pod’s owner. If evicting a pod would make available replicas drop below minAvailable (or exceed maxUnavailable) the eviction is blocked and drain fails unless you override (--force or bypass PDB).- Involuntary evictions (node failure, kubelet crash, network partition): These are not mediated by the eviction API. Pods are removed/killed by kubelet/node controller and PDBs cannot stop that — availability can temporarily fall below PDB until controllers reschedule pods onto healthy nodes.

Cluster Autoscaler (CA) interaction- CA respects PDBs when considering scale-down. It simulates evictions for all pods on a candidate node using the eviction API semantics: if simulating eviction would violate a PDB, CA treats that pod as non-evictable and will not choose that node for removal.- If many pods are at or near their PDB thresholds, CA will avoid scaling down nodes because it cannot safely evict the pods without violating PDBs. CA may postpone scale-down until replicas increase, pods become movable (e.g., new nodes added), or PDBs relax.

What happens during maintenance- During a planned maintenance drain: drain will stop at the PDB threshold; you must either temporarily lower PDB, scale out extra replicas, schedule maintenance during low load, or force eviction (accepting outage risk).- During unplanned failure: PDBs don’t protect you; controllers will attempt to recreate pods elsewhere but rescheduling can be delayed by capacity or scheduling constraints.- Practical mitigations: design PDBs with realistic headroom (avoid minAvailable equal to current replicas), ensure horizontal autoscaling or spare capacity, prefer disruption-aware maintenance windows, and use PodDisruptionBudgets in combination with Cluster Autoscaler and capacity buffers to prevent blocked drains or unintended outages.

Edge notes- PDBs do not apply to DaemonSets or static pods.- CA’s simulation respects controller-owned replicas; singletons or StatefulSets with strict minAvailable are common causes of blocked drains.

Unlock Full Question Bank

Get access to hundreds of Container Orchestration and Kubernetes Operations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.