Containerization and Orchestration Questions

End to end container platform knowledge covering both containerization and orchestration at production scale. This includes advanced Docker topics such as image optimization, container networking at scale, build pipelines and registries, and security considerations, plus advanced Kubernetes topics such as cluster management, performance tuning, multi cluster strategies, custom controllers and operators, service mesh and ingress architectures, storage classes and persistent volumes, monitoring and logging, cost optimization, and real world operational challenges. Candidates should be prepared to discuss trade offs between managed and self managed services, deployment strategies, scaling patterns, and lessons learned from production incidents.

HardTechnical

31 practiced

A recent rollout caused many pods to be OOMKilled in production after a configuration change. Describe a comprehensive triage plan: which logs and metrics to check (kubelet, container runtime, application logs), how to identify the failing release and offending container image, immediate mitigations to restore service, and steps to perform root cause analysis and prevent recurrence.

Sample Answer

**Situation & goal**Quickly restore service after many pods were OOMKilled, identify the bad release/image, mitigate impact, and perform RCA to prevent recurrence.

**1) Immediate triage — logs & metrics to check**- Cluster metrics: check node memory pressure, pod OOM counts, and kubelet eviction metrics (Prometheus / Grafana).- Kubelet & node logs:

bash

# on affected node
journalctl -u kubelet -u docker --since "1h"
dmesg | grep -i oom

- Container runtime logs (containerd/docker) for OOM events and exit codes.- Pod & container logs:

bash

kubectl get pods -A --field-selector=status.phase=Failed
kubectl describe pod <pod>
kubectl logs <pod> -c <container> --previous

- Deployment/ReplicaSet rollout history:

bash

kubectl rollout history deploy/<name>
kubectl get replicaset -l app=<app> -o wide

- CI/CD / image tags and recent commits from pipeline logs.

**2) Identify failing release & offending image**- Correlate timestamp of OOM events with rollout time from deployment annotations and CI pipeline.- Inspect ReplicaSets: the new RS with many failing pods points to the release. Check pod spec.image and imageDigest:

bash

kubectl get pod <pod> -o yaml | yq '.spec.containers[].image'
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[].imageID}'

- Cross-check image digest with registry to map to a build/commit.

**3) Immediate mitigations to restore service**- Roll back to previous stable ReplicaSet:

bash

kubectl rollout undo deploy/<name>

- Temporarily scale up healthy replicas / scale down failing rollout to reduce noise:

bash

kubectl scale deploy/<name> --replicas=<n>
kubectl scale rs/<bad-rs> --replicas=0

- Add a memory limit guard or QoS change if missing: patch deployment to set safe memory requests/limits (use small patch, then finalize after testing).- If nodes under pressure, cordon/drain compromised nodes and add capacity:

bash

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-local-data

**4) Root cause analysis**- Reproduce in staging using same image/diffed config. Run stress tests while capturing: - Application logs, heap profiles, metrics (memory RSS, GC), and container metrics.- Inspect image changes: added memory-heavy feature, dependency bump, different JVM options, or missing limits.- Analyze runtime metrics: RSS vs cgroup memory, memory spikes patterns, OOM killer logs (dmesg shows process killed).- Check resource requests/limits: missing or too-low limits cause OOMKilled or eviction.- CI: inspect build that produced offending digest for config changes (env vars, startup flags).

**5) Prevent recurrence**- Enforce mandatory resource requests/limits via admission controller (LimitRange/ResourceQuota or OPA/Gatekeeper policy).- Add pre-deployment memory/soak tests in CI using the same image and realistic load.- Improve observability: instrument app with memory metrics (heap, native), add Prometheus alerts for memory growth and OOM events, and record image digests in deployment annotations.- Automate safe rollbacks in pipeline and add canary deployments with traffic shaping.- Postmortem: document timeline, root cause, decisions, and action items with owners and deadlines.

This plan prioritizes quick recovery, precise identification of the offending image/release, and both technical and process fixes to avoid repeat incidents.

EasyTechnical

60 practiced

Describe Kubernetes core objects and how they relate: Pod, Deployment, Service, ConfigMap, Secret, and Namespace. Provide a brief example (3-4 sentences) showing how these objects work together to deploy a stateless web service with environment configuration and secret credentials.

HardSystem Design

43 practiced

Design container networking for a large cluster (10k nodes, ~500k pods). Decide between overlay and underlay approaches, propose an IPAM strategy, consider use of BGP-capable CNI (e.g., Calico with BGP), address SNAT scaling issues, network policy enforcement at scale, and outline debugging approaches for network issues in such an environment.

Sample Answer

**Clarify requirements & constraints**- 10k nodes, ~500k pods, multi-AZ, hybrid cloud possibility, high-throughput east-west, strict network policies, low latency.

**High-level decision: underlay (BGP) preferred**- Use BGP-capable CNI (Calico in BGP mode) to avoid overlay encapsulation overhead and per-packet overhead at this scale. Underlay gives predictable latency, better MTU, and simpler troubleshooting. Overlay (VXLAN/IPIP) acceptable for environments without BGP or across fragmented networks, but expect performance and scaling trade-offs.

**IPAM strategy**- Allocate /16 per node (or /24 per node depending on pod density) from a large cluster CIDR (e.g., 10.0.0.0/8) so each node advertises a contiguous block. This minimizes RIB state: nodes advertise their host route, not per-pod. Use hierarchical allocation: global pool → AZ pools → node pools. Automate leases with etcd-backed allocator or Calico IPAM with strict IP reuse and reclamation TTL.

**BGP CNI details**- Calico in BGP mode: each node runs a BGP speaker advertising its node CIDR to top-of-rack/border routers or to a BGP route-reflector fabric. Use route reflectors for scalability (avoid full mesh). Use node-local Felix agent with iptables/ipsets for policies. Secure BGP with MD5/TLS and route filters.

**SNAT scaling**- Avoid cluster-wide SNAT: use egress gateways per AZ (small fleet of nodes with SNAT/NAT44) or node-local egress NAT with kube-proxy IPVS to localize SNAT. Use scalable NAT (e.g., AWS NAT Gateway or cloud NAT) for north-south traffic. Ensure ephemeral port capacity by scaling egress gateways horizontally and tracking port exhaustion.

**Network policy at scale**- Compile policies to ipsets/iptables via CNI (Calico compiles to ipsets for performance). Prefer namespace + label selectors to minimize rule count. Use policy tiers, default deny on critical namespaces. Offload heavy policy enforcement to eBPF dataplane (Calico/eBPF) to reduce iptables churn and improve performance.

**Observability & debugging**- Instrument: node-level flow logs (eBPF/conntrack), BGP session dashboards, route-reflector metrics, iptables/ipset counters, Prometheus, Grafana.- Debug workflow: 1) verify node-to-node BGP routes (bird/FRR show); 2) check node route tables and kernel routes; 3) check pod IP presence and node CIDR; 4) conntrack and iptables counters for SNAT issues; 5) capture flows with tcpdump/eBPF (bcc, px) on node and pod; 6) use Calicoctl/calico diagnostics and Kubernetes networkpolicy debug tools.- Automation: health checks, synthetic traffic probes (canary pod mesh), alert on route drift, BGP flaps, port exhaustion.

**Trade-offs**- BGP underlay adds ops complexity (route reflectors, BGP security) but gives performance and scalable routing. Overlays simplify ops at cost of performance and harder debugging.

This design balances performance, predictable routing, scalable IPAM and policy enforcement, and operational observability suitable for a 500k-pod cluster.

HardTechnical

39 practiced

Explain Kubernetes scheduling end-to-end: how the kube-scheduler evaluates pending pods, predicate and priority phases, how predicates/filters like affinity/taints/taints-tolerations are applied, preemption behavior, scheduler extenders/custom schedulers, and strategies to tune scheduling performance for very large clusters (50k+ pods).

Sample Answer

**Overview (brief)** Kubernetes scheduling finds a node for each Pending Pod by running the scheduler component (kube-scheduler) through filtering (predicates) and scoring (priorities) phases, then binding the Pod. As a DevOps engineer you must understand both correctness (constraints) and performance at 50k+ Pod scale.

**Scheduling flow** - Watch API for Pending Pods → use scheduler cache (Nodes, Pods, PodsPerNode). - Filter phase: run plugins that eliminate nodes (NodeReady, Disk/EphemeralStorage, Resources, NodeSelector, NodeAffinity, PodAffinity/AntiAffinity, Taints/Tolerations, TopologySpread). Only feasible nodes remain. - Score phase: rank remaining nodes (LeastRequested, BalancedAllocation, InterPodAffinity, NodeAffinity weights, etc.). Highest score chosen. - Reserve/Permit/PreBind/Bind hooks may run; scheduler writes Bind API.

**Predicates / Filters specifics** - NodeSelector/NodeAffinity: exact matches vs weighted preferences. - PodAffinity/AntiAffinity: expensive (O(n)) checks—use topology keys to limit scope. - Taints & Tolerations: taints filter out nodes unless toleration exists; tolerations are evaluated in filter phase. - TopologySpread and InterPodAffinity use topology keys to distribute load.

**Preemption** - If no node fits, scheduler considers preemption: selects victim Pods to evict based on Priority/PreemptionPolicy to make room, minimizing disruption (lowest priority, minimal number of victims). Preemption is last-resort; tune PriorityClasses and set PreemptionPolicy ("Never" or "PreemptLowerPriority").

**Extenders & Custom Schedulers** - Scheduler extenders: external HTTP services to filter/score nodes for specialized constraints (e.g., proprietary hardware). They run after built-in filters/scores. - Custom schedulers: run alongside kube-scheduler by setting Pod.spec.schedulerName; useful for specialized placement logic or isolation.

**Performance tuning for very large clusters (50k+ Pods)** - Run multiple scheduler profiles or dedicated schedulers to split workload (by namespace, priority class, or workload type). - Scale control plane: more API server replicas, larger etcd (resource tuning, disk IOPS). - Optimize scheduler plugin set: disable expensive plugins (full PodAffinity) or rewrite as less-global constraints; prefer NodeAffinity/NodeSelectors. - Increase scheduler parallelism/workers and tuning scheduler cache/gc settings. - Use predicate caches and nodeInfo reuse; enable profiling and measure with scheduler_perf tests. - Reduce event churn: batch Pod creations, use admission controllers to validate constraints early. - Use TopologySpread and PodDisruptionBudgets carefully; offload balancing to controllers (HPA, custom reconcilers) where possible. - Monitor scheduling latency, bind success rate, and etcd compaction; adopt canary testing when changing scheduler logic.

**Trade-offs** - Accuracy vs speed: richer constraints increase placement quality but raise CPU/memory and latency. Balance by splitting schedulers and simplifying predicates. - Preemption reduces tail latency but increases churn—use sparingly.

This demonstrates how kube-scheduler enforces constraints, handles preemption, integrates extenders/custom schedulers, and how to scale/tune scheduling for large DevOps-managed clusters.

EasyTechnical

43 practiced

Describe an image tagging and promotion strategy for a team using dev, staging, and production environments. Include how you use semantic versioning, commit SHAs, CI job IDs, and promotion mechanics (e.g., copy or retag), and explain how deployments should be triggered from registry tags to ensure immutability and traceability.

Unlock Full Question Bank

Get access to hundreds of Containerization and Orchestration interview questions and detailed answers.

Join thousands of developers preparing for their dream job.