Kubernetes Architecture and Core Components Questions

Fundamental architecture of Kubernetes and its core components: control plane design including API server, etcd distributed state store, scheduler, controller manager, and cloud controller manager; worker node components including kubelet, kube proxy, and the container runtime; core objects and abstractions such as pods, services, deployments, stateful sets, daemon sets, jobs, config maps, and secrets; cluster state management, service discovery, health checks, and self healing. Candidates should understand how components interact, data flow, failure modes, and how to design for availability and scalability at the component level.

MediumTechnical

77 practiced

Discuss the Container Runtime Interface (CRI) and compare common container runtimes such as containerd and CRI-O (and historical dockershim). Explain how runtime choice affects image handling, container startup latency, debugging approaches, and security considerations.

Sample Answer

**Overview (what CRI is)** The Container Runtime Interface (CRI) is Kubernetes’ gRPC API that decouples kubelet from container runtimes. It standardizes lifecycle, image management, and sandboxing so kubelet can work with containerd, CRI‑O, or other CRI-compliant runtimes.

**Runtimes comparison** - containerd: CNCF project, broad ecosystem, supports OCI images, integrated shim for per-container processes. Good balance of features and tooling. - CRI‑O: Lightweight, Kubernetes-focused, minimal extras, built around OCI and Kubernetes CRI needs. Lower attack surface. - dockershim (historical): Docker Engine included a non-CRI shim; deprecated because Docker Engine is broader than required.

**How runtime choice affects...** - Image handling: All CRI runtimes use OCI images, but performance differs. containerd’s built‑in registry and snapshotters (overlayfs/zstd) offer flexible caching; CRI‑O is optimized for Kubernetes image pulls and thin overlays. - Container startup latency: Lighter runtimes (CRI‑O) and efficient snapshotters reduce startup time. containerd with native runc or kata will vary—sandboxing (gVisor/Kata) increases latency. - Debugging approaches: containerd exposes ctr/ctr‑plugins and debug logs; CRI‑O uses crictl + journal logs. For both, use kubectl logs, crictl inspect, and host-level tools (nsenter, ps, strace). containerd may give richer tooling for image/content stores. - Security: CRI‑O’s minimalism reduces attack surface; both support seccomp, AppArmor, SELinux, and rootless modes. Runtime choice affects support for isolated sandboxes (Kata, gVisor) and vulnerability surface (complexity of daemon).

**Practical recommendation (DevOps perspective)** For typical Kubernetes CI/CD clusters: containerd is a solid default (tooling + performance). For hardened clusters with minimal runtime surface and strict compliance, prefer CRI‑O and consider Kata/gVisor for untrusted workloads. Ensure consistent snapshotter, runtimeclass, and centralized logging to simplify debugging and security posture.

EasyTechnical

55 practiced

Describe Kubernetes Service types: ClusterIP, NodePort, LoadBalancer, and ExternalName. For each type, explain typical use cases, how traffic routing works, and implications for external access, load balancing, and DNS entries.

Sample Answer

**ClusterIP**- What it is: Default internal-only virtual IP inside cluster.- Traffic routing: kube-proxy programs iptables/ipvs to NAT traffic to pod endpoints via Endpoints/EndpointSlices.- Use cases: Internal microservice communication, service discovery for backends, headless services combined with DNS when needed.- External access: Not directly accessible from outside cluster; expose via Ingress, NodePort, or port-forward.- Load balancing/DNS: kube-proxy performs round-robin/load distribution; Kubernetes DNS maps service name to ClusterIP.

**NodePort**- What it is: Allocates a static high port on every node that forwards to ClusterIP.- Traffic routing: External traffic hits node:<nodePort> → kube-proxy → service endpoints.- Use cases: Quick external exposure for dev/test, simple bare-metal setups without cloud LB.- External access: Accessible via node IPs and port; you must manage node IPs and firewall.- Load balancing/DNS: Basic per-node forwarding; external load balancer or DNS round-robin needed for multi-node balancing.

**LoadBalancer**- What it is: Provisions cloud provider LB (AWS ELB, GCP LB, Azure LB) and maps to Service.- Traffic routing: External LB IP → node(s) → NodePort/ClusterIP → pods (cloud provider may do direct pod routing).- Use cases: Production external services with managed LBs and health checks.- External access: Externally reachable via provider-assigned IP/DNS; provider handles certificate/SSL offload if configured.- Load balancing/DNS: Use provider LB DNS or map your DNS to LB IP/name; LB does cross-node balancing and health checks.

**ExternalName**- What it is: DNS-only service mapping service name to external CNAME.- Traffic routing: Kubernetes DNS returns CNAME to external hostname; no proxying.- Use cases: Integrating external services (managed DB, APIs) into cluster DNS without pods.- External access: Client resolves to external host directly; cluster does not provide network path.- Load balancing/DNS: External host handles LB; Kubernetes DNS simply aliases the name.

Notes for interviews: mention security (firewalls, network policies), Ingress for HTTP+TLS, and using ExternalTrafficPolicy or sessionAffinity when preserving client IP/session is required.

HardTechnical

71 practiced

Design and outline how you would implement a custom scheduler (or scheduler extender) for specialized workloads (for example GPU jobs with affinity and anti-affinity rules). Describe the integration points with the default scheduler framework, binding flow, failure handling, idempotency, and how to safely test and roll it out.

Sample Answer

**Approach summary**I’d implement a scheduler extender + optional scheduler-framework plugin to handle GPU affinity/anti-affinity, integrating with kube-scheduler via extender API and the scheduler framework extension points (Filter, Score, Reserve, Permit, PreBind, Bind, PostBind).

**Integration points**- Use scheduler framework plugin for Filter/Score to compute candidate nodes using topology, GPU counts, and affinity rules.- Implement Reserve/PreBind to mark GPUs (soft reservation) and a Bind implementation (or extender Bind) to perform final device allocation via CRD (e.g., DeviceClaim) and patch Pod status.- Expose an HTTP extender for fallback decisions and complicated heuristics if needed.

**Binding flow**1. kube-scheduler calls Filter/Score plugin → returns ranked nodes.2. Reserve marks resources in a lightweight cache/CRD.3. Permit ensures constraints globally (e.g., gang scheduling).4. Bind calls extender or controller that creates DeviceClaim and updates Pod.spec.nodeName atomically with optimistic concurrency.

**Failure handling & idempotency**- All reservation/claim operations idempotent: use resource names derived from Pod UID.- Use optimistic concurrency with retries and backoff; if Bind fails, PostBind/Unreserve cleans reservations.- Implement reconciliation controller to detect leaked reservations and reassign after timeout.- Record events and metrics for failed binds; implement exponential backoff and alerting.

**Testing & rollout**- Unit tests for plugin logic; integration tests using KinD and GPU simulator (fake device plugin).- Canary rollout: enable plugin in staging scheduler first, run mixed workloads, use feature flags and capacity limits.- Gradual traffic shift via node labels and pod toleration selectors targeting test nodes.- Rollback: scheduler config snapshots, automated drain of affected pods, and reassign to default scheduler.

This design ensures consistent placement, safe binding, recoverable failures, and controlled rollout for GPU workloads.

MediumTechnical

68 practiced

Explain how the Kubernetes scheduler makes placement decisions. Cover the scheduler pipeline (filtering and scoring or scheduler framework plugins), the impact of resource requests and limits, taints/tolerations, node affinity/anti-affinity, topology spread constraints, and how these interact to produce a final binding.

Sample Answer

**Overview — what scheduler does**The kube-scheduler selects a node for a Pod by evaluating cluster state against constraints and preferences, then issuing a Bind. It runs as a loop: watch unscheduled Pods → find feasible nodes → rank them → pick best → bind.

**Pipeline / Framework plugins**- Filter (predicate) phase: plugins eliminate nodes that violate hard constraints (insufficient allocatable resources considering Pod resource requests, unschedulable taints, nodeSelector/nodeAffinity hard rules, topology spread hard constraints).- Score (priority) phase: remaining nodes are scored by preference plugins (e.g., resource binpacking, balanced resource allocation, node affinity soft rules, topology spread scoring). Extenders and custom scheduler plugins can participate.- Reserve/Permit/Prebind/Bind: optional phases for multi-stage logic (e.g., reservation, admission gating), final Bind writes Pod.spec.nodeName.

**Resource requests vs limits**- Resource requests (cpu/memory) drive scheduling decisions: scheduler sums requests per node against capacity/allocatable. Limits do not affect placement directly except via Admission or QoS classification; limits matter for eviction and runtime cgroups.- Overcommit: using requests lower than limits allows packing, but risks OOM/CPU contention.

**Taints and tolerations**- Taints mark nodes with "NoSchedule/PreferNoSchedule/NoExecute"; a Pod must have matching tolerations to be considered (NoSchedule) or will be evicted (NoExecute). Taints act as filters; tolerations can be soft (PreferNoSchedule handled in scoring).

**Node affinity / anti-affinity**- nodeAffinity requiredDuringScheduling acts as a filter. preferredDuringScheduling influences scores. Pod anti-affinity can be expensive (cross-node checks) and can be hard or soft; hard anti-affinity filters nodes hosting matching pods.

**Topology Spread Constraints**- Hard constraints ensure the selected node doesn't violate maxSkew across topology domains; soft constraints contribute to scoring so pods spread evenly across domains (zones, nodes).

**How they interact to produce final binding**- Order: filters (requests, taints/tolerations, required affinities, hard topology) remove nodes → scoring combines multiple plugin scores (resource utilization, preferred affinities, topology spread) → tie-breakers (least requested, random) choose node → Bind. Any dynamic changes (resource updates, taints) between scoring and bind can cause Bind to fail; scheduler requeues Pod and retries.

**Practical notes for DevOps**- Tune requests to reflect real usage to avoid binpacking-induced OOMs.- Use taints for dedicated nodes; use tolerations sparingly.- Prefer topologySpread for zone-aware high availability.- Monitor scheduler logs and scheduling events to diagnose placement failures.

HardSystem Design

56 practiced

Design a deployment strategy for stateful databases on Kubernetes that aims for zero data loss during rolling upgrades and node failures. Discuss PersistentVolume choices, replication and leader election strategies, fencing, snapshotting and backup procedures, and safe upgrade steps for both the storage and database software.

Sample Answer

**Situation overview (goal)** Design a Kubernetes deployment for stateful DBs (zero data loss during rolling upgrades and node failures) — choose storage, replication/election, fencing, snapshot/backup, and safe upgrade steps.

**PersistentVolume choices**- Use a durable CSI-backed StorageClass with ReadWriteOnce + dynamic provisioning (cloud block storage with multi-attach if needed) or a distributed block store (Portworx, Longhorn, Rook/Ceph) that supports snapshots, cloning and encryption.- Set reclaimPolicy=Retain, volumeBindingMode=WaitForFirstConsumer to avoid scheduling vs. topology issues.- Enable CSI VolumeSnapshots and ensure snapshot controller is installed.

**Replication & leader election**- Prefer a quorum-based HA topology (e.g., 3+ nodes for etcd/Postgres Patroni with 3 replicas). Use asynchronous replicas only when acceptable; for zero data loss use synchronous or semi-sync replication (sync commit / majority commit).- Use an orchestrator that integrates with K8s: Patroni for Postgres (consensus via etcd/Consul/Kubernetes API), or K8s operator that manages leader election and safe failover.- Maintain at least N=3 voting members; ensure watches survive API server restarts.

**Fencing**- Implement fencing to avoid split-brain: - Use cloud provider APIs to detach volumes or revoke access for old leader nodes (STONITH). - Use CSI node-level access control (node publish/unpublish) and strict volume attachment policies. - Operators should perform pre-promote checks and use regulated leases (Kubernetes Lease API) for leadership.

**Snapshotting & backup**- Point-in-time safe backups: combine consistent CSI snapshots (application-consistent if possible using fs freeze or DB quiesce hooks) + logical backups: - For Postgres: basebackup (pg_basebackup) + continuous WAL shipping to object storage (S3) for PITR. - For MongoDB: use filesystem snapshots + oplog archiving.- Automate periodic full backups to off-cluster object storage, encrypted at rest and in transit. Retention and lifecycle per RPO/RTO.- Test restores regularly (runbook + CI job that restores to a separate namespace).

**Safe upgrade steps**- Storage/CSI driver upgrade: 1. Read release notes; ensure CSI snapshot CRDs compatible. 2. Drain upgrade of driver/control-plane components (one control-plane instance at a time), verify volume attach/detach operations. 3. Run integration tests: create snapshot, restore, mount, read/write.- Database software / schema upgrade (rolling, zero-loss): 1. Ensure replicas are healthy and fully caught up. 2. For leader-in-place upgrades: perform controlled leader stepdown (operator API), promote fresh replica if needed. 3. Upgrade followers first, verify sync, then upgrade leader. If in-place leader upgrade required, perform stepdown, upgrade old leader, rejoin as follower. 4. Use PodDisruptionBudgets and Pod anti-affinity so replicas spread across nodes. 5. If schema changes are required, apply non-breaking migrations first, then backfill, then enforce constraints.- For node replacement: use volume detachment + attach with CSI fencing; verify replica promoted only after fence success.

**Observability & runbooks**- Monitor replication lag, commit latency, CSI attach/detach events, snapshot success, backup health.- Maintain SLOs: RPO=0 with strict sync replication; RTO targets and tested runbooks for leader failover and restore.- Regular chaos tests (node kill, kubelet restart, network partition, storage failover) in staging.

This design balances strong consistency (synchronous/quorum replication + fencing) with practical operational safety (snapshots, PITR, tested restores, staged upgrades).

Unlock Full Question Bank

Get access to hundreds of Kubernetes Architecture and Core Components interview questions and detailed answers.

Join thousands of developers preparing for their dream job.