Containerization and Orchestration Fundamentals Questions
Practical understanding of container technology and orchestration platforms. Topics include container image creation and registries, container runtime basics, core orchestration concepts such as pods, services, deployments, replicas, stateful sets, persistent storage, configuration management, service discovery and networking, scaling and autoscaling, rolling updates and rollbacks, self healing, resource management, and common deployment patterns. Emphasis is on how containers and orchestrators change deployment and operations practices and on at least one major platform such as Kubernetes or a cloud managed service.
EasyTechnical
43 practiced
Describe Kubernetes StatefulSet and how it differs from a Deployment. Explain the guarantees StatefulSet provides (stable network identity, ordered deployment/termination, stable storage) and cite at least two example applications that should use StatefulSets. Mention any operational caveats.
Sample Answer
StatefulSet is the Kubernetes controller for managing stateful applications that require stable identities and persistent storage. Unlike a Deployment (which treats pods as interchangeable), StatefulSet gives each pod a stable, unique identity and associated storage over its lifecycle.Key guarantees:- Stable network identity: each pod gets a stable DNS name and ordinal (pod-0.my-headless-svc.namespace.svc.cluster.local). Clients can rely on fixed hostnames.- Ordered deployment/termination: pods are created and terminated in ordinal order (0→N-1 on create; N-1→0 on delete). Updates follow the configured updateStrategy (RollingUpdate with controlled ordering or Partitioned updates).- Stable storage: using volumeClaimTemplates, each pod receives its own PVC that is bound to a PV and remains attached to that pod across restarts (unless PVC is manually deleted).Example applications that should use StatefulSets:- Databases requiring stable identity and per-node storage: PostgreSQL primary/replica clusters (when manual topology is used) or CockroachDB in simpler setups.- Distributed systems with node identity or local disks: Cassandra, ZooKeeper, or Kafka brokers (often used for broker identity and local log directories).Operational caveats (important for architecture/design):- StatefulSet does not provide data replication, backup, or automatic recovery across failure domains — design replication/HA at the application layer (e.g., database clustering) and include backup/restore.- PVC lifecycle: by default PVCs persist after pod deletion (retain policy); cleanup must be explicit. Storage class capabilities matter (ReadWriteOnce vs ReadWriteMany) and affect scheduling and failover.- Headless Service required for stable DNS. Ensure readiness/liveness probes are correct so ordered operations don’t proceed before app is ready.- Scaling down can be disruptive—removing a pod may require rebalancing application-level data (quorum sensitive systems).- Pod scheduling: StatefulSet doesn’t prevent multiple pods landing on the same node unless you add anti-affinity rules; for data redundancy, combine with topology/affinity rules.- Rolling updates: can be slow due to ordering; use Partitioned updates for staged rollouts.As a solutions architect, choose StatefulSet when app topology and persistent per-node state are required, and complement it with storage design, backup strategy, and cluster topology to meet availability SLAs.
EasyTechnical
59 practiced
Explain the differences between Kubernetes Service types: ClusterIP, NodePort, LoadBalancer, and ExternalName. For each type give a concrete scenario where it is appropriate. Also describe briefly how cloud providers implement LoadBalancer services and any implications for cost and security.
Sample Answer
ClusterIP- Definition: Default service type; exposes service on an internal virtual IP reachable only within the cluster (kube‑proxy / iptables or IPVS).- Scenario: Backend microservice (e.g., user-profile API) consumed by other services or ingress controllers; no external exposure.NodePort- Definition: Allocates a port (30000–32767) on every node; traffic to nodeIP:nodePort is forwarded to the service.- Scenario: Simple dev or legacy setup where you lack a cloud LB or need quick external access for testing; or when running on bare metal with external hardware load balancer pointing to node ports.LoadBalancer- Definition: Requests a cloud provider’s external load balancer (Service controller creates cloud LB and maps to nodes/pods).- Scenario: Public web app needing managed external endpoint, health checks, and scalable distribution.- Cloud provider note: Controller creates provider-specific LB (e.g., AWS ELB/ALB, GCP LB, Azure LB) and configures target groups, health checks, security groups/firewall rules. Implications: extra cost per LB, potential quota limits, and per‑hour/GB billing. Security: ensure least-privilege, restrict source CIDRs, enable TLS at LB or use Ingress with certs; consider internal LBs for private apps.ExternalName- Definition: Maps service DNS to an external CNAME; no proxying by cluster.- Scenario: Integrating managed SaaS (e.g., db.company.com) into cluster DNS so pods use service name without changing client code.Tradeoffs summary:- ClusterIP for internal-only and lowest cost.- NodePort simple but exposes node surface and relies on nodePort range and node IPs.- LoadBalancer convenient but costs money and may require cloud IAM/network setup.- ExternalName is DNS-only—no traffic control or health checks—best for stable external endpoints.
HardTechnical
57 practiced
Explain Kubernetes networking end-to-end: pod IP model, service virtual IPs, kube-proxy modes (iptables vs IPVS), CNI plugins and how they implement pod networking, and practical debugging steps when pods on different nodes cannot reach each other on specific TCP ports. Include commands and sources of truth to inspect.
Sample Answer
Kubernetes networking overview (end-to-end):Pod IP model- Every pod gets a unique IP in a flat cluster network (no NAT by default). Pods communicate using those IPs. The CNI plugin is responsible for assigning and programming routes/bridges for pod IPs so any node can route to any pod.Service virtual IPs (ClusterIP)- Services provide a stable virtual IP (ClusterIP) and DNS name. Traffic to ClusterIP is load-balanced to backend pod IPs via kube-proxy rules (or via service mesh/Envoy) and may perform DNAT to pod IP:port.kube-proxy modes- iptables mode: kube-proxy programs iptables rules to match ClusterIP and rewrite DNAT to endpoints. Fast and simple, but rule count grows with services/endpoints.- IPVS mode: uses Linux IPVS kernel load-balancer (better performance, scales to large numbers of services, supports scheduling algorithms). Requires kernel support and ipvsadm tooling.CNI plugins and how they implement pod networking- Bridge-based (calico can run with IP-in-IP or BGP; flannel uses VXLAN/host-gw; weave uses overlay mesh): create veth pairs, attach one end to pod netns, other to host bridge; program routes or encapsulation so pod IPs are reachable across nodes.- Calico: can do pure L3 with BGP or encapsulation; programs host routes in kernel (no overlay if BGP).- Flannel: simple VXLAN or host-gw overlay; programs routes via vxlan interfaces.- Cilium: eBPF-based, programs routing and policies in kernel using XDP/eBPF, can replace kube-proxy.Choice trade-offs: overlay vs underlay, performance, policy model, complexity.Practical debugging when pods on different nodes cannot reach each other on specific TCP ports1) Reproduce and gather facts- From pod A run:2) Check endpoints & service3) Node-level networking- On source node:- On dest node:4) CNI and kube-proxy health5) Node firewalls / security groups- Check host iptables/nftables/ufw and cloud SGs (AWS SG/NACL) for port blocking.6) Sources of truth- Pod IPs: kubectl get pod -o wide- Node routes: ip route, ip neigh- CNI state: CNI plugin docs, daemonset logs, eBPF maps (cilium)- kube-proxy rules: iptables/ipvs state- Kernel captures: tcpdump, ss/sockstatCommon root causes and fixes- CNI not running or crashlooping → fix daemonset- Host routing missing (overlay misconfigured) → check CNI config and MTU, host routes or BGP peering- Firewall/security group blocking specific ports → open ports- kube-proxy rule missing (ClusterIP access) → restart kube-proxy or use IPVS mode- Port in use on host or container not listening → check process inside podFinal tip: narrow scope—start ping/tcpdump at node-level, then pod netns, then kube-proxy/iptables/ipvs, then CNI logs, then cloud/host firewall.
bash
# Check pod IP/name
kubectl get pod -o wide -n <ns>
kubectl exec -n <ns> <podA> -- ip addr && ip route
kubectl exec -n <ns> <podA> -- curl -v telnet://<podB_ip>:<port> || nc -vz <podB_ip> <port>bash
kubectl get svc -n <ns> <svc> -o yaml
kubectl get endpoints -n <ns> <svc> -o widebash
# Can host reach destination pod IP?
sudo ip route get <podB_ip>
ping -c3 <podB_ip>
sudo ss -tnlp | grep <port>bash
# Verify pod network namespace receiving traffic
sudo tcpdump -n -i any host <podB_ip> and port <port>
sudo ip netns exec <pod_netns> ss -tnlpbash
kubectl get daemonset -n kube-system # calico/cilium/flannel pods
kubectl logs -n kube-system ds/<cni-daemon> --tail 200
kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system kube-proxy-<node>
# ipvs inspect
sudo ipvsadm -Ln --stats
# iptables check
sudo iptables -t nat -S | grep <ClusterIP or podIP>HardSystem Design
62 practiced
Propose an observability architecture for 50 Kubernetes clusters producing logs, metrics, and traces. Address centralization vs local aggregation, retention and indexing strategy, cost controls, GDPR/data-residency requirements, multi-tenant access controls, and query performance. Recommend tools and patterns (prometheus/thanos, Loki/Fluentd, OpenTelemetry) and explain trade-offs.
Sample Answer
Requirements & constraints (clarify up-front):- 50 K8s clusters, multi-region, mixed tenants, must ingest logs, metrics, traces- GDPR / regional residency, cost sensitivity, multi-tenant isolation, query latency SLAsHigh-level approach- Hybrid: local aggregation + central long-term store. Keep fast short-term data local for low-latency ops; centralize long-term retention, cross-cluster queries and compliance.Architecture1. Collection & instrumentation- Metrics: Prometheus node/sidecar per-cluster (federation-friendly) + OpenTelemetry exporters for app metrics where needed.- Logs: Fluentd/Vector as cluster DaemonSet to collect, filter, PII-scrub, and forward.- Traces: OpenTelemetry SDKs to send spans to local collector (OTel Collector DaemonSet).2. Local aggregation (per-cluster)- Short-term Prometheus TSDB (e.g., 15–48h) for fast alerting and dashboards.- OTel collectors batching & Adaptive Sampling (to control trace volumes).- Fluentd/Vector performs enrichment, PII redaction, routing rules.3. Central long-term storage- Metrics: Thanos (Prometheus + Thanos Sidecar) or Cortex/VMware Mimir for multi-tenant, scalable long-term metrics with downsampling.- Logs: Loki (index-light, cost-effective) or Elasticsearch depending on query needs. Recommend Loki for Kubernetes labels + cost efficiency; Elasticsearch if complex full-text queries required.- Traces: Grafana Tempo (S3-backed, index-less) or Jaeger with Cassandra/Elasticsearch. Tempo + traces indexing via trace IDs in Loki is cost-effective.Retention & indexing strategy- Hot tier (local): high-resolution metrics (raw), logs for 7–14 days, traces sampled at full fidelity for 48–72h.- Warm tier (central): downsampled metrics (1m/5m) indefinitely per SLA, logs indexed by minimal labels (cluster, namespace, app) retained 30–90 days in hot storage; cold archive (S3/Glacier) for 1+ year as required.- Indexing: minimal structured indexes (labels) in Loki; avoid indexing full text to reduce cost. For metrics, use label cardinality controls.Cost controls- Sampling & rate-limits on traces (OTel tail-based where needed).- Ingest filtering and PII scrubbing at collectors.- Downsampling metrics via Thanos/Downsample component.- Use object-storage (S3/GCS) for long-term data; use lifecycle policies (glacier) for old data.- Enforce cardinality limits and label normalization to avoid explosion.GDPR & data-residency- Deploy regional aggregator endpoints: route data to region-specific central stores based on cluster or tenant metadata.- PII handling: collectors perform deterministic scrubbing/masking before export; hash or tokenization for identifiers.- Encryption at rest (KMS) and in transit (mTLS/TLS). Audit logs for access.- Data residency policy enforced via routing rules and storage location tags; legal hold support in archive.Multi-tenant access control- Multi-tenant backends: Thanos/Cortex with tenant isolation via tenant IDs and per-tenant quotas.- Loki multi-tenancy with tenant headers; Tempo supports tenant separation.- RBAC: integrate with OIDC (Keycloak/Azure AD) and enforce tenant-based RBAC in Grafana/Query frontends.- Network isolation and separate namespaces for collectors; per-tenant quotas and rate-limits.Query performance & UX- Fast queries: keep recent data local; use query frontends (Thanos Query, Loki Query Frontend) with caching layer (Redis) for repeated queries.- Shard by tenant/cluster for parallel query execution.- Precompute rollups/aggregates for expensive queries (Prometheus recording rules).- Provide unified UI (Grafana) with datasource proxying and tenant-aware dashboards.Operational patterns & tooling- Observability platform: Grafana + Prometheus + Thanos (or Cortex/Mimir) + Loki + Tempo + OpenTelemetry collectors + Fluentd/Vector.- CI for pipeline configs, Terraform for infra, Helm for deployments.- Monitoring of the observability stack itself (SLOs, budgets, alerting).- Runbook for GDPR requests (data deletion/exports), audits, and incident response.Trade-offs- Loki + Tempo + Thanos: cost-efficient, scale well, lower indexing cost; trade-off: less flexible full-text search than ES.- Elasticsearch: powerful querying but higher cost and operational overhead.- Centralizing everything reduces duplication but increases egress, complexity for residency — hybrid minimizes egress and latency while centralization supports cross-cluster observability.Resulting benefits- Scalable multi-tenant architecture, cost controls via sampling/downsampling and object storage, GDPR-compliant regional routing and scrubbing, performant queries for recent data and affordable long-term retention.
HardSystem Design
53 practiced
Design a hybrid architecture that integrates on-prem Kubernetes clusters with a cloud-managed EKS environment to enable bursting to cloud capacity. Address secure network connectivity (VPN/Direct Connect), unified authentication and RBAC, image registries and caching, data replication strategies, CI/CD pipelines that deploy across both environments, and operational considerations for monitoring and policy enforcement.
Sample Answer
Requirements & constraints:- Burst from on-prem K8s to EKS with secure low-latency connectivity, unified auth/RBAC, shared image access, data replication for stateful apps, CI/CD that can target both, and unified monitoring/policy.High-level architecture:- On-prem K8s (K8s upstream/CKE) + AWS EKS clusters in same AWS account(s). Secure connectivity via AWS Direct Connect (primary) with IPSec VPN (failover) and Transit Gateway for multi-VPC routing. BGP over DX for deterministic routing and private IP connectivity.Network & security:- Private VPC peering/Transit Gateway connecting EKS VPCs to on-prem via Direct Connect. Use encrypted NFS/iSCSI only over trusted links for block storage; prefer application-level encryption for data-in-transit. Apply network segmentation, AWS security groups, and Calico/Antrea NetworkPolicy on clusters. Use VPC endpoints for ECR/S3 to avoid public egress.Authentication & RBAC:- Central identity with corporate IdP (Okta/ADFS) using SAML for AWS SSO and OIDC for Kubernetes. Configure EKS and on-prem clusters to trust the same OIDC provider via Dex or directly with IdP; map IdP groups to Kubernetes RBAC roles. Use IAM Roles for Service Accounts (IRSA) on EKS; on-prem use bound service-account-to-VM roles via short-lived tokens and an STS proxy if needed. Enforce least privilege and use Kubernetes RBAC plus PodSecurityAdmission and OPA/Gatekeeper for policy.Image registries & caching:- Primary registry: Amazon ECR (multi-region). On-prem cache: Harbor or local registry configured as pull-through cache for ECR (or use ECR pull-through). Use lifecycle policies and signed images (COSIGN) for provenance. Configure nodes to prefer local cache; implement registry replication for pre-warming images before bursts.Data replication strategies:- Stateless services: nothing required.- Object storage: Use S3 as canonical store. On-prem MinIO or S3 gateway with async replication to S3 (replicate writes to cloud). For low-latency reads during burst, replicate hot dataset to S3 and to on-prem cache via MinIO mirroring or AWS DataSync.- Databases: Use logical replication or CDC (Debezium) to replicate from on-prem primary to cloud read-replicas. For failover, implement planned promotion workflows and careful RPO/RTO SLAs. For high-consistency needs, prefer active-active with conflict resolution (CRDTs) or application-level reconciliation.- Stateful workloads on Kubernetes: use CSI drivers that support cross-site replication (Portworx, NetApp Astra, or Velero for backups). Snapshot replication to AWS EBS snapshots for recovery.CI/CD & deployment:- Central GitOps model: Git repos (GitHub/GitLab) + ArgoCD/Flux instances per cluster. Pipelines run in cloud (GitHub Actions/CodeBuild) with runners in both environments for local deployments and artifact caching. Pipeline steps: - Build/push images to ECR. - Run security scans (SCA/SAST), sign images. - Update manifests/Helm charts and PR to repo. - ArgoCD sync to target clusters with environment overlays (kustomize).- For bursting: a deployment manifest with HPA and Cluster-aware scheduler (Karpenter on EKS + Cluster Autoscaler on on-prem), and a controller to create cloud-targeted resources when clusterCapacity < threshold.Operational considerations:- Monitoring: Prometheus + Thanos for long-term metric federation (thanos sidecars on clusters, object store in S3), Grafana with unified dashboards, Fluentd/Fluent Bit to central logging (AWS OpenSearch or ELK in cloud), CloudWatch integration for EKS metrics. Use OpenTelemetry for traces and a central tracing backend (Jaeger/Tempo).- Security policy enforcement: OPA/Gatekeeper for admission policies, Kyverno for mutating policies, image policy webhook for signed images, runtime security with Falco, and regular vulnerability scanning with Trivy/Anchore.- Observability for bursts: synthetic tests to validate connectivity, health-checks for replicated data freshness, and dashboards for replication lag, DX link utilization, and cost.- Backup & DR: Velero snapshots to S3, periodic DB backups to S3, test restores. Runbooks for failover (promote read replicas, reconfigure DNS/ingress).- Cost, latency, and failover trade-offs: Direct Connect reduces egress cost & latency but adds cost/lead time; VPN is cheaper but higher latency. Active-active replication lowers RTO but increases complexity and consistency concerns.Trade-offs & risks:- Strong consistency across sites is hard — prefer async replication with clear RPO/RTO and app-level reconciliation for most services.- Image pull latency can be mitigated with caching but requires cache population strategy.- Centralized identity simplifies RBAC but needs careful token management and availability planning.Why this works:- Uses managed cloud services (EKS, ECR, S3) for scalability while keeping on-prem control. GitOps ensures reproducible deployments across both environments. Federation (Thanos, Fluentd) provides unified ops visibility. OPA/SSO and network design provide secure, compliant bursting capability with explicit trade-offs for consistency, cost, and complexity.
Unlock Full Question Bank
Get access to hundreds of Containerization and Orchestration Fundamentals interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.