Kubernetes Deployment and Gitops Questions

Focuses on implementing safe deployment patterns specifically in Kubernetes and on GitOps practices for declarative, version controlled operations. Topics include Kubernetes native rollout models such as rolling updates, canary releases, and blue green deployments; pod lifecycle considerations, readiness and liveness probes, resource limits, and health checks; implementing automated continuous deployment pipelines that target Kubernetes clusters; and GitOps principles including declarative manifests, version control as the source of truth, automated reconciliation, and observable drift management. Candidates should also know common GitOps tools such as Argo CD and Flux, manifest management approaches like Helm and Kustomize, and operational concerns such as rollback procedures, security and access controls, and integration with monitoring.

MediumTechnical

95 practiced

An operator reports the cluster drifted because someone updated a Deployment directly on the cluster. Using Argo CD, explain how you'd detect this drift, alert the team, and define a policy for auto-sync vs manual reconciliation. Mention concrete Argo CD configuration knobs (e.g., app settings, autoPrune, project restrictions) you'd use.

Sample Answer

**Brief approach**Detect drift quickly with Argo CD's built-in reconciliation + notifications, then enforce a policy that auto-syncs safe resources and requires manual approval for sensitive ones.

Detection & alerting- Enable frequent reconciliation by setting App resync and/or controller arg (--app-resync-rate) to a lower interval so diffs are detected quickly.- Install argocd-notifications and configure a trigger for app.spec.status.health or app.status.sync to fire on "OutOfSync" or "Degraded" events.- Example notification trigger: subscribe to built-in event SyncStatusChanged or ResourceUpdated and send Slack/Teams/webhook with app name, resource, user who changed it (from Kubernetes audit logs if linked).

Concrete Argo CD knobs- App-level: spec.syncPolicy.automated with: - automated.prune: true/false (prune deleted resources during auto-sync) - automated.selfHeal: true (re-apply when cluster drift detected) - automated.allowEmpty: false (avoid empty manifests auto-applying)- Project-level restrictions (argocd Project CRD): - spec.sourceRepos to limit repos - spec.destination.clusters/namespaces to restrict where apps can deploy - spec.clusterResourceWhitelist / resourceExclusions to prevent auto-reconcile of certain cluster-scoped types- SyncWindows: schedule automatic syncs only in allowed windows for production changes- RBAC + admission: require SSO/RBAC for who can modify apps; enforce policy with deny in project to prevent direct cluster edits by non-approved service accounts

Policy: auto-sync vs manual reconciliation- Auto-sync (safe): stateless apps, ConfigMaps/Secrets managed by GitOps, infra that tolerates instant rollbacks - Enable spec.syncPolicy.automated.selfHeal = true and automated.prune = true- Manual (sensitive): stateful workloads (Databases), migrations, cluster-scoped resources, or production fast-paths - Disable automated on those App CRs; set syncPolicy: null so changes go "OutOfSync" and require a human-triggered sync - Use project.resourceExclusions or clusterResourceWhitelist to force manual reconciliation for specified kinds (e.g., PersistentVolume, CustomResourceDefinitions) - Combine with SyncWindows and require approvals via Git PRs + CI checks

Operational steps after detection- Notification includes: app, resource, diff link (argocd UI CLI: argocd app diff), and last-modifying user (from audit)- Triage: if change is legitimate, merge the Git change or allow manual sync; if illegitimate, either auto-sync rollback (if allowed) or run `argocd app sync --revision=<git-rev>` to reconcile- Add post-incident: tighten project restrictions, add audit webhook to log direct kubectl edits, and consider mutating/validating admission to block unauthorized edits.

This combination uses:- App spec.syncPolicy.automated.{selfHeal,prune,allowEmpty}- argocd controller flags (resync rate)- argocd-notifications for alerts- Projects (sourceRepos, destinations, resource white/blacklists)- SyncWindows and RBAC to implement safe auto-sync vs manual reconciliation policies.

MediumSystem Design

69 practiced

Design release gates and approval workflows for a GitOps pipeline that must satisfy compliance requirements (auditable approvals for production). Describe where gates are enforced (CI, PR, Argo CD), what metadata should be stored in Git, and how to implement human approvals without breaking declarative principles.

Sample Answer

**Clarify goals**- Declarative GitOps with auditable, human approvals for production; minimal manual intervention; full audit trail stored as immutable metadata.

**Where gates are enforced**- CI (pre-merge): automated security, tests, SBOM generation, policy-as-code (OPA/Gatekeeper, Snyk) — fail fast.- PR (pre-merge): reviewers, required approvals, signed commits, link to ticket/intent. PR merge is the policy-controlled gate into the protected branch that represents intended state.- Argo CD (deploy-time): sync gating to control actual cluster change — use manual/automated sync windows, AppProject allowlists, or an admission controller that enforces signed approval attestations before sync.

**Metadata stored in Git**- For each release (manifest commit or kustomize/helm values): - release-id, target-environment, image digests, SBOM/reference provenance - approval-record: approver id, timestamp, justification, ticket id, commit sha, signature - policy-attestations (in-toto/SLSA or sigstore bundle)- Store as small YAML/JSON alongside manifests or in a separate approvals directory; every approval is a signed commit to preserve immutability.

**Implement human approvals without breaking declarative principles**- Model approval as declarative resource: Approval CRD or approvals/<env>/<release>.yaml in Git. A trusted operator/controller watches Git and only allows Argo CD sync when: - an approval resource exists with valid signature and policy attestations - CI artifacts referenced (image digests) match- Workflow options: - Git-native: approver opens PR against protected production branch containing the approval file; required reviewers and signed commit enforced by branch protection; CI policy validates signatures and then merge triggers Argo CD sync. - Controller-native: approver uses UI/tool to create signed attestation stored back in Git (via bot) or submits signed JWS; controller enforces and records acceptance as a commit.- Use sigstore/in-toto for non-repudiation; push approvals as commits (not out-of-band) so Git remains source of truth and audit log.

**Audit & compliance**- Keep Git history + signed attestations; mirror approvals into SIEM or immutable audit store (WORM) if required.- Ensure RBAC mapping from approver identity to corporate directory; require MFA and verify signatures.

**Trade-offs**- Git-native PR approvals keep everything auditable and declarative but add human latency.- Controller/UIs improve UX but must write back to Git to preserve declarative source-of-truth.

This design preserves declarative GitOps while providing auditable, policy-driven human approvals enforced at CI/PR and deploy-time via Argo CD and controllers.

HardSystem Design

70 practiced

Design a progressive delivery platform for a 50-service microservices application using Argo Rollouts (or equivalent) and a service mesh. The platform must support canary and blue-green strategies, automated metric-driven promotions, cross-service coordination for multi-service releases, observability hooks, and safe rollback. Describe architecture, key components, and orchestration approach.

Sample Answer

**High-level summary**I’d build a GitOps-driven progressive delivery platform on Kubernetes using Argo Rollouts + ArgoCD, a service mesh (Istio/Linkerd), Prometheus-based metric gates, and an orchestration layer to coordinate multi-service releases. The platform supports canary & blue/green, automated metric promotion/rollback, cross-service coordination, observability hooks and safe rollback.

**Architecture & key components**- GitOps: ArgoCD + ApplicationSets for per-env manifests (Kustomize/Helm).- Progressive delivery: Argo Rollouts for canary/blue-green with service mesh traffic-shift integration.- Service mesh: Istio (or Linkerd) for fine-grained routing, traffic mirroring, mTLS, and observability.- Metrics & SLOs: Prometheus + Alertmanager + Thanos; weighted metrics exported to Argo Rollouts via Prometheus metrics provider.- Orchestration/coordination: Release Coordinator (CRD + controller) or Argo Workflows to define multi-service Release CRDs that create coordinated Rollout resources and enforce ordering/parallelism.- Observability UI: Grafana dashboards, Kiali for mesh, Loki for logs, Tempo for traces.- CI: Pipeline triggers (Tekton/Jenkins/GitHub Actions) update Git; ArgoCD syncs.

**Orchestration approach**- Developer creates a Release PR. CI builds images and updates manifests in Git (image tags).- Release CRD (created by CD pipeline) lists services, strategy (canary/blue-green), and SLOs/metrics.- Coordinator controller instantiates Argo Rollouts per service with consistent rollout windows and traffic weights. It can: - Start canaries in waves (e.g., 1%, 5%, 25%, 100%) across services in sequence or parallel. - For blue/green, create new k8s service backends and flip routing atomically via mesh VirtualService.- Argo Rollouts uses Prometheus metric provider to evaluate configured metrics (latency, error rate, business KPI). Promotion occurs automatically when all metric checks pass; failures trigger automated rollback or pause for manual judgment.- Cross-service coordination: Coordinator waits for each service’s Rollout status (healthy) before moving to next wave. Supports transactional semantics: all-or-nothing by halting and rolling back others on failure.- Safe rollback: Immediate traffic shift back to stable via mesh, Argo Rollouts rollback operation, and automated incident creation (PagerDuty/Slack) with diagnostics (logs/traces snapshot).

**Observability & safety**- Hooks: pre/post-analysis webhooks and Prometheus queries; run distributed request traces to validate behavior. Use canary analysis tooling (Kayenta/Prometheus-based) for statistical confidence.- Chaos & load testing: Canary runs can include synthetic traffic and load-testing hooks.- RBAC & approval gates: Slack/GitOps approvals, SLO thresholds, and manual pause steps in Rollouts/Workflows.

**Trade-offs**- Complexity vs control: Mesh + Rollouts adds operational overhead but gives fine traffic control and safer releases.- Consistency: Coordinator CRD centralizes cross-service releases but must be resilient; design for idempotency and reconciliation.

This design gives safe, observable, and automated progressive delivery for 50 microservices with strong cross-service coordination and clear rollback semantics.

HardTechnical

66 practiced

Write pseudocode for a Kubernetes validating admission webhook that enforces resource policies: every Pod/Deployment must set requests.cpu >= 100m and limits.cpu <= 500m; requests.memory >= 128Mi and limits.memory <= 1Gi. Show the core validation logic handling an AdmissionReview, an example rejection message payload, and briefly explain how you would register and secure this webhook in the cluster (TLS and RBAC highlights).

Sample Answer

**Approach (brief)** Validate AdmissionReview requests for Pod/Deployment specs; reject if cpu/memory requests/limits outside policy.

**Pseudocode (core webhook handler)**

python

# webhook.py (pseudocode)
def handle_admission_review(admission_review):
    req = admission_review.request
    obj = decode_resource(req.object)  # Pod template for Pod or Deployment
    containers = obj.spec.containers

    violations = []
    for c in containers:
        req_cpu = parse_quantity(c.resources.requests.cpu or "0")
        lim_cpu = parse_quantity(c.resources.limits.cpu or "0")
        req_mem = parse_quantity(c.resources.requests.memory or "0")
        lim_mem = parse_quantity(c.resources.limits.memory or "0")

        if req_cpu < parse_quantity("100m"):
            violations.append(f"{c.name}: requests.cpu {c.resources.requests.cpu} < 100m")
        if lim_cpu > parse_quantity("500m"):
            violations.append(f"{c.name}: limits.cpu {c.resources.limits.cpu} > 500m")
        if req_mem < parse_quantity("128Mi"):
            violations.append(f"{c.name}: requests.memory {c.resources.requests.memory} < 128Mi")
        if lim_mem > parse_quantity("1Gi"):
            violations.append(f"{c.name}: limits.memory {c.resources.limits.memory} > 1Gi")

    if violations:
        return admission_response(allowed=False, status_message="; ".join(violations))
    else:
        return admission_response(allowed=True)

**Example rejection AdmissionReview response (payload)**

json

{
  "apiVersion":"admission.k8s.io/v1",
  "kind":"AdmissionReview",
  "response":{
    "uid":"<request-uid>",
    "allowed": false,
    "status":{
      "code": 403,
      "message":"container nginx: requests.cpu 50m < 100m; container nginx: limits.memory 2Gi > 1Gi"
    }
  }
}

**Register & secure webhook (high-level)**- Register ValidatingWebhookConfiguration pointing to service namespace/name, operations CREATE/UPDATE, resources pods, deployments, and appropriate failurePolicy and sideEffects.- TLS: serve webhook over HTTPS with a serving certificate signed by cluster CA or use cert-manager to provision secrets; put CA bundle in ValidatingWebhookConfiguration's clientConfig.caBundle.- RBAC: give the webhook service account minimal privileges (if it needs to read objects, grant get/list/watch on pods/deployments); typically webhook only needs to read request body, so minimal RBAC.- Additional: restrict webhook to desired namespaces via namespaceSelector, and enable logging/metrics and health/readiness probes to avoid blocking API server.

HardTechnical

85 practiced

Explain how reconciliation loops work in GitOps controllers (for example Argo CD and Flux). Discuss scaling challenges when managing thousands of manifests across many clusters, including informer/watch scaling, reconcile frequency, rate-limiting, caching strategies, and design decisions to reduce API-server and controller load.

Sample Answer

**Overview — reconciliation loop basics**

A GitOps controller runs a continuous reconcile loop: it reads desired state from Git, compares to cluster actual state, computes a diff, and applies changes (kubectl/apply or server-side apply). Reconciliation is eventual, idempotent, and convergent — controllers requeue resources until observed state equals desired.

**Scaling challenges**

- Informer/watch scaling: Kubernetes informers reduce API-server pressure by sharing watches, but thousands of manifests across many clusters multiply memory and connection counts. Each controller instance maintaining many informers increases heap and FD usage.- Reconcile frequency & storming: Naive frequent full-syncs cause API-server spikes and thundering-herd updates across clusters.- Rate-limiting: Both client-side (token bucket) and server-side (API QPS/burst) limits must be respected; aggressive retries can exhaust quotas.- Caching & CDN: Controllers should cache manifests, use checksum-based change detection, and avoid reapplying unchanged resources.

**Design decisions to reduce load**

- Shared informers and dynamic informer factories to limit watch count; one shared cache per resource type.- Event-driven reconciles: reconcile on Git webhook + resource events, not fixed polling; use debounce/deduplication windows.- Hierarchical/partial-sync: group resources into apps/helm charts and reconcile at app-level to reduce ops.- Optimistic server-side apply and patch operations to minimize diff payloads.- Adaptive rate limiters that back off on 429s and use jitter.- Sharding controllers by cluster or namespace to bound memory and connections.- Use read-only cache layers (e.g., cache-server or aggregator) and object ownership indexing to avoid listing whole clusters.

**Example practices (Argo/Flux)**

- Flux: uses Kustomize/manifest caching, Git event-triggered syncs, and backoff policies.- ArgoCD: app-of-apps pattern, repo-server caching, and configurable sync windows/QPS.

These patterns prioritize event-driven, cached, sharded, and rate-limited reconciliation to keep API-server and controller load predictable at scale.

Unlock Full Question Bank

Get access to hundreds of Kubernetes Deployment and Gitops interview questions and detailed answers.

Join thousands of developers preparing for their dream job.