Specific Experience with Core DevOps Tools Questions

Be prepared to discuss hands-on experience with specific tools relevant to the job description: CI/CD platforms (Jenkins, GitLab CI, GitHub Actions), containerization (Docker), orchestration (Kubernetes basics), cloud platforms (AWS, Azure, GCP), and Infrastructure as Code (Terraform, CloudFormation). Prepare concrete examples of how you used these tools.

EasyTechnical

31 practiced

Explain Terraform core concepts: providers, resources, modules, state, and remote backends. Give an example of how remote state and locking (e.g., S3 + DynamoDB) prevent conflicts when multiple engineers apply changes concurrently.

HardTechnical

31 practiced

You must migrate an existing set of AWS resources that were created manually into Terraform without downtime. Explain the end-to-end migration plan: how to import resources into state, reconciling Terraform resource addresses with current naming, renames and replacements, protecting state consistency, validating after import, and rolling back if there are discrepancies.

Sample Answer

**Situation & goal**Migrate manually created AWS resources into Terraform without downtime; preserve live infra, ensure state consistency, detect/reconcile drift, and have a tested rollback.

**Plan (high level)**1. Inventory & map - Use AWS Config, awscli, Terraform AWS provider data sources, and scripts to list resources and metadata. - Produce mapping: AWS identifiers ↔ desired Terraform addresses (module paths).

2. Prepare repo & configs - Create Terraform files reflecting current desired resource attributes (only essential args needed to import). - Add lifecycle { prevent_destroy = true } where accidental deletion is risky. - Commit to VCS and open a PR for review.

3. State safety - Enable remote state with locking (e.g., S3 + DynamoDB) and versioning. - Take an initial state snapshot: download backend bucket copy and tag a backup. - Lock pipeline access; schedule maintenance window if necessary.

4. Import workflow (incremental) - Work per-resource or per-group. Example commands:

bash

# create resource block in .tf with matching attributes
terraform init
terraform plan   # confirms no resources in state yet
terraform import aws_s3_bucket.mybucket my-bucket-name

- For resources whose Terraform address differs from desired naming, import to a temp address then move:

bash

terraform import aws_instance.tmp_web i-0123456789
terraform state mv aws_instance.tmp_web module.web.aws_instance.web

- For renames that would cause replacements, use terraform state mv to avoid destroy/create.

5. Validate after each import - Run terraform plan; expect no changes. If plan shows diffs, reconcile by updating TF config or accepting drift via import of missing attrs. - Run integration tests (smoke tests, health checks), monitor CloudWatch metrics.

6. Handling replacements & renames - If a resource requires replacement (immutable change), plan that change in a maintenance window and use blue/green or add new resource and cutover to avoid downtime. - Prefer refactor via state mv to keep the same resource id.

7. Rollback & recovery - If discrepancies or corruption occur: lock backend, restore the prior state file from the snapshot, run terraform init and plan to confirm. - If partial imports cause issues, use terraform state rm for problematic entries and re-import after fixing .tf.

8. Automation & verification - Automate import steps in CI with manual approval gates, log all state changes, and tag state backups with timestamps. - After full import, run drift detection periodically and enforce changes via CI.

**Why this works**- Incremental imports + state backups avoid downtime and destructive actions.- State mv prevents unintended replacement.- Remote state locking and snapshots protect consistency and enable quick rollback.

MediumTechnical

33 practiced

Outline how you would secure CI runners/executors and Kubernetes cluster access for a medium-sized engineering organization. Cover build isolation (sandboxing/ephemeral runners), least-privilege IAM/service accounts, secrets access policies, use of network segmentation and Pod Security admission controls to prevent lateral movement from compromised runners.

Sample Answer

**Approach overview**I’d apply defense-in-depth: make CI runners ephemeral and isolated, enforce least-privilege identities for pipeline processes, restrict secrets access, and use network + pod admission controls to block lateral movement.

**Build isolation / ephemeral runners**- Use short-lived runners (GitHub Actions self-hosted, GitLab, Jenkins) spun from immutable images via IaC (Terraform/Packer).- Run each job in its own ephemeral namespace/VM/VM-sandbox (e.g., Fargate, GKE Autopilot node pools, or k3s VMs) and destroy on completion.- Enforce image signing (in-toto/COSIGN) and run workloads as non-root with seccomp and AppArmor.

**Least-privilege IAM / service accounts**- Map CI jobs to minimal service accounts via Workload Identity (GCP) or IRSA (AWS) — no use of cluster-wide admin tokens.- Create dedicated roles per pipeline action (deploy, read-artifacts) with time-limited credentials and MFA/approval for high-risk ops.- Use short TTL tokens (STS) and monitor token usage.

**Secrets access policies**- Centralize secrets in Vault/KMS/Secret Manager; do not store creds in repo.- Grant pipelines secret access via dynamic, lease-based credentials and scoped policies (path-based ACLs).- Use sealed-secrets or HashiCorp Vault Injector for runtime injection; audit and rotate regularly.

**Network segmentation & Pod Security**- Enforce NetworkPolicies to restrict egress/ingress from runner pods to only needed endpoints (artifact stores, registry, control plane).- Put runners in separate node pools/taints+tolerations and dedicated namespaces with ResourceQuotas.- Enable Pod Security Admission (restricted profile) to forbid hostNetwork, hostPath, privileged containers.- Use PSP successors (OPA Gatekeeper/ Kyverno) to enforce labels, image registries, and allowed capabilities.

**Detect & respond**- Central logging and EDR for runners; alert on anomalous behavior (suspicious egress, unexpected kube-api calls).- Regular pentests, pipeline policy as code, and runbooks to revoke compromised service accounts.

Why this works: ephemeral isolation reduces dwell time; least-privilege + scoped secrets minimize blast radius; network + admission controls prevent lateral movement; detection closes the loop.

HardSystem Design

35 practiced

Design a disaster recovery plan for a Kubernetes service and Terraform-managed infrastructure across AWS regions to meet RTO=1 hour and RPO=15 minutes. Include backups for etcd, persistent volumes, database replicas, Terraform state replication, failover steps, and verification/testing procedures.

Sample Answer

**Situation & goals**Design a cross-region DR plan for Kubernetes + Terraform on AWS with RTO = 1 hour, RPO = 15 minutes. Focus on automated backups, fast failover, and verification.

**Architecture & assumptions**- Primary region (us-east-1), DR region (us-west-2)- EKS for Kubernetes, RDS for relational DB, S3 for Terraform state (with locking via DynamoDB)- PVs using EBS; supporting object stores on S3/MinIO

**Backup strategy (meeting RPO=15m)**- etcd: Use managed EKS control plane snapshots if available; otherwise run Velero with restic and frequent etcd snapshots (cronjob) stored in cross-region S3 bucket. Snapshots every 5–10 minutes; retain last N hourly/daily.- Persistent Volumes: Use EBS snapshots via lifecycle policy triggered every 10–15 minutes for critical PVs (or use EBS fast snapshot + multi-volume consistency with application quiesce scripts). Replicate snapshots to DR region automatically.- Database replicas: Use RDS cross-region read replica with automated binary log/stream replication; configure automated failover promotion. Binlog/Aurora backtrack configured for sub-15m recovery points.- Terraform state: Store state in S3 bucket with versioning, server-side encryption, and cross-region replication (CRR) to DR bucket. DynamoDB lock table replicated via global table (DynamoDB global tables) or export snapshots.- Configuration/artifacts: Container images pushed to ECR with cross-region replication.

**Failover steps (to RTO=1h)**1. Detect incident via monitoring/Runbooks trigger (CloudWatch, PagerDuty).2. Promote DR DB replica to primary (automated script) — target <10 min.3. Restore latest etcd snapshot to EKS control plane in DR cluster or spin up new cluster from IaC.4. Restore PVs from most recent EBS snapshots and attach to pods (or use Velero restore to rehydrate PVCs) — target 20–30 min.5. Point DNS (Route53 health checks + weighted policies) to DR ALB/EKS cluster; lower TTLs pre-configured (60s).6. Re-deploy apps via CI pipeline using Terraform state from DR S3 (already replicated) — use terraform init with backend config to DR bucket; run apply.7. Validate application health.

**Automation & orchestration**- Automation playbooks (Terraform + CI job) that can create DR cluster, restore etcd, promote DB, restore PVs, update DNS.- Use step functions or SSM automation runbooks to sequence operations and run checks.

**Verification & testing**- Quarterly full failover drill: run failover to DR cluster, validate traffic, validate DB integrity, RPO/RTO measured.- Monthly partial tests: restore etcd, restore subset of PVs, promote replica in staging.- Runback tests: simulate return to primary and re-sync data.- Define SLIs/SLOs: failover time, data loss window, application latency post-failover.

**Monitoring & validation**- Continuous checks: snapshot success, replication lag, Terraform state CRR status, RDS replica lag alarms.- Post-failover smoke tests and business transactions automated.

**Trade-offs & notes**- Cross-region EBS snapshot frequency increases cost; prioritize critical PVs.- Managed control plane snapshots reduce operational burden.- Keep runbooks and credentials securely available (Secrets Manager with cross-region replication).

This plan emphasizes automation, frequent replication, and scripted failover to meet RTO=1h and RPO=15m while allowing regular testing and continuous validation.

MediumTechnical

30 practiced

Your CI pipeline is failing intermittently because integration tests are flaky under load. Walk through a step-by-step triage and remediation plan: how to isolate flaky tests, capture artifacts, add retries smartly, mock external services, parallelize safely, and prevent new flaky tests from being merged.

Sample Answer

**Overview (goal)** As a DevOps engineer I’d drive a reproducible, minimal-risk remediation: identify flaky tests, capture evidence, add mitigations (retries/mocks/parallelism) safely, and prevent regressions via pipeline policy and observability.

**Step-by-step triage**1. Reproduce & isolate - Run failing job locally / on dedicated runner (same Docker image/K8s node type). - Add tagging: mark integration tests vs unit; run suspected flaky tests repeatedly (e.g., 100x) to get failure rate.2. Capture artifacts - Persist test logs, JVM/OS metrics, container stdout/stderr, network traces, and test screenshots to S3 (or CI artifacts). - Ship logs to ELK/Datadog and correlate timestamps with infra metrics (CPU, mem, IO, network).3. Root-cause analysis - Classify flakes: timing/race, resource exhaustion, external service timeouts, order-dependency. - Reproduce under load using load generators (locust/jmeter) or resource constraint tools (stress-ng, tc/netem).

**Remediation**4. Smart retries - Add limited, idempotent retries at test-wrapper level only for known transient errors (e.g., network timeout). Example: test runner flag --retry-on=TimeoutException, max 2. - Record retry counts and mark tests that only pass on retry for quarantine.5. Mock external services - Replace flaky third-party dependencies with deterministic mocks (WireMock, Pact) in CI; use contract tests for integration with real services in a separate pipeline. - Run a small set of end-to-end smoke tests against real environments (canary).6. Parallelize safely - Ensure tests are isolated: avoid shared state (DB schema per test using ephemeral DBs, namespaces in K8s). - Use per-job ephemeral resources via Docker Compose / testcontainers or spin test namespaces in Kubernetes to run parallel shards.7. Prevent future flakes - Add gating: require < X% flakiness on repeated runs, deny merge if test has >1 retry in CI. - Enforce test ownership, flaky-test dashboard, quarantine label, and pipeline job that runs flaky candidates under stress tests before merge. - CI policies: block merges if new integration tests are added without mocks or isolation.8. Monitoring & feedback - Alert when test flakiness trend rises; store historical flake metrics. - Run periodic chaos or load tests to validate pipeline resilience.

**Example tools**- Jenkins/GitHub Actions + S3 artifacts, ELK/Datadog, WireMock/Testcontainers, Kubernetes namespaces, pytest/jest retry plugins.

This approach balances quick mitigation (retries, mocks) with long-term fixes (isolation, observability, policy) so CI becomes reliable without hiding real issues.

Unlock Full Question Bank

Get access to hundreds of Specific Experience with Core DevOps Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.