Amazon DevOps Engineer (Entry Level) Interview Preparation Guide

Long-running SAST/SCA scans on a large monorepo, or across ten thousand repositories nightly, both come from the same root cause: re-analyzing code that has not changed since the last scan. The fix is to stop treating every scan as a full, from-scratch analysis.

Techniques, layered

• Changed-files analysis: on a pull request, only analyze the files that actually changed plus their direct dependents (a change to a shared utility function should re-trigger analysis of its callers, not the whole repository). This alone typically cuts PR-time scan scope by an order of magnitude on a large monorepo.
• Dependency-graph-driven incremental scanning: build a graph of which modules depend on which, so a change to module A only re-triggers full analysis of A and everything that transitively imports A, not the entire codebase; this is the same idea as changed-files analysis but applied at the module level rather than the file level, catching cases where a change has an effect beyond the literal lines touched.
• Caching: cache the analysis result for any file whose content hash hasn't changed since the last scan, keyed on the file's content hash plus the scanner's rule-set version, so a rule-set upgrade correctly invalidates the whole cache rather than silently reusing stale results.
• Pre-commit / fast-lint checks: push the very cheapest checks (a linter, a fast subset of SAST rules) to run locally before the developer even pushes, so the CI-side scan only needs to catch what local checks missed.
• Distributed scanning: for the case where a genuinely full scan is still needed (a nightly, org-wide sweep, or a rule-set upgrade that invalidates the whole cache), shard the work across many parallel workers by module or by repository rather than running one long serial scan.

Applying this at two different scales

On a single large monorepo with hours-long PR scans, changed-files plus dependency-graph incremental scanning is usually enough on its own to hit a sub-30-minute target, since the bottleneck is almost always re-analyzing unchanged code. Across ten thousand separate repositories where nightly SAST takes days, the bottleneck is different: it's the sheer number of independent scan jobs, so the highest-leverage fix there is prioritized scheduling (scan the repositories that changed today first, and the ones with no recent commits on a slower cadence) plus distributed, parallel execution across the fleet, alongside language-specific optimization (a scanner tuned per language runtime rather than one generic scanner run against everything) and developer-local tooling so an individual repository's own PR-time scan stays fast regardless of the nightly org-wide sweep's total duration.

Trade-offs

Caching and incremental scanning both introduce a real risk: a bug in the dependency graph (missing an indirect dependency) or a stale cache key can let a genuinely-affected file skip analysis entirely, silently reducing coverage rather than just speed. The mitigation is to run a periodic full, non-incremental scan (nightly or weekly) as a backstop that would catch anything the incremental path missed, and to treat the dependency graph itself as something that needs its own test coverage.

Affinity and anti-affinity are attraction rules: they pull a pod toward, or away from, nodes or other pods based on labels. Taints and tolerations are repulsion rules: a taint on a node repels every pod except the ones that explicitly tolerate it. Use affinity to express a placement preference from the pod's side; use a taint when the node itself needs to say nothing runs here unless it opts in.

Affinity and anti-affinity

Every one of these rules is written as either requiredDuringSchedulingIgnoredDuringExecution (hard: the scheduler rejects any node that fails the rule, and the pod stays Pending if none match) or preferredDuringSchedulingIgnoredDuringExecution (soft: it adds a weighted score toward better nodes but never blocks scheduling). The shared IgnoredDuringExecution suffix matters: none of these rules are re-checked once a pod is already running, so a label change elsewhere in the cluster does not evict an already-scheduled pod.

Taints and tolerations

Worked example: protecting spot-instance workloads with taints, tolerations, and priority

A common pattern is a batch-processing node pool built on spot (interruptible, discounted) instances, tainted so only workloads that explicitly accept interruption land there:

tolerations:
  - key: "workload-class"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

Pair that with a PriorityClass so that if the cloud provider reclaims spot capacity, the scheduler's preemption logic clears room in the right order:

priorityClassName: batch-low

If a pod carrying priorityClassName: batch-low (say, priority value 100) and one carrying priorityClassName: critical-high (priority value 1000000) are both competing for the same scarce capacity, preemption always targets the lowest-priority pod first, evicting enough batch-low pods to fit the pending critical-high pod rather than the reverse. Pairing a short tolerationSeconds on a NoExecute toleration with this setup gives batch pods a bounded grace window to checkpoint their work before eviction instead of an unbounded, unpredictable one.

Trade-offs and pitfalls

• Required pod affinity against a large, frequently-changing pod population is expensive for the scheduler to evaluate (it must check co-location against existing pods, not just static node labels); this shows up as scheduler latency growth in large clusters. Prefer preferred rules unless the hard requirement is genuinely non-negotiable.
• Overusing required anti-affinity for spreading can leave pods permanently Pending in a small cluster, or briefly during a rolling update if there are not yet enough distinct failure domains to satisfy the rule.
• A NoExecute taint applied broadly, for example across an entire zone during maintenance, evicts every pod without a matching toleration at once. Confirm which workloads actually tolerate it before applying it, or a planned maintenance action becomes a self-inflicted outage.
• A taint only protects the node's side of the relationship; it says nothing about what else can run there once a pod tolerates it. Multiple unrelated tolerating workloads can still crowd the same tainted node, so pair taints with resource requests or a dedicated node pool if true isolation is the goal.

The goal is to capture enough detail to diagnose an intermittent failure without paying the overhead of full verbose tracing on every request, all the time.

Approach

1. Baseline cheap signal always on: lightweight metrics and low-cardinality logs run continuously with near-zero overhead, giving you the "something is wrong, roughly when" signal.
2. Conditional/triggered detailed capture: when an anomaly threshold fires (an error, a latency outlier, a specific status code), automatically escalate to a detailed, ephemeral capture for a short window around that event: full request/response payloads (redacted), a stack trace, or a short eBPF trace of the exact syscalls/queries involved, stored only for that flagged window rather than continuously.
3. eBPF/kernel-level sampling for very low-overhead, always-on visibility into syscall latency or scheduling behavior without modifying the running application, useful when you cannot add application-level instrumentation cheaply.
4. On-demand debug snapshots: an operator-triggered dump of in-process state (thread stacks, queue depths, cache sizes) for a live investigation, taken sparingly since it can itself add latency.

Trade-offs and pitfalls

The riskiest failure mode is instrumentation changing the behavior you're trying to observe (turning a heisenbug into something that no longer reproduces once instrumented) - the anomaly-triggered design specifically avoids this by keeping steady-state overhead near zero and only paying the instrumentation cost in the rare window right around a real anomaly. The second risk is threshold tuning: too sensitive and you capture (and pay for) noise constantly; too loose and you miss the actual event. Start conservative and tighten based on how often the trigger fires versus how often it actually captures something useful in review.

High-level approach
I’d shift incentives and processes so reliability targets (uptime, SLAs) and safe experimentation (feature delivery, learning) are explicit, measurable, and complementary — not competing. I’d combine policy, rituals, tooling, and incentives to normalize learning while preserving uptime.

Policies & incentives

• Error budget model: allocate a measurable window for experiments; teams can trade error budget for faster deploys.
• 10–20% “learning time” policy (similar to Google’s 20%): protected weekly time for experiments, certification, or docs.
• Promotion/KPI rubric includes knowledge sharing, runbook quality, and postmortem participation, not just uptime.
• Rewards: public recognition, learning stipends, and career-path credits for mentoring/runbook authorship.

Rituals & cadence

• Blameless postmortems within 48 hours, with follow-up action items tracked to closure; share TL;DRs team-wide.
• Weekly 60-min “learning sprint” demos where engineers present experiments, failures, and test results.
• Monthly game days / chaos engineering exercises tied to real runbooks and incident playbooks.
• Pair rotations and on-call shadowing for cross-pollination of knowledge.

Tooling & automation

• CI/CD with feature flags & canary releases (LaunchDarkly / Flagr) to reduce blast radius.
• Observability stack (Prometheus/Grafana, ELK/Tempo) with error-budget dashboards and SLO alerts.
• Postmortem templates in Jira/Confluence; automatic incident capture (PagerDuty -> playbook creation).
• Runbook automation (RPA or scripts) and a searchable runbook repo with versioning.
• Safe sandbox/cloud credits + infra-as-code templates for reproducible experiments.

Why this works
Error budgets align incentives: teams get permission to iterate while preserving reliability. Rituals institutionalize learning; tooling lowers friction. Policies make learning a measurable part of success, not an extra. Together they create a feedback loop: experiments inform SRE practices; postmortems and game days surface gaps; runbooks and automation harden systems — all without compromising uptime.

Direct answer
Blue-green for a stateful service with local disk and sticky sessions means the "just switch traffic" simplicity of blue-green breaks down: you can't just point the load balancer at green, because green's local disk starts empty and any user pinned to a blue instance mid-session needs a plan for what happens to them at cutover.

Structured elaboration

• Session migration: for a user with an active session pinned to a specific blue instance, cutover has to either let their session finish naturally on blue (draining blue gradually rather than an instant hard switch) or externalize session state to a shared store (Redis, a database) BEFORE the cutover so any instance, blue or green, can serve any user's session; the second option is the more robust long-term fix, since it removes the sticky-session problem from future deploys entirely, not just this one.
• Data replication for local disk: if the local disk holds data that matters (a local cache that's expensive to rebuild, or genuinely durable state that shouldn't exist only on local disk at all), green needs that data BEFORE it starts serving real traffic. Two practical approaches: replicate/sync the data from blue to green as part of the deploy process before cutover (rsync-style, or an application-level replication mechanism), or, if the local disk is really just a rebuildable cache, accept a cold-cache warm-up period on green and validate performance under that condition specifically before committing to full cutover.
• Minimizing downtime given both constraints: a gradual, session-aware cutover rather than an instant, all-or-nothing traffic switch: route NEW sessions to green once it's validated and its disk state is synced, while existing sessions on blue are allowed to drain naturally (finish their current session) before blue is decommissioned, meaning both environments serve real traffic simultaneously for a bounded transition window, a deliberate deviation from blue-green's usual clean all-at-once switch.
• Minimizing data loss: any data written to blue's local disk during the transition window (after the sync to green but before blue is fully drained) needs to either continue being captured (an ongoing sync/replication, not just a one-time copy) or be small and low-value enough that losing writes from that narrow window is acceptable, a trade-off that needs to be made explicitly, not discovered after the fact.

Worked example
A service with an in-memory/local-disk session cache: before cutover, session state is migrated to a shared Redis instance both blue and green can read (a change made well before the blue-green cutover itself, as a prerequisite), which removes session-affinity from being a cutover concern at all. For the local-disk cache specifically (assume it's a rebuildable performance cache, not durable data), green is pre-warmed by replaying a sample of recent read traffic against it before cutover, and cutover proceeds as an instant switch once green's cache-hit rate is confirmed close enough to blue's to avoid a latency regression.

Trade-offs and pitfalls
Externalizing session state is real upfront engineering work that pays off on every future deploy, versus a gradual, session-aware cutover which solves THIS deploy but leaves the underlying stickiness problem for the next one; a team doing blue-green deploys regularly should treat session externalization as the real fix rather than repeatedly working around it. The common mistake is treating a stateful service's blue-green cutover exactly like a stateless one's instant switch, discovering only after cutover that in-flight sessions were silently dropped or that green's cold local disk caused a performance cliff nobody validated for.

Direct answer

Policy as code means writing governance and compliance rules (tagging requirements, network exposure limits, allowed instance types) as version-controlled, machine-readable code that a policy engine evaluates automatically, most often against a Terraform plan in CI, instead of relying on a human reviewer to remember and apply a checklist.

Policy engines

• OPA (Open Policy Agent): general-purpose, uses the Rego language, and works against Terraform plan JSON, Kubernetes admission requests, and most other structured input, so one engine covers multiple systems.
• Sentinel: HashiCorp's own policy framework, tightly integrated with Terraform Cloud/Enterprise, with first-class access to plan, state, and configuration data at each Terraform run stage.

A concrete CI policy

"Reject any Terraform plan that creates a security group ingress rule with cidr_blocks containing 0.0.0.0/0 on a port other than 443." In CI: run terraform plan -out=plan.binary, convert to JSON with terraform show -json, and evaluate it with the policy engine (conftest test for OPA, or a Sentinel policy set for Terraform Cloud). A match fails the pipeline before apply runs. This is the kind of check an SRE team's PR-validation workflow would run on every infrastructure PR, with the specific ports and exposure rules tuned to the team's own threat model.

Why code beats a checklist

• Consistent: a reviewer skimming a large diff on a Friday afternoon misses things a machine does not.
• Scales: one policy evaluates every PR from every team; a checklist scales with reviewer headcount.
• Versioned and testable: the policy lives in git, gets reviewed, and can carry unit tests, so a change to the rule is itself auditable.
• Fast, uniform feedback: the author sees the failure in the same CI run, not two days later in a review comment.

Trade-offs & pitfalls

• Policy as code only checks what someone thought to encode; it does not replace judgment for a novel risk a human reviewer might catch from context a machine cannot see.
• An overly broad policy that denies everything it is unsure about trains developers to route around it instead of trusting it. Policies need the same code review rigor as the infrastructure they gate.

Approach summary
Compare scratch, distroless, and minimal distros for compiled Go services across size, security surface, debugging, library deps, and trade-offs, then give best-practice recommendations for production and debugging.

Image size

• scratch: smallest (single binary) — often <10–20MB for a statically linked Go binary + layers.
• distroless: minimal runtime files (ca certs, passwd) — slightly larger than scratch but still small (20–40MB).
• alpine / debian-slim: larger (30–120+MB) depending on packages.

Security surface

• scratch: minimal attack surface; no package manager or shells to exploit. Fewer CVEs, but you must ensure binary itself and build pipeline are secure.
• distroless: low attack surface; includes only runtime artifacts needed (e.g., CA certs).
• alpine/debian-slim: larger surface due to glibc/musl, shells, package managers — more CVEs and patching burden.

Debugging capability

• scratch/distroless: poor — no shell, no strace, limited runtime introspection. Must build debug images or use ephemeral debug sidecars.
• alpine/debian-slim: good — contains /bin/sh, common tools, easier to exec into container for live troubleshooting.

Library dependencies and cgo

• scratch/distroless: best when Go binary is statically linked (CGO disabled). If your service uses cgo or native libraries, scratch may fail — you must include required .so files or use distroless/busybox-based images. Alpine uses musl which can cause subtle differences versus glibc; debian-slim has glibc compatibility.

Trade-offs

• scratch: maximal minimalism and security, but lower observability and trickier startup (must bake certs, user files into image).
• distroless: balanced — small and secure while providing minimal runtime files, still limited debugging.
• minimal distros: easier troubleshooting and compatibility at cost of size and attack surface.

Recommendations (DevOps perspective)

• Production: prefer distroless for Go services that are statically linked; use scratch only if you need absolute minimal size and have CI-trusted reproducible builds. Ensure multi-stage builds, vulnerability scans, and signed images. Disable CGO or vendor required native libs intentionally.
• Debugging / staging: publish separate debug images based on alpine or debian-slim (same binary but linked/packaged with shells and tooling) or use ephemeral debug sidecars and init containers. Keep debug images out of production registries or tag clearly.
• Pipeline best practices: use multi-stage builds, reproducible build flags (CGO_ENABLED=0, -trimpath, -ldflags="-s -w"), image vulnerability scanning, and automated rebuilds when base images are patched.

Direct answer

At the scale of thousands of builds a day across a large, multi-language codebase, the platform needs four things working together: autoscaling ephemeral runners so capacity tracks real demand rather than sitting idle or queuing, dynamic test sharding so any given run's tests are spread evenly across available capacity, shared artifact/cache reuse so repeated work isn't repeated, and explicit multi-tenant fairness so one team's burst of activity can't starve another team's pipeline.

Structured elaboration

Core components:

• Autoscaling ephemeral runners: a scheduler that watches queue depth and provisions runners on demand (scaling to zero when idle), with pre-warmed pools or fast-starting base images to keep cold-start latency from dominating small jobs.
• Dynamic test sharding: rather than a fixed shard count decided once, compute shard assignment per run based on current historical duration data (as covered in the sharding-heuristics survivor), so shard balance stays good even as the suite's composition changes over time.
• Artifact and cache reuse across jobs: a shared, content-addressed cache (keyed by dependency lockfile hash, or similar) so two jobs building the same dependencies don't each pay the full build cost; this needs to be shared across the whole platform, not per-team, to actually pay off at this scale.
• Multi-tenant fairness: without an explicit fairness mechanism, one team's traffic spike (a large batch of PRs, a big refactor triggering many builds) can consume enough shared capacity to delay every other team's builds; a fair-share scheduling policy (per-team quotas, priority weighting, or a fair queuing algorithm) is what prevents that.
• Graceful degradation under load: rather than falling over when demand spikes past provisioned capacity, the system should degrade predictably (queue times increase evenly across tenants rather than one tenant being starved entirely, or a "shed low-priority work first" policy) so a spike is a visible slowdown rather than a systemic outage.

Worked example

At one variant of this scale problem, framed around millions of tests per week: the same architecture applies, just with a scheduler that also needs data locality awareness (routing a job to a runner already holding the relevant cached dependencies, rather than treating all runners as interchangeable) to avoid re-fetching large caches repeatedly across a very high job volume. At another variant, framed around a fixed cost budget and phased rollout: the same components apply, phased in roughly in this order (autoscaling and caching first, since they have the clearest immediate ROI; dynamic sharding next; fairness mechanisms once multiple teams are genuinely contending for the same shared capacity).

Trade-offs & pitfalls

The most common early mistake is under-investing in the multi-tenant fairness piece because it isn't needed until the platform is shared across enough teams to actually contend for capacity; by the time contention becomes visible (one team's builds mysteriously slow during another team's release crunch), retrofitting fairness into an already-live, heavily-used scheduler is considerably harder than designing it in from the start.

Direct answer

At two days of silence with a deadline approaching, keep working the problem in parallel on two tracks: escalate progressively (wider audience, shorter response window) instead of waiting indefinitely or jumping straight to someone's manager, and start a temporary workaround so your own deadline isn't hostage to someone else's response time.

Structured elaboration

1. Reconfirm the ask was clear before escalating. Silence sometimes means the original message was ambiguous or buried, not that it's being ignored. A quick, sharper re-send (what's needed, by when, what breaks if it slips) is worth trying before widening the audience.
2. Widen the channel and audience, not just the volume. Loop in a teammate of the owner's, or their tech lead, with a concise summary: what's blocked, since when, and what you need. This isn't going over anyone's head yet, it's making sure the request isn't sitting unseen in one inbox.
3. Escalate to management if there's still no response, framed around unblocking the work, not blaming the person: bring your own manager or a shared point of contact (like a PM) into a short, direct conversation rather than an open-ended thread.
4. Start a workaround in parallel, not sequentially after escalation: a mock, a stub, or a scoped assumption that lets you keep making progress while the real dependency gets resolved, clearly labeled as temporary so it doesn't quietly become permanent.
5. Close the loop afterward. Once unblocked, note what caused the delay (no on-call coverage, unclear ownership, a channel nobody monitors) so the same two-day silence doesn't repeat next time.

Worked example

Say another team owns a data pipeline, and a schema change they need to ship is blocking your dashboard launch, due in three days. You messaged the pipeline owner two days ago and got no reply.

1. Reconfirm: you send a sharper follow-up in the same thread: "Following up: I need the orders table schema change merged by Thursday EOD to hit our dashboard launch Friday. Anything blocking you on it, or should I loop in someone else?"
2. Widen: a few hours pass with no reply, so you message the pipeline team's tech lead directly (not a reply-all): "I've been blocked on the orders schema change since Monday and our Friday launch depends on it. Can you help me find the right person, or unblock it yourself?"
3. Escalate: by end of day, still nothing, so you bring it to your manager or a shared PM in a short conversation, not a long thread: "I've tried the owner directly and through their lead over two days with no response, and Friday's launch depends on this. Can you help get it unblocked?"
4. Workaround, run in parallel from day one: while those messages are going out, you build your dashboard against a stubbed version of the new schema (a local view with the expected new columns backfilled from sample data), clearly commented as temporary, so the launch timeline doesn't wait on the real merge landing.
5. Close the loop: once the schema change lands, you raise in the team retro that the pipeline team had no on-call coverage for urgent schema requests, and propose a shared "blocked on us" channel so a two-day silence doesn't happen again.

(The same five-step shape applies outside engineering: a designer blocked on a brand asset from marketing, or a QA engineer blocked on a test environment from infra, would reconfirm, widen, escalate, work around, and close the loop the same way.)

Trade-offs & pitfalls

• Pitfall: escalating too fast, before trying a second direct attempt, which can read as skipping over someone unnecessarily.
• Pitfall: waiting too long out of politeness, which puts your own deadline at risk and, in review, looks like you didn't flag a risk early enough.
• Pitfall: treating escalation and workaround as either/or. Doing them in parallel protects the deadline regardless of how fast the escalation resolves.
• Senior differentiator: framing every step (the re-send, the widened ask, the escalation) around getting unblocked, not around who's at fault, so the relationship with the owning team survives the deadline pressure.

Direct answer

The phrases I try to avoid all do the same thing: they shift energy toward defending myself instead of understanding the critique. Things like "well, actually..." or "yeah, but..." or "that's not really my fault" all signal I'm arguing, even if that's not the intent. Instead I reach for language that keeps me listening: acknowledging what landed, and asking a specific question about the part I don't yet understand.

Structured elaboration

Phrases and reactions to avoid. "Well, actually..." (frames the other person as wrong before you've fully engaged with what they said). "Yeah, but..." (technically agrees while immediately undercutting it). "I already thought of that" or "I know" (implies the feedback was unnecessary, even when it wasn't wrong). "That's not really my fault, because..." (redirects to blame before addressing the substance). Silence with a flat expression is its own version of this: it reads as shutting down rather than processing, even if that's not what's happening internally.

What to say instead. "That's fair, let me look into it" acknowledges the point without conceding every detail you haven't verified yet. "Can you say more about which part specifically?" turns a vague or stinging comment into something actionable, and buys a genuine second to think. "I hadn't considered that, give me a moment" is an honest, non-defensive way to signal you're processing rather than ignoring. If you do disagree after actually understanding the point, "I see it differently, here's why, but tell me if I'm missing something" keeps the door open rather than shutting the conversation down.

Worked example

A reviewer says my API design "doesn't handle the failure case well." The instinctive response is "well, actually, I did handle it, just not for that specific scenario," which sounds like disagreement even if it's technically accurate, because it leads with correction. A better response: "can you say more about which failure case you mean?" That question does real work: it might reveal a case I genuinely missed, or a case I did handle but explained poorly, and either way I now know exactly what to address instead of guessing.

Trade-offs and pitfalls

Swapping in softer language without actually changing the underlying reaction is a common trap: saying "that's fair" while internally still building a rebuttal is not real progress, and it usually leaks through in tone anyway. Also, "can you say more" used as a delay tactic every single time, rather than when genuinely needed, starts to read as evasive rather than curious. And going too far the other way, agreeing with everything to avoid seeming defensive, is its own problem: it stops you from ever raising a genuine disagreement when one is actually warranted.