Microsoft Site Reliability Engineer (Entry Level) - Comprehensive Interview Preparation Guide

Direct answer

Containing a long incident means separating stopping the damage from understanding the cause, and doing the first one fast, even with an imperfect fix. Managing stress, mine and the team's, while it drags on means pacing the response deliberately rather than sprinting the whole time, and afterward I pick exactly one concrete change, the one that would have prevented this specific incident, rather than a long list that never gets done.

Structured elaboration

• Containing it: the first move is limiting blast radius (how many users or systems are affected), for example turning off a recently added code path via a feature flag (a runtime toggle) or shedding non-critical load, even before the root cause is understood, since stopping user-facing damage doesn't require a full diagnosis, and waiting for one while damage continues is a choice with its own cost.
• Managing my own stress across a long incident: pace matters more than intensity for anything past the first fifteen or twenty minutes. I deliberately slow my own decision-making once initial containment is in place, since the pressure to move fast is highest exactly when the actual urgency has already dropped after containment.
• Managing the team's stress: for others on the call, I try to be explicit and calm rather than transmitting my own tension, name what's actually still urgent versus what's now stable, and rotate people out of the highest-pressure roles if the incident runs long enough that fatigue becomes a real factor, rather than letting everyone grind the whole time.
• What changed afterward: I resist the instinct to list every possible improvement and instead pick the single change most directly tied to why this specific incident happened and dragged on as long as it did, since a long list of good intentions is much less likely to actually get done than one concrete change with an owner.

Worked example

During an on-call shift, a core service started returning errors for a growing share of traffic. My first move, before I understood why, was containment: I flagged off a recently added code path that touched the failing component, which brought error rates down substantially within a few minutes even though I didn't yet know if that path was the actual cause. That bought time to investigate without users continuing to take the full impact.

The incident still ran well over an hour because the underlying cause, a resource leak, something like memory or open connections that wasn't being released and slowly accumulated, that had been building for days before finally tipping over, took real digging to find. Partway through, I noticed I was rushing my own log reads and re-checking the same query results without really absorbing them, a sign I was pushing past the point where I was actually thinking clearly rather than just moving fast, so I deliberately slowed down, said out loud in the channel that containment was holding and there was no new urgency to rush the diagnosis, and kept working at a steadier pace. For the rest of the team on the call, I gave clear status splits, contained, investigating cause, no current user impact, rather than letting the tone stay at incident-start intensity for the full hour, and when a teammate had been staring at the same dashboard for a long stretch without progress, I asked them to switch to a different angle of investigation rather than grinding on the same dead end.

Afterward, rather than listing every improvement that came up in discussion, I picked the one change most directly tied to why this became an hour-long incident instead of a five-minute one: a leak-detection alert on that specific resource, tuned to fire well before it reached the level that caused user-facing errors, so the next instance of the same underlying issue gets caught during a quiet afternoon instead of turning into another long incident.

Trade-offs and pitfalls

A common mistake is treating containment and root-cause fixing as the same step, trying to fully understand the problem before doing anything to limit damage, which extends user impact for no real benefit. On the stress side, the trap is either grinding at incident-start intensity for the entire duration, which produces worse decisions the longer it runs, or swinging the other way into complacency once things feel contained, forgetting the incident isn't actually over. And on follow-up, listing many good ideas feels thorough but usually results in none of them getting done; naming the one change most tied to the actual failure mode is what survives past the retrospective.

Direct answer
This is a behavioral question best answered with a specific story showing you distinguished a genuine pattern from a one-off, kept the accountability conversation separate from the group postmortem, and led with coaching rather than punishment, while still being clear about the standard expected going forward.

Structured elaboration
A strong answer covers: the situation (what happened, why it warranted an accountability conversation rather than just the standard blameless process), how you determined it was a genuine pattern rather than bad luck, how you had the conversation itself (privately, focused on support and root cause rather than judgment), and the outcome, including how you protected the person's and the team's trust in future incident reporting.

Worked example
"An engineer on my team was involved in a second incident within two months, both traced to skipping the same pre-deploy verification step under time pressure. Before assuming it was negligence, I checked whether this was a genuine pattern: both incidents shared the same specific gap, so it was, rather than two unrelated issues that happened to involve the same person. I had a private conversation, not as part of the group postmortem, focused on understanding what was driving it. It turned out they'd been covering an unsustainable amount of on-call load for a teammate on leave and had started rushing to keep up. I addressed both sides: the workload issue got fixed by redistributing on-call coverage, which was the real systemic driver, and I paired them with a more experienced engineer on deploy discipline for a few weeks as direct support, framed clearly as investment in them, not punishment. Separately, the second incident's actual group postmortem stayed fully blameless and produced an automated pre-deploy gate that makes that specific step impossible to skip regardless of who's deploying, which is the fix that protects the whole team, not just this one person. Three months later there'd been no recurrence, and, just as important, the engineer's incident-reporting behavior stayed just as open as before, which told me the accountability conversation hadn't damaged their trust in the process."

Trade-offs and pitfalls
The most common weak answer either skips the distinction between pattern and coincidence (treating any second incident as automatic proof of a problem) or focuses entirely on the disciplinary angle without addressing what was actually driving the pattern, which misses the more senior insight that repeated mistakes are frequently a symptom of a systemic issue (workload, unclear process, inadequate training) rather than a character flaw.

Direct answer
The circuit breaker pattern stops calls to a dependency that's likely down, converting a slow, cascading failure into a fast, contained one: once failures cross a threshold, the breaker 'trips' and rejects calls immediately for a cooldown window, instead of letting every caller independently discover the outage the slow way (via its own timeout).

Structured elaboration

• States and thresholds: CLOSED (normal, counting failures), OPEN (rejecting immediately once the threshold is crossed within a window), HALF_OPEN (after a cooldown, allow a limited probe to test recovery).
• Half-open probing logic: exactly one (or a small, bounded number of) request is allowed through in half-open state; success closes the circuit, failure reopens it with a fresh cooldown.
• Testing without brittleness under flapping: inject a controllable clock/timer (rather than relying on real sleeps) so tests can deterministically advance past the cooldown window; construct explicit flapping scenarios (fail, recover, fail again) and assert the breaker's STATE after each step, rather than asserting on wall-clock-dependent timing, which is what makes such tests reliably reproducible instead of occasionally failing in CI under load.

Worked example
A test harness with an injectable fake clock drives the breaker through: 3 failures (assert OPEN), a call attempt before the cooldown elapses (assert still OPEN, and assert the wrapped function was NOT invoked), advancing the clock past the cooldown (assert the next call transitions to HALF_OPEN and the wrapped function runs exactly once as the probe), and then separately testing both outcomes of that probe (success closes it, failure reopens it with the cooldown timer reset).

Trade-offs and pitfalls
A test that calls time.sleep() for real to wait out the cooldown is both slow (adding real seconds to every test run) and flaky (a loaded CI runner's actual elapsed wall-clock time can occasionally exceed what the test assumed), which is exactly the brittleness the question is warning about; a fake, explicitly-advanced clock removes both problems and lets you test edge cases (like the exact millisecond boundary of the cooldown) that would be impractical to hit reliably with real timing.

Direct answer

A reusable Terraform module for data infrastructure needs its BOUNDARIES drawn around the resources that genuinely change together as a unit (an S3 bucket plus its own lifecycle policy and encryption config; an EMR/Dataproc cluster plus its own IAM role and security group), with variable design that exposes only what a CONSUMING team actually needs to decide, remote state scoped so a compute cluster's state does not couple to the storage layer's state unnecessarily, and environment-specific differences (sizing, retention, secrets) handled entirely through VARIABLES and per-environment configuration, never through copy-pasted module variants.

Structured elaboration

Module boundaries. Group resources by what actually changes TOGETHER and is meaningfully re-used as a unit: an s3-data-bucket module (bucket, lifecycle policy, encryption, access logging) is one coherent boundary; a spark-cluster module (the EMR/Dataproc cluster itself, its dedicated IAM role, its security group) is another, SEPARATE boundary, even though a given data pipeline consumes both, because a bucket's lifecycle rarely needs to change in lockstep with a cluster's sizing, and coupling them into one module would force every bucket-only change to also touch cluster configuration.

Variable design. Expose the MINIMAL parameter surface a consuming team actually needs to decide (bucket name/prefix, retention period, cluster size, which IAM permissions the cluster needs beyond the module's own sane defaults), keeping everything else (the specific lifecycle-rule structure, the specific encryption algorithm, the specific security-group baseline rules) fixed inside the module so consumers cannot accidentally diverge from the platform's baseline data-governance posture (encryption, retention floors, access logging) the way an unconstrained, fully-parameterized module would allow.

Remote state strategy and locking. Each data-infrastructure COMPONENT (storage, compute) gets its OWN state, following the same per-component isolation principle used generally; a data pipeline's compute layer referencing the storage layer's OUTPUTS (a bucket ARN, for instance) via terraform_remote_state or an explicit variable pass, rather than both living in one combined state file, which would otherwise couple a bucket-retention-policy change's blast radius to the cluster's own apply.

Testing of modules. Module-level integration tests (terratest) provisioning a REAL, ephemeral instance of the module (a genuinely small test bucket, a minimally-sized test cluster) and asserting on its actual configuration (encryption enabled, lifecycle rules present, IAM role scoped correctly), catching a module regression before it reaches any consuming team, run on the module's OWN CI (path-filtered to the module itself), not on every consumer's pipeline.

Handling environment-specific differences and secrets. Environment differences (dev's smaller cluster, prod's stricter retention) are VARIABLE VALUES per environment, never a forked or duplicated module variant; secrets a data pipeline needs (a database credential the cluster reads data from, for instance) are handled through the same referenced-secret pattern used elsewhere, never passed as a raw Terraform variable value that would land in state in plaintext.

Worked example

A concrete module structure for a data pipeline's infrastructure:

modules/
  s3-data-bucket/          # bucket, lifecycle, encryption, access logging
  spark-cluster/           # EMR/Dataproc cluster, its IAM role, security group
pipelines/
  ingestion-pipeline/
    dev/main.tf             # module "bucket" { source = "../../../modules/s3-data-bucket" retention_days = 7 }
                             # module "cluster" { source = "../../../modules/spark-cluster" size = "small" }
    prod/main.tf             # same modules, retention_days = 365, size = "large"

The cluster module's IAM role references the bucket module's OUTPUT (its ARN), passed explicitly as a variable from the consuming pipelines/ingestion-pipeline/*/main.tf (which has access to both modules' outputs within the SAME root configuration), rather than the two modules directly referencing each other, keeping each module independently reusable outside this specific pipeline's combination.

Trade-offs and pitfalls

• Common mistake: bundling storage and compute into ONE module "because this pipeline always uses both together." This couples their state and their change cadence unnecessarily; even when a specific pipeline always pairs them, keeping them as SEPARATE, independently-versioned modules composed together at the pipeline level (as in the worked example) preserves the ability to reuse either one alone in a different context later, and avoids forcing a storage-only change through the compute module's own apply.
• Over-parameterizing a data-infrastructure module (exposing the encryption algorithm, the exact lifecycle-rule shape, as free-form variables) trades platform-team governance for flexibility nobody actually needed; a consuming team should be ABLE to request more permissive settings only through an explicit, reviewed exception, not through an open parameter that quietly lets every consumer set their own baseline.
• Passing a secret value directly as a Terraform variable, even for a "just a connection string" case, lands it in Terraform state in plaintext; a data-pipeline module specifically needs this discipline since data pipelines are exactly the kind of resource that commonly needs database credentials, making this a genuinely common trap for this specific module type.
• Module-level testing costs real money for compute-heavy resources (a test EMR/Dataproc cluster is not free to spin up); sizing the TEST cluster to the smallest viable configuration, and gating the test to run only on changes to the module itself (not on every consuming pipeline's own changes), keeps this cost proportionate.

Direct answer

I treat "getting up to speed" as a series of checkpoints where I test my own understanding against something real, not a quiet study period followed by a reveal. What actually made me useful was checking early and often against people who already owned the domain, and the real signal that I had become genuinely useful was when they started using my output instead of re-deriving it themselves.

Situation, what I did, how I checked it

I was moved onto a project supporting a freight-pricing team after the person who normally handled that relationship left, and I had no background in logistics or freight contracts. In the first week I read the existing pricing agreements and sat in on calls with two carriers, mostly to build a glossary of terms I did not understand, like accessorial charges and fuel surcharges. Rather than waiting until I felt ready, I produced a first draft of a rate analysis by day ten and walked it through with the account lead who did know the domain, asking her specifically to find what was wrong with it. She caught two mistakes: I had treated a seasonal surcharge as a permanent rate change, and I had missed that one lane's pricing was governed by a separate contract entirely. Both were errors that would have looked reasonable to me and obviously wrong to anyone who actually knew freight contracts, which is exactly why I needed that check instead of trusting my own read of the documents.

By week four, the account lead started forwarding pricing questions to me directly instead of answering them herself, which is the signal I actually use for "genuinely useful": not that I felt confident, but that someone who owned the domain started trusting my output enough to stop double-checking it. Learning the domain also changed how I approached the underlying analysis, not just the words I used to describe it. Once I understood that fuel surcharges moved independently of base rates, I restructured the pricing model to track them as a separate line instead of folding them into a blended rate, which is a decision I would not have known to make without the domain context.

Trade-offs and pitfalls

Getting up to speed while still delivering means something gets deprioritized. For me that was breadth: I deliberately went deep on the two carrier relationships that mattered most to the immediate decision and stayed shallow everywhere else until there was time to circle back. The pitfall I watch for is mistaking a plausible-sounding answer for a checked one. Both of my early mistakes sounded reasonable; only a domain owner's review caught them, which is why I build that check in early rather than waiting for the final deliverable to get feedback.

Direct answer

Design serverless observability around three moves: keep the runtime path async and cheap (never block the response to ship telemetry), aggregate by function/version/region instead of by individual invocation, and alert on SLO burn rather than any single failed call. The hard part isn't collecting data, it's collecting it without either slowing down every invocation or blowing up the cost of your metrics backend.

Structured elaboration

Instrumentation on the hot path
Use an OpenTelemetry SDK (or the platform's native equivalent) inside the function, but do the minimum synchronous work: record a span/metric event in an in-memory buffer and return immediately. Never make a blocking network call to a telemetry backend before returning a response, since that adds real latency to every invocation for the sake of observability.

Cold starts (a "cold start" is when the platform has to provision a fresh execution environment before your code can run, instead of reusing one that's already warm) need to be visible without becoming noise: tag each invocation's span with cold_start=true/false and separately record init_duration (time spent provisioning) versus handler_duration (time spent in your code). That split lets you distinguish "the platform is slow to start us" from "our code got slower."

Getting telemetry off the box asynchronously
Because the execution environment can be frozen or torn down right after the response is sent, you can't rely on a background thread finishing a network call after returning. Use the platform's extension/sidecar mechanism (a small companion process that gets a brief extra window to flush) to batch and export whatever accumulated in the buffer, or write to a local queue that a separate always-on collector drains.

Cardinality control
The single most common failure mode here is tagging metrics with invocation_id. Each invocation has a unique ID, so that turns every metric into its own unlimited-cardinality time series, which most time-series backends handle very badly and bill for expensively. Keep invocation_id in logs and sampled trace data only (where it's a value inside a document, not an index key); aggregate metrics by function_name, version, region, and cold_start (a handful of distinct values, not millions).

Sampling for traces
Use tail-based sampling (the decision to keep a trace is made after it completes, based on what actually happened) rather than a flat random rate: always keep traces that errored or had a cold start, and keep a small fixed fraction of ordinary successful ones. That way rare failures aren't statistically diluted away.

Alerting
Alert on SLO burn rate (how fast you're consuming your error budget relative to a time window) and on aggregate error-rate or cold-start-ratio deviations sustained over a few minutes, not on any single invocation's outcome. A serverless fleet at any real scale will always have some function fail on some invocation; alerting on that individually guarantees pager fatigue.

Worked example

Take a function running at 35 events/sec (about 3,000,000 invocations/day) with a measured cold-start probability of 12%.

Expected cold starts per day:

Now compare two tagging strategies for the same day of data:

• Tag by invocation_id: up to 3,000,000 distinct series per metric per day for this one function alone.
• Tag by function_name + version + cold_start: at most a handful of series (one version deployed, two cold_start values) no matter how many invocations occur.

The second approach gives you the exact same aggregate signal (cold-start ratio, p99 duration split by cold vs warm) at a bounded, predictable series count, while the first is unqueryable in practice and will get throttled or dropped by most backends.

flowchart LR
  A[Function invocation] --> B[In-process buffer]
  B --> C[Async extension / sidecar collector]
  C -->|batched export| D[Telemetry pipeline]
  D --> E[Time-series backend]
  D --> F[Log store]
  D --> G[Trace store]
  E --> H[SLO burn-rate alerting]
  G --> I[Tail-sample: keep errors + cold starts]

Trade-offs & pitfalls

• Tail-based sampling needs somewhere to hold the trace until it completes, which adds a small amount of buffering complexity compared to head-based (decide-at-start) sampling; for very high-throughput fleets this buffering layer itself needs to be sized and monitored.
• If the extension/sidecar flush window is too short relative to the platform's freeze behavior, telemetry from the tail end of an invocation can be silently dropped; verify the flush actually completes before the environment freezes, don't assume it does.
• It's tempting to alert on absolute counts ("more than 5 errors in a minute") which behaves inconsistently as traffic scales; rate-based and burn-rate-based thresholds hold up better across low-traffic and high-traffic periods.
• Treating every invocation as independently alert-worthy is the single most common mistake teams make moving from long-lived services to serverless; a fleet-level view (aggregated by function and version) is what actually reflects user impact.

Approach

A retry decorator needs to separate three concerns cleanly: which exceptions are worth retrying (the predicate), how long to wait between attempts (backoff+jitter), and what to do on each attempt (logging, and eventually giving up). Keeping these as parameters rather than hardcoding them is what makes the decorator reusable across call sites with very different retry needs.

import functools
import logging
import random
import time

def retry(max_attempts=5, base_delay=0.5, max_delay=30.0,
          retryable=(Exception,), logger=None):
    """Retry a synchronous function with full-jitter exponential backoff.

    retryable: a tuple of exception types, OR a callable(exc) -> bool that
               classifies whether a given exception is worth retrying.
    """
    def is_retryable(exc):
        if callable(retryable) and not isinstance(retryable, type):
            return retryable(exc)
        return isinstance(exc, retryable)

    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            attempt = 0
            while True:
                attempt += 1
                try:
                    return fn(*args, **kwargs)
                except Exception as exc:
                    if not is_retryable(exc) or attempt >= max_attempts:
                        raise
                    ceiling = min(max_delay, base_delay * (2 ** (attempt - 1)))
                    delay = random.uniform(0, ceiling)  # full jitter
                    if logger:
                        logger.info("attempt %d/%d failed (%r), retrying in %.2fs",
                                     attempt, max_attempts, exc, delay)
                    time.sleep(delay)
        return wrapper
    return decorator

Verified in a sandbox: wrapping a function that raises ValueError twice then succeeds, @retry(max_attempts=4, retryable=(ValueError,)) returns the correct result after exactly 3 attempts; wrapping a function that always raises, it makes exactly max_attempts attempts and then re-raises the original exception rather than swallowing it.

Idempotency and the decorator

The decorator itself has no idea whether the wrapped function is safe to call twice -- that judgment has to be made by whoever applies it. Two consequences: (1) the retryable predicate should exclude exceptions that indicate the operation may have partially succeeded in an ambiguous way (a timeout on a POST is the classic ambiguous case: did the server process it and the response got lost, or did it never receive the request?), and (2) for genuinely non-idempotent external calls, the caller should generate an idempotency token before the first attempt and pass the same token on every retry, so the server-side API can deduplicate. The decorator's job is to retry; the idempotency token's job is to make retrying safe -- they're separate concerns that have to be composed correctly by the caller, not something the decorator can enforce on its own.

Trade-offs

A class-based retry policy (rather than a decorator) is worth it once you need per-call overrides (retry THIS call with a shorter window because it's on a critical path) -- decorators are static at definition time unless you thread configuration through explicitly.

Edge cases: a wrapped function called with no arguments, a function whose exception has a non-standard __repr__ that could itself throw during logging, and max_attempts=1 (which should behave as a single unretried call, not loop) are all worth explicitly testing -- the last one is a common off-by-one where an implementation accidentally still retries once even at max_attempts=1.

Direct answer

Match the pricing model to how predictable and interruptible the workload is: pay full price (on-demand) for unpredictable or short-lived work, commit to a discount (a reserved instance or savings plan) for the steady baseline you know you'll run for a year or more, and reach for the deep spot or preemptible discount only when the work can be killed and resumed without harm. Real workloads mix all three: a steady floor on committed pricing, a growth buffer on on-demand, and interruption-tolerant batch work on spot.

Structured elaboration

When each fits

Mixing for one workload

Split the workload into a steady floor, a variable middle, and interruption-tolerant batch, and price each piece with the model that fits it, rather than choosing one model for the whole workload.

Worked example (illustrative unit prices, not vendor-quoted rates)

Assume on-demand costs $0.10 per instance-hour, a committed baseline (reserved instance or savings plan) costs $0.06 per instance-hour (40% off), and spot averages $0.03 per instance-hour (70% off, an illustrative average across reclaim risk).

Baseline: 20 instances running 24/7 (730 hours/month):

Variable batch work: 10 additional instances running 8 hours/day on average (240 hours/month) on spot:

Blended monthly cost with the mix versus all on-demand:

Trade-offs & pitfalls

• Never put a single-master, stateful database on spot; there is no fallback for the primary if it is reclaimed mid-write.
• Overcommitting reserved or savings capacity beyond your actual steady floor turns a discount into waste; size the commitment to your measured baseline, not your hoped-for growth.
• Spot needs operational work to be safe: checkpointing, diversified instance types and zones so a single reclaim wave doesn't take out your whole batch fleet, and an automatic fallback to on-demand.
• Managing multiple pricing commitments has its own administrative overhead (tracking utilization, renewal timing); factor that into whether the discount is worth it at small scale.

Planning capacity for 4x growth against an existing SLO means modeling headroom, redundancy, and auto-scaling policy TOGETHER, since each one changes how much of the other two you actually need.

Structured elaboration

Headroom modeling: project current peak utilization forward at 4x traffic and identify at what point specific resources (CPU, database connections, a specific downstream dependency) hit saturation, using historical scaling curves rather than a naive linear assumption (some components scale sub-linearly with good caching, others hit a hard ceiling well before 4x, like a single-writer database). Redundancy: decide the minimum number of healthy instances needed to absorb both routine failure AND the increased load, since redundancy sized for TODAY'S traffic won't be adequate once volume quadruples even if failure rates stay the same. Auto-scaling policy: define scale-out triggers with enough lead time to actually provision new capacity before saturation hurts the SLO (if scaling takes 5 minutes to provision, the trigger threshold needs enough headroom below the true ceiling to absorb 5 minutes of continued growth). Cost trade-offs: more aggressive headroom (scaling earlier, keeping more idle capacity) costs more in steady-state infrastructure spend but reduces SLO risk during unexpected bursts; the SLO target itself should directly inform how much of that cost is justified (a looser SLO can tolerate thinner headroom; a tight, revenue-critical SLO usually can't).

Worked example

If current peak CPU utilization is 60% and a specific downstream dependency (a shared database) shows early saturation signs (rising queue depth) at around 2.5x current load, that dependency, not raw compute, is the actual first bottleneck at 4x growth, and the capacity plan needs to prioritize addressing IT specifically (e.g. read-replica scaling, a caching layer) well before general compute headroom becomes the limiting factor. Auto-scaling triggers might be set to scale out at 50% utilization (rather than a more traditional 80%) specifically because the SLO's tight p95 latency target leaves little tolerance for the brief period between trigger and new-capacity-online.

Trade-offs and pitfalls

The most common mistake in this kind of exercise is modeling headroom for the OBVIOUS bottleneck (compute) while missing a specific, non-obvious dependency that actually saturates first at a much lower multiplier; a thorough capacity plan needs to trace the ACTUAL request path end-to-end and identify each component's individual scaling ceiling, not just the aggregate service's. It's also easy to under-model the LEAD TIME between an auto-scaling trigger firing and new capacity actually being usable (image build, warm-up, connection-pool ramp-up all add real delay), and setting the trigger threshold without accounting for that lead time can mean the scale-out arrives too late to prevent the SLO breach it was meant to avoid.

ImagePullBackOff is the pod status shown when the kubelet has repeatedly failed to pull a container's image and is now backing off further attempts; like CrashLoopBackOff, it names the symptom, not the cause. The four usual causes are registry authentication, DNS resolution to the registry, a wrong image name or tag, and network reachability from the node to the registry.

Causes and their signatures

Representative event text you'd actually see:

Failed to pull image "myrepo/app:v2": rpc error: code = NotFound desc = failed to pull and unpack image "myrepo/app:v2": failed to resolve reference "myrepo/app:v2": myrepo/app:v2: not found

Failed to pull image "123456789012.dkr.ecr.us-east-1.amazonaws.com/app:v1": rpc error: code = Unknown desc = failed to authorize: 401 Unauthorized

Step-by-step: cluster, node, or registry?

1. Read events first.

kubectl describe pod <pod> -n <ns>
kubectl get events -n <ns> --sort-by=.metadata.creationTimestamp

2. Check the pull secret and service account actually being used.

kubectl get sa <sa> -n <ns> -o yaml
kubectl get secret <secret> -n <ns> -o yaml

3. Test from inside the cluster (isolates cluster-level DNS/NetworkPolicy problems from node-level ones):

kubectl run -i --rm debug --image=alpine --restart=Never -- sh
# inside: apk add --no-cache curl bind-tools; nslookup registry.example.com; curl -v https://registry.example.com/v2/

4. Test from the node itself. Since dockershim was removed in Kubernetes 1.24, the node-level container runtime on essentially every current cluster is containerd (or CRI-O), so the right tool is crictl, not docker. docker pull only applies if you're intentionally still running Mirantis cri-dockerd or looking at a pre-1.24 node.

kubectl get pod <pod> -o wide -n <ns>
# on the node:
sudo crictl pull <image>
sudo journalctl -u kubelet -e

5. Decide scope:
   • Registry-related: the error is 401/403/manifest unknown both from the debug pod and from the node directly.
   • Cluster-level: the in-cluster debug pod can't resolve or reach the registry, but the node itself can (curl succeeds from the node); points at CoreDNS or a NetworkPolicy blocking egress.
   • Node-level: only some nodes fail; others pull the same image fine. Points at a node-specific proxy, MTU, or firewall rule.

Two variants worth naming explicitly

Registry rate limiting across many pods at once. If ImagePullBackOff appears on many unrelated pods across the cluster at roughly the same time, rather than one pod, suspect the registry itself: unauthenticated pulls against a public registry are commonly rate-limited (429 Too Many Requests or an equivalent 401 once the anonymous quota is spent), and every node hitting that same registry trips it simultaneously. The fix is authenticated pulls, a pull-through cache/mirror, or spreading pulls out over time, not chasing one pod's config.

ECR token expiry as a time-correlated failure. On EKS Amazon Elastic Kubernetes Service, if pulls are authenticated with a static, manually-refreshed token stored in an imagePullSecret rather than through IRSA (IAM Roles for Service Accounts), that token is only valid for 12 hours. Pulls succeed for a while and then start failing cluster-wide at almost exactly the token's issuance time plus 12 hours, which is the tell that distinguishes this from a one-off misconfiguration: correlate the failure's onset timestamp against when the credential was last refreshed. The fix is to stop relying on a static Secret and either use IRSA so kubelet-side credential retrieval is automatic, or run a credential-refresh mechanism (such as the ECR credential helper or a scheduled Secret-rotation job) instead of a manually maintained token.

Trade-offs and pitfalls

• Don't assume "it's the registry" just because the error message is generic; a node-level DNS or proxy issue can produce an error that looks identical to a registry outage.
• A fix that works from a debug pod but not from the node (or vice versa) is itself diagnostic information: it tells you the failure is layer-specific, not systemic.
• Long-lived static pull secrets are an operational trap even when they work today; anything time-boxed (ECR tokens, expiring service-account tokens) will eventually fail in a way that looks like a random new bug unless the expiry is tracked explicitly.