Microsoft DevOps Engineer (Mid-Level) Interview Preparation Guide

Direct answer

Gatekeeper enforces policy as a validating admission webhook: a ConstraintTemplate defines reusable Rego (OPA's policy language) logic and the CRD (Custom Resource Definition) shape it is configured with, and a Constraint is an instance of that template scoped to specific kinds and namespaces. For "deny privileged pods or pods missing resource limits," the template checks spec.containers[].securityContext.privileged and resources.limits.cpu/resources.limits.memory across every container and returns one violation message per offending container. The safe way to ship it is audit-only first, then targeted enforcement, never enforce cluster-wide on day one against an unaudited cluster.

Structured elaboration

ConstraintTemplate: the reusable Rego logic

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredpodsecurityandresources
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredPodSecurityAndResources
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredpodsecurityandresources

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          c.securityContext.privileged == true
          msg := sprintf("container '%v' is privileged", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.cpu
          msg := sprintf("container '%v' is missing resources.limits.cpu", [c.name])
        }

        violation[{"msg": msg}] {
          c := input.review.object.spec.containers[_]
          not c.resources.limits.memory
          msg := sprintf("container '%v' is missing resources.limits.memory", [c.name])
        }

This is Rego v0 syntax, still Gatekeeper's default. Gatekeeper 3.19 and later also supports opt-in Rego v1, which requires an explicit if before each rule body, but v0 remains what ships by default and what most existing ConstraintTemplates use.

Constraint: applying the template

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredPodSecurityAndResources
metadata:
  name: deny-privileged-or-no-limits
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]

enforcementAction: dryrun records violations without blocking anything, the correct starting state for any new policy on an existing cluster.

Validating vs mutating admission, and when to reach for either over OpenAPI schema validation

Gatekeeper is a validating admission webhook: it can accept or reject an object but cannot change it. A mutating admission webhook runs earlier in the chain and can rewrite the object before it is persisted, for example a sidecar injector adding a container, or a default value filled in. Kubernetes ships built-in admission controllers doing exactly these two jobs without any webhook at all, and they are the closest analogues to what Gatekeeper does here: LimitRanger (mutating; injects default resource requests and limits when a pod omits them) and ResourceQuota (validating; rejects a request that would exceed a namespace's aggregate quota).

The choice between a webhook and plain OpenAPI schema validation on a CRD comes down to what the rule needs to know:

• If the rule is fully expressible as a shape constraint on one object in isolation (a field must be one of an enum, a string must match a pattern, a number must sit in a range), OpenAPI schema validation on the CRD costs nothing at admission time and needs no separate service running.
• Reach for a webhook only when the rule needs something schema validation cannot express: cross-field logic (a field is required only if another field has a certain value), cross-object lookups (checking a Secret exists, checking sibling objects against a quota), or a policy that must apply uniformly across many unrelated resource kinds, exactly the "any Pod, any namespace" shape of the privileged/no-limits policy here.

Safe rollout plan

1. Deploy in enforcementAction: dryrun; let it run against real traffic for a full deploy cycle while pulling violations from Gatekeeper's audit results.
2. Share violations with owning teams with the exact fix needed (add resources.limits.cpu/memory, remove privileged: true), not just "you're non-compliant."
3. Flip to enforcementAction: deny first in one low-risk namespace, watch for unexpected rejections, then expand namespace by namespace.
4. Keep excludedNamespaces narrow and explicit, system namespaces only; a broad exclusion list defeats the point of a cluster-wide policy.

Worked example

A pod with two containers: one declares resources.limits: {cpu: "500m", memory: "256Mi"} and passes cleanly; the other has no resources block at all. The template's second and third rules each fire once for the second container, producing two separate violation messages ("missing resources.limits.cpu" and "missing resources.limits.memory"). That per-container, per-field granularity is what makes the audit output actionable rather than a single opaque "pod rejected."

Trade-offs and pitfalls

• Rego policy is powerful but opaque to most application developers; ship it with plain-language violation messages as above, and do not expect teams to read Rego to understand why they were blocked.
• excludedNamespaces is a blunt instrument. Overusing it to unblock a team quickly erodes the policy's coverage silently; track exclusions the same way you would track a firewall exception.
• A validating webhook adds a synchronous hop to every matched request. Keep the Rego evaluation cheap, no external calls, so it does not become the very apiserver latency problem it would otherwise be diagnosing.

A company-wide chaos engineering program scales the practice from an individual engineer running one-off experiments to an organization-level capability with governance, a shared experiment catalog, and a deliberate maturity path, because without that structure, chaos experiments either stay too rare to build real confidence or grow risky enough to cause the outages they're meant to prevent. The program's job is to make running experiments safe, repeatable, and steadily more ambitious over time, not just to run a few high-visibility experiments once.

Governance and approval

• Risk scoring per experiment, based on blast radius, the criticality of the system under test, and whether it targets production or a lower environment, determining what level of approval is required (a team lead for a small staging experiment, a cross-functional review for a production experiment against a critical, customer-facing system).
• A standing approval process, not a one-off request each time, so teams know in advance what tier of experiment they can run under their own authority versus what needs sign-off, which is what makes the program scale past a handful of teams.
• A hard, tested abort mechanism required as a precondition for any production experiment, verified to actually work before the experiment that depends on it runs.

Experiment catalog and risk scoring

• A shared, versioned catalog of experiment types (dependency failure, resource exhaustion, network partition, region failure) that teams can adopt rather than each inventing their own from scratch, with each entry documenting its typical blast radius and required safeguards.
• A risk score attached to each catalog entry and adjusted per target system, so the same experiment type (killing a node) is scored differently against a stateless, redundant service than against a stateful system with a single point of failure.

Escalating blast radius over time

1. Staging first, always, for a new experiment type or a new target system, before it's ever run in production.
2. Small-scope production, a single instance or a small percentage of traffic, in a controlled, low-traffic window, with the team actively watching.
3. Broader, scheduled production experiments (game days), run on a cadence once a system has passed the earlier stages repeatedly, involving the full on-call team as practice for a real incident, not just a single engineer.

Introducing the practice to a low-maturity organization

Start narrow and visible rather than broad and mandated: pick one team and one well-understood system, run a single well-prepared experiment that finds something real (there is almost always something), and use that concrete result to build the case for expansion, the same way a strong reliability-investment business case works. Pushing a full program with mandatory participation before any team has seen a successful experiment tends to generate resistance rather than adoption.

Worked example

An enterprise with several dozen services and strict availability commitments starts its program with one team and one experiment: killing a single replica in a well-redundant caching layer, in staging first, then in production during a low-traffic window. The experiment finds a real gap (client-side connection pooling doesn't recover cleanly from a replica disappearing, causing a longer-than-expected error spike). Fixing that gap and re-running the experiment successfully becomes the case study used to onboard the next three teams, each starting at the same "staging first" stage rather than being handed the full catalog and asked to run production experiments immediately.

Trade-offs and pitfalls

The most common failure mode is skipping the escalation path, running an ambitious production experiment against a critical system before the abort mechanism and the team's response process have been proven at smaller scale, which risks turning the experiment into the very outage it was meant to prevent. A slower-moving but more durable failure mode is over-governing the program until the approval process itself becomes the bottleneck and teams quietly stop proposing experiments; the risk-scoring and approval tiers exist to make the process proportionate to actual risk, not to gate every experiment through the same heavy review regardless of scope. Note that capacity testing (verifying a system handles expected load) is a related but distinct practice from chaos engineering (verifying a system survives unexpected failure); a mature program keeps the two in the same overall resilience-testing calendar without treating them as the same kind of experiment.

Approach

Structure Terraform (a declarative infrastructure-as-code tool: you describe the desired end state in HCL,
HashiCorp Configuration Language, and it computes and applies the difference) around three separate
concerns that are easy to tangle together if you do not separate them from day one: reusable modules that
are cloud-specific underneath but expose a consistent interface, a state backend with real locking so two
engineers can never corrupt the same state file, and a directory-per-environment layout so a change to dev
cannot accidentally reach prod through a shared workspace.

Module design

One module per logical piece of infrastructure (a network, a database, a compute cluster), parameterized so
the same module works across environments by varying inputs, not by copying the module. For a genuinely
multi-cloud module, accept a cloud variable and use count or for_each to conditionally create the
AWS, Azure, or GCP resource, exposing one common set of outputs regardless of which cloud instantiated it,
so callers of the module do not need cloud-specific logic:

variable "cloud" {
  type = string
  validation {
    condition     = contains(["aws", "gcp", "azure"], var.cloud)
    error_message = "cloud must be aws, gcp, or azure."
  }
}
variable "cidr_block" { type = string }
variable "name" { type = string }

resource "aws_vpc" "this" {
  count      = var.cloud == "aws" ? 1 : 0
  cidr_block = var.cidr_block
  tags       = { Name = var.name }
}

resource "google_compute_network" "this" {
  count                   = var.cloud == "gcp" ? 1 : 0
  name                    = var.name
  auto_create_subnetworks = false
}

resource "azurerm_virtual_network" "this" {
  count               = var.cloud == "azure" ? 1 : 0
  name                = var.name
  address_space       = [var.cidr_block]
  location            = "eastus"
  resource_group_name = "rg-${var.name}"
}

output "network_id" {
  value = coalesce(
    try(aws_vpc.this[0].id, null),
    try(google_compute_network.this[0].id, null),
    try(azurerm_virtual_network.this[0].id, null),
  )
}

State backend architecture

Use a remote backend with native locking so concurrent applies fail loudly instead of corrupting state: an
S3 bucket (Amazon S3, AWS's object storage service) with versioning enabled plus a DynamoDB table (AWS's
managed NoSQL database, used here only to hold a single lock record) for locking is the common AWS-hosted
pattern (Azure Storage with a lease-based lock, or Terraform Cloud's built-in locking, work the same way
conceptually).
Split state per environment, never one shared state file for dev, stage, and prod, using a partial backend
configuration filled in at init time:

# envs/prod/main.tf
terraform {
  backend "s3" {} # values supplied via: terraform init -backend-config=backend.hcl
}

# envs/prod/backend.hcl
bucket         = "acme-tfstate-prod"
key            = "envs/prod/network/terraform.tfstate"
region         = "us-east-1"
dynamodb_table = "acme-tf-locks"
encrypt        = true

Secrets in IaC

Never write a secret literal into a .tf or .tfvars file that reaches version control. Reference secrets
by their non-secret identifier (an ARN, Amazon Resource Name, or a resource path) and pull the value at
plan/apply time through a data source, so the secret itself never touches the Terraform source tree:

data "aws_secretsmanager_secret_version" "tunnel_psk" {
  secret_id = "arn:aws:secretsmanager:us-east-1:111111111111:secret:hybrid/tunnel-psk"
}

Terraform state can still contain the resolved secret value in plaintext, so the state backend's own access
control and encryption at rest matter as much as keeping secrets out of source files.

Environment structure

A directory per environment (envs/dev, envs/stage, envs/prod), each with its own backend
configuration and its own .tfvars, calling the same shared modules from modules/, gives every
environment an independent state file and an independent blast radius: a broken plan in dev cannot lock or
corrupt prod's state, and promoting a change from stage to prod is a deliberate terraform apply in a
different directory, not an implicit consequence of a shared workspace switch.

# envs/prod/main.tf
module "prod_network" {
  source     = "../../modules/network"
  cloud      = "aws"
  cidr_block = "10.50.0.0/16"
  name       = "prod-hub-vpc"
}

Key points

Module interfaces stay stable even as the underlying cloud resource changes; state locking prevents the
single most common cause of a corrupted apply, two engineers running terraform apply at once; secrets
referenced by identifier, not embedded, keep the source tree safe to commit even though the state file still
needs its own protection; and directory-per-environment makes "which environment does this change affect"
an answerable question by looking at which directory you ran the command in.

Complexity and edge cases

Terraform's plan/apply cost scales with the size of the state file and the depth of the dependency graph in
that state, not with lines of HCL, so keeping state split per environment also keeps plan times bounded as
the estate grows. Edge cases worth naming: a state lock left stranded by a crashed CI job needs a documented
force-unlock procedure with an approval gate, since force-unlocking blind risks exactly the corruption
locking exists to prevent; a module renamed or refactored can orphan existing resources in state unless
moved blocks are used to tell Terraform the resource's new address; and a count-based conditional
resource, as in the module above, shifts every subsequent resource's index if an earlier one is removed,
which for_each avoids by keying resources on a stable name instead of a position.

terraform init, terraform fmt -check, and terraform validate all pass cleanly against the module and
environment configuration shown above; there are no cloud credentials in this environment, so plan/apply
output is not something this answer can execute or claim.

Direct answer
Pin the current version everywhere, then treat the upgrade as its own change: scan every repository that pins this provider for usage patterns the new major version is known to break, run plan with the new version against a canary environment first and read the plan for forced replacement on anything that shouldn't be recreated, then roll dev to staging to production only after the canary plan is clean and the automated scan across the other repos comes back with nothing unexpected.

Structured elaboration

Pin explicitly before touching anything
Every root module and shared module pins the provider version explicitly in a required_providers block, so an upgrade is always a deliberate, reviewed change to that pin, never something that happens silently on the next init.

Automated detection across many repositories
With the provider pinned across dozens, or hundreds, of repositories and separate state files, manually reviewing each one before staging the upgrade isn't realistic. Run an automated scan across every repo that pins the provider: terraform providers schema -json against the new version diffed against the old one identifies renamed, removed, or type-changed arguments, and a grep-based pass across each repo's .tf files for usage of any argument that diff flags catches the specific modules that will actually break, before a single one of their state files has been touched. Feed the list of known-affected repos to their owning teams as a heads-up before the org-wide rollout starts, rather than everyone discovering it independently when their pipeline breaks.

Canary the upgrade in an isolated environment
Pick a small environment that mirrors production's critical resource types, not full production scale, just resource-type coverage, and run the upgrade there first, entirely through the normal CI pipeline (terraform init -upgrade then plan), never applied locally.

Reading the plan for forced replacement
The plan output is the actual signal, not the changelog: scan the JSON plan for any resource marked for delete-then-create, a replace, when it should have been a no-op upgrade. A schema change that looks cosmetic in the release notes can still force replacement in practice, so trust the plan over the changelog.

Progressive rollout dev to staging to production
Once the canary passes clean and the cross-repo scan shows no other unexpected replacements: dev, then staging, then production, each gated by its own clean plan, with a manual approval step specifically before the production apply, not before dev/staging, to keep velocity on the low-risk stages.

Rollback strategy

• Before any apply that could recreate something: back up state, a versioned backend, or an explicit snapshot for backends without native versioning.
• If the plan itself shows unwanted replacements: don't apply, pin back to the previous version and investigate; nothing has changed yet so there's nothing to roll back.
• If an apply already ran and something was wrongly recreated: restore from the state backup, and if the resource itself was destroyed, restore from its own backup/snapshot, since a state restore alone doesn't undo a real deletion.

Worked example

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.65"   # pinned current version, bumped only via reviewed PR
    }
  }
}

# Back up state before any upgrade-related apply
aws s3 cp s3://infra-terraform-state/prod.tfstate "prod.tfstate.backup.$(date +%s)"

# Run the upgrade against the canary, plan-only, in CI
terraform init -upgrade
terraform plan -out=plan.tfplan
terraform show -json plan.tfplan > plan.json

# Detect forced replacements programmatically instead of reading the plan by eye
jq '.resource_changes[] | select(.change.actions | index("delete") and index("create")) | .address' plan.json

That last command is what actually gates the rollout: any output means stop and investigate before this version goes anywhere near staging or production.

Trade-offs & pitfalls

• Cross-repo scanning catches syntactic incompatibilities, renamed or removed arguments, but not semantic ones, an argument that still exists but now behaves differently; the canary plan is still necessary even after the scan comes back clean.
• A canary environment too small to include a resource type production actually uses is a false green light; match resource-type coverage, not just "some environment exists."
• Coordinating dozens of teams on the same upgrade timeline is a project-management problem as much as a technical one; give affected teams a specific window and a rollback commitment, not an open-ended "upgrade whenever."
• Delaying an upgrade indefinitely because it's inconvenient has its own cost: falling far enough behind on a provider version eventually means jumping multiple majors at once, which is strictly harder than doing them one at a time.

Direct answer

Build trust before you need anything, by being reliable on small things, transparent about your constraints and capacity, and by giving the other person visibility into your world so they aren't surprised later. Waiting to invest in the relationship until you need a favor makes the ask feel transactional.

Framework

Lead with reliability on small things. Deliver on small, early commitments, answer a question promptly, show up to their planning session, so your word has a track record before there's a high-stakes ask on either side.

Be transparent about constraints. Proactively share capacity, risk, and known limitations rather than letting the other person find out the hard way, mid-project.

Give visibility into your world. Invite them into a review or share a roadmap or dashboard, so they understand your constraints without needing you to explain from scratch every time.

Make it reciprocal early. Ask what they need and what's on their plate too. Trust runs both directions, not just from you demonstrating value to them.

Worked example

Situation: a new product manager joins and will depend on your team, for example a platform or infrastructure team, for their roadmap.
Action: in the first couple of weeks, gave the PM read access to the team's capacity and roadmap view along with a short walkthrough, rather than waiting for them to ask. Proactively flagged one known constraint, a piece of infrastructure that was close to capacity, before it affected their planning. Followed through quickly and visibly on a small early request, answering a scoping question the same day, to establish reliability before anything high-stakes came up.
Result: by the time the PM had a genuinely high-stakes ask, an accelerated timeline, there was already a working relationship and a shared understanding of constraints. The conversation started from what's actually possible given what you already know, instead of starting from zero.

Trade-offs and pitfalls

• Trust-building gestures can look like busywork if they aren't tied to something concrete. Keep them small and genuinely useful, not performative.
• Over-sharing every constraint upfront can read as excuse-making before there's even a request. Calibrate to what's actually relevant to their planning.
• The senior differentiator is doing this proactively, before there's a need, rather than scrambling to build rapport only once you need something from the other person, which reads as transactional.

Quick answer

Defend depth by naming the decisions you made and the trade-offs you weighed, not the tools you touched, then back it with one concrete design or debugging story you can go two levels deeper on if pressed.

How to build it

Decisions over tool names

"I used X" is a claim anyone can make after a weekend tutorial. "I chose X over Y because of [a constraint]" is a claim only someone who actually did the work can make convincingly. Reframe the challenged skill as a series of decisions: what you picked, what you rejected, and why.

The depth self-test

Before the interview, for any skill listed on your resume, ask whether you could explain a failure mode of it, not just its happy path, and whether you could describe a debugging session where the obvious fix turned out to be wrong. If the answer is no for something on the list, either prepare that story now or be ready to be honest about the depth of your exposure.

Structuring the defense in the room

1. Name your specific ownership: what part of the system was yours, not the team's.
2. Give one concrete story, a design decision or a debugging session, with the trade-off or root cause named.
3. Invite the harder follow-up rather than hoping it doesn't come; that signals a confidence the resume line alone can't.

When the pushback is fair

Sometimes exposure really was real but shallow, you used the tool inside a system someone else designed. Naming that precisely, what you owned versus what you only touched, holds up better under a skeptical question than overstating and getting caught in the next one.

Worked example

"If someone challenged my depth with [a technology on my resume], I'd point to what I actually owned. On a past team I was responsible for [a specific piece, e.g. how a service handled retries under load]. That meant choosing between [option A, e.g. a simple fixed backoff] and [option B, e.g. an adaptive one], and I picked the one I did because of [a constraint tied to the system, not a generic reason]. A specific example: I once tracked down a failure that looked like [a surface symptom] but turned out to be [the real root cause], found by [the step that actually mattered, e.g. comparing logs across two dependent services rather than trusting the first error message]. Fixing the real cause instead of the symptom is the kind of story a tool name alone can't prove I lived through."

Trade-offs and pitfalls

Listing tools and certifications instead of decisions is the most common way this defense falls apart under real pushback. A second failure is picking a story where you were present but someone else made the calls, if you can't say what you decided, the story defends the team, not you. Overclaiming ownership you don't actually have is the riskiest pitfall: a skeptical interviewer's natural next move is a specific follow-up, and getting caught inflating costs more credibility than an honest "I owned part of this, here's exactly which part" would have.

Direct answer

The risk in migrating a manual runbook into an automated playbook isn't usually the happy path, it's that the human judgment that silently caught edge cases during manual execution disappears the moment the steps become unattended -- so the migration has to explicitly surface and handle what the human was implicitly doing.

Risk controls

Start by explicitly documenting what judgment calls the human executor was actually making at each step (not just the mechanical actions) -- 'check that the error rate looks normal before proceeding' is often an unwritten but critical part of a manual runbook that a naive automation would skip entirely. Build the automation to make the SAME checks explicit and machine-evaluable wherever possible, and where a check genuinely can't be automated reliably, keep it as an explicit human-approval gate rather than silently dropping it.

Testing approaches: dry-run and canary

Dry-run the automated version against real inputs (without executing the actual side-effecting actions) and have the person who used to run this manually review the dry-run output against what they'd have actually done -- this catches cases where the automation's logic diverges from the real judgment the manual process embedded. Once dry-run output looks trustworthy, canary the real automation against a small, low-blast-radius slice (one host, one low-priority case) before trusting it against the full scope the manual runbook used to cover.

Observability to validate automation

Instrument the automated version to emit the same signals a human executor would have implicitly noticed (error rates, unusual values, anything that would have made a careful human pause) as explicit metrics/log events, and compare the automation's outcomes against the historical outcomes of manual runs for a validation period, not just 'did it complete without throwing an exception.'

Feature flags

Gate the automated path behind a flag that can be flipped back to 'require manual execution' instantly if something looks wrong post-rollout, without needing a code deploy to revert -- this is the single cheapest safety net for a migration like this, since the whole point is replacing something that used to have a human safety net with something that initially has less of one.

Human override during incidents

The automated playbook must have an explicit, well-documented way for an on-call engineer to intervene mid-execution -- pause it, take over a specific step manually, or abort and fall back to the original manual process entirely -- because the FIRST time this playbook runs during a real incident under time pressure is exactly when an edge case the automation didn't anticipate is most likely to surface, and 'the automation is stuck and there's no way to intervene' is a strictly worse outcome than the manual process it replaced.

Trade-offs and pitfalls

The most common failure in this kind of migration is treating it as a one-time translation exercise (write the automation, ship it, done) rather than an ongoing validation process -- the automation should run in a shadow/dry-run mode ALONGSIDE the still-manual process for a real validation period before the manual process is retired, not switched over on faith the day the automation first passes its own tests.

Direct answer. Bring online everything that doesn't strictly need the down dependency, and put a firm, visible hold on anything that does, rather than letting it silently degrade or silently retry. Decide what "handling a transaction" means without the dependency in a way that never risks a customer being charged, billed, or committed twice: hold, don't guess. And tell customers proactively, in plain language, before they have to ask.

1. Decide what comes online, using criticality, not convenience

This decision should trace back to the business impact analysis, the process that ranks business functions by how much an hour or a day of downtime actually costs, so it isn't made ad hoc mid-incident. Functions that don't touch the down dependency come online first. Functions that touch it only on a non-essential path (browsing, viewing account history) come online in a read-only or informational mode. Functions where the dependency is essential to the transaction itself (authorizing a new charge) stay explicitly gated, not silently attempted.

The authority to declare "we're operating in a degraded state" and to approve which functions run that way should be defined in the continuity plan ahead of time, not improvised in the room. That's usually someone senior enough to own the customer and regulatory risk of the decision, not just whoever is closest to the outage.

2. Handle the affected transactions without guessing

Accept the transaction, record the customer's intent, and hold it in a clearly marked pending state rather than attempting it against a dependency that isn't there, or silently retrying it in the background where the customer can't see what happened. Be honest about the state: "received and pending" is very different from letting a customer believe it went through, and that distinction is also what stops a customer from trying again themselves out of uncertainty.

When the dependency comes back, process the backlog in order and reconcile before declaring the incident closed: confirm, transaction by transaction, that what your system believes happened matches what the dependency's own records show, and follow up individually on anything that doesn't match rather than assuming the queue drained cleanly.

3. Communicate the degraded state to customers

Say something before they have to ask. A visible status message in the product itself, not only on a status page nobody checks mid-transaction, that names what's affected, sets expectations (their action is saved but pending, not lost), and gives a realistic timeframe, even a wide one, beats silence. Update it as the situation changes, and if any pending items need a customer to take action, reach out directly instead of leaving them to notice on their own. Keep the message consistent across channels so a customer who checks two of them doesn't get two different stories on top of the outage itself.

4. Protect service levels for what's still running

The functions still online need their own expectations reset for the duration. It's reasonable to temporarily relax targets for anything adjacent to the affected dependency (background jobs that would also normally touch it) so they don't cause a second incident by retrying aggressively against something that's down. That should be a deliberate, communicated decision, not something that just happens because nobody planned for it.

Worked example

A checkout flow depends on a third-party payment processor down for two hours, well past what anyone expected. Browsing, cart building, and order history don't touch the processor, so they stay fully online. Checkout is gated: a customer can complete every step through "place order," and the order is recorded as pending payment with the cart and the chosen payment method captured, but no charge is attempted. The product tells them directly: order saved, payment processing is delayed, confirmation will follow once it's back, expected within the hour. No background retry loop fires against the processor. When it recovers, pending orders are processed in the order placed, and each is reconciled against the processor's own transaction record before being marked complete, so a customer is never charged twice even if an earlier attempt is later discovered to have partially gone through.

Trade-offs & pitfalls. The tempting shortcut is to keep retrying the transaction in the background hoping the dependency returns soon; that's exactly how a customer ends up double-charged if the retry succeeds silently after they've already tried again themselves out of frustration. Hold and communicate, don't guess and retry. Silence is worse than an honest "we don't know exactly when," because customers who get no information assume the worst or attempt workarounds that make reconciliation harder afterward. And bringing too much online too fast, without a clear degraded-mode decision from someone with the authority to own that risk, is how a team ends up shipping a feature that looks live but isn't actually safe to depend on.

The key mechanic is consistent-hash routing by trace_id at the load balancer, so every span belonging to a given trace lands on the same collector instance and no cross-collector coordination is needed to assemble a trace before deciding whether to keep it. Memory is then bounded with a fixed decision window plus per-shard and per-trace caps, not by trying to hold every trace indefinitely.

Coordination architecture

flowchart LR
  A[Application Spans] --> B[Load Balancer: hash by trace_id]
  B --> C[Collector Shard 1: buffer]
  B --> D[Collector Shard 2: buffer]
  B --> E[Collector Shard N: buffer]
  C --> F[Sampling Decision Engine]
  D --> F
  E --> F
  F -->|keep| G[Export Full Trace]
  F -->|drop| H[Discard]
  F -->|timeout| I[Partial-Trace Fallback]

Because routing is consistent-hash on trace_id, adding or removing collectors only reshuffles a small fraction of trace-to-collector assignments (standard consistent-hashing property), so scaling the fleet doesn't require a coordinated rebalance of in-flight traces.

Sizing the buffer

Take the stated 100,000 traces/sec, an average of 20 spans/trace (a typical microservice call depth), an average compressed span size of 500 bytes (a labeled assumption), and a 10-second decision window (wait up to 10 seconds after a trace's apparent last span before deciding, which covers the large majority of trace completion times):

traces_per_sec = 100_000
avg_spans_per_trace = 20
avg_span_bytes = 500
decision_window_s = 10
num_collectors = 50

span_rate = traces_per_sec * avg_spans_per_trace                       # 2,000,000 spans/sec
spans_buffered_systemwide = span_rate * decision_window_s              # 20,000,000 spans
bytes_buffered_systemwide = spans_buffered_systemwide * avg_span_bytes # 10 GB
spans_per_collector = spans_buffered_systemwide / num_collectors       # 400,000 spans
bytes_per_collector = spans_per_collector * avg_span_bytes             # 200 MB

At 50 collectors, each instance buffers about 400,000 spans (200 MB), a footprint that fits comfortably in a modest container (2-4 GB), while the system-wide live buffer is about 10 GB spread across the fleet. This is the concrete argument for sharding by trace_id: a single collector holding the full 10 GB buffer would need a memory profile most container platforms would flag as oversized, while 50 shards each holding 200 MB is unremarkable.

Bounding memory beyond the happy path

The 10-second window handles typical traces, but slow or stuck traces need explicit handling so they don't grow the buffer without bound:

• Per-trace TTL: destroy a trace's buffer if no new span arrives within some multiple of the decision window (e.g., 2x), forcing a decision (keep as partial, or drop) rather than waiting indefinitely.
• Per-shard memory cap with eviction: each collector enforces a hard memory ceiling; if exceeded, evict the lowest-priority buffered traces first (e.g., traces with no error/latency signal yet) rather than failing open.
• Global admission control: a lightweight control-plane process aggregates each collector's buffer occupancy and kept-rate on a slow control loop (seconds, not per-request) and adjusts the decision window or sampling probability fleet-wide if the system is trending toward the memory ceiling, rather than each collector reacting in isolation and potentially over-correcting.

Decision logic

• Cheap deterministic rules first (status code indicates an error, latency exceeds a fixed threshold): mark "must keep" immediately without waiting for the full window, since these are unambiguous.
• For everything else, wait out the decision window, then apply either a lightweight scoring model (feature-based, comparing this trace's shape against a recent rolling baseline) or a straightforward probabilistic sample at a rate tuned to the fleet-wide keep-rate budget.
• On TTL expiry before a full decision, export whatever spans were captured as a partial trace rather than silently dropping everything; a partial error trace is still more useful for incident response than nothing.

Trade-offs and pitfalls

• The most common design mistake is trying to coordinate the sampling decision across collectors (e.g., a central service that all collectors ask before deciding); at 2,000,000 spans/sec that coordination service becomes the bottleneck. Consistent-hash-by-trace_id avoids this entirely by guaranteeing the decision can be made locally.
• A fixed decision window is a trade-off, not a free parameter: too short and slow-but-successful traces (a legitimately slow but non-erroring downstream call) get truncated into partial traces; too long and the buffer grows for no benefit on traces that were always going to be dropped.
• Eviction policy under memory pressure needs to bias toward keeping traces that already show error/latency signal; a naive LRU eviction can evict exactly the traces you most want to keep just because they arrived earlier.
• Rebalancing collector count changes which shard owns which traces going forward, but in-flight traces already buffered on their original shard need to either finish there or be explicitly drained; a hash-ring change that silently orphans in-flight buffers loses those traces' decisions.

Direct answer

Treat this as three sequential phases: triage stops active harm and buys time, over days; stabilization fixes the highest-leverage problems without a full rewrite, over weeks; the durable phase installs the ownership and process change that stops the same failure recurring, ongoing. Jumping straight to fixes before triage, or straight to process before things are stable, are the two most common ways this goes wrong.

Structured elaboration

• Immediate triage: stop active harm first, not root cause. Add monitoring or alerting where there's none, pause risky changes if instability is from churn, and put a stopgap on the single highest-frequency failure.
• Medium-term (weeks): fix the highest-leverage share of the causes behind most of the pain, real root-cause work, not a rewrite, and add the missing basics: tests, documentation, a clear ownership map.
• Long-term (ongoing): install the process or ownership change that prevents the same class of problem recurring, an on-call rotation with a real escalation path, a review gate for the kind of change that caused the mess, a recurring health metric with a named owner.
• Throughout: communicate what's stable now, what's still fragile, and what's next, so stakeholders aren't surprised mid-fix.

Worked example

You inherit a service with 6 unplanned outages last quarter, roughly one every two weeks, and no alerting, so every one was first reported by a user. Triage (days 1-3): add basic uptime and error-rate alerting, and roll back the recent deploy pattern correlated with 4 of the 6 outages; over the next four weeks outages drop to 1. Medium-term: the root cause is no staging environment, changes went straight to production; you build one and require a passing smoke test (a quick, basic check that the core paths still work, not full regression coverage) before deploy. Long-term: a standing monthly uptime target with a named owner and monthly review, plus a second reviewer for changes touching the two riskiest components. Six months later the team tracks against that target instead of learning about outages from customers.

Trade-offs and pitfalls

Jumping to a full rewrite during triage is a common overcorrection, it's slower, and you don't yet know what's actually broken versus just old. Treating the stopgap as the fix and never returning to root cause leaves the real risk in place. Too much process for a small team is the opposite failure. And claiming credit for stability that was really just a quiet period, with no way to tell the difference, is exactly why the alerting and health metric matter.