Microsoft Site Reliability Engineer - Mid-Level Interview Preparation Guide

CrashLoopBackOff means the container keeps exiting and the kubelet is backing off its restart attempts with exponential delay (10s, 20s, 40s, up to a 5-minute cap). It is a symptom, not a root cause: something inside the container is exiting, or something outside it is killing the container, and the job is to figure out which.

Step-by-step diagnosis

1. Describe the pod first. This shows the Events list and the container's last termination reason before you even look at logs.

kubectl describe pod <pod> -n <ns>

2. Read current and previous logs. The -p (previous) flag is what most people forget, and it's the only way to see output from the crashed instance once it has already restarted.

kubectl logs <pod> -c <container> -n <ns>
kubectl logs <pod> -c <container> -p -n <ns>

3. Check the exit code, visible in describe's Last State block.

4. Separate "app crashed" from "kubelet killed it." A liveness probe failure produces its own event and a different signature than an application-level crash:

Containers:
  app:
    State:      Waiting
      Reason:   CrashLoopBackOff
    Last State: Terminated
      Reason:   Error
      Exit Code: 1
    Restart Count: 5
Events:
  Warning  Unhealthy  90s (x3 over 3m)   kubelet  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": dial tcp 10.1.2.3:8080: connect: connection refused
  Warning  BackOff    30s (x12 over 6m)  kubelet  Back-off restarting failed container

If the events show Liveness probe failed right before the restart, the app may actually be starting fine but too slowly, or the probe is pointed at the wrong port or path, and kubelet is killing a container that would otherwise have recovered.
5. Check image and config next, since a large share of CrashLoopBackOff cases are not code bugs: wrong environment variable, a ConfigMap/Secret key that doesn't exist, a missing volume mount, or an image tag that doesn't match what's expected.

Reproduce and test a fix safely

• Recreate the same image/env in a disposable pod without a restart policy fighting you:

kubectl run debug --image=<image> --restart=Never -n dev --env="KEY=value" --command -- sleep 3600
kubectl exec -it debug -n dev -- /bin/sh

• If you suspect the liveness probe is the culprit, don't leave it disabled permanently; temporarily loosen initialDelaySeconds or failureThreshold in a copy of the manifest in a non-prod namespace, confirm the app actually stabilizes, then fix the probe config for real rather than shipping it disabled.
• Once you have a fix, patch and roll it out normally rather than editing the live object by hand:

kubectl set image deploy/<d> <c>=<image>:<new-tag> -n <ns>
kubectl rollout status deploy/<d> -n <ns>

• Capture the failing pod's YAML before you touch anything (kubectl get pod <pod> -o yaml), since the evidence disappears once the pod is replaced.

Trade-offs and pitfalls

• Don't "fix" a crash loop by disabling the liveness probe or setting restartPolicy more leniently; that hides the symptom and turns a visible failure into a silent one.
• Exit code 137 is a strong hint but not proof of OOMKilled; always confirm via the Reason field in describe, since a probe-triggered kill or an external kill -9 produces the same code.
• If a pod never produces logs at all before crashing, the failure is likely before your application's logging even initializes (missing dependency, bad entrypoint, immediate segfault); at that point kubectl logs -p is empty and you have to fall back to the exit code, describe events, and possibly running the image manually to see stdout that never reached the log driver.

Situation/Goal: Reduce MTTR by 30% in one quarter without hiring, using learning/knowledge interventions.

Prioritized interventions (with estimated impact, timeline, owner):

1. Targeted runbooks for top 10 incident types — High priority
   • Impact: 12–18% MTTR reduction (most incidents fixed faster)
   • Timeline: 2–4 weeks to draft + 1 week pilot
   • Owner: on-call leads + SME
2. War-room drills (table-top + live simulation) twice/month for critical services
   • Impact: 6–10% (faster diagnosis, fewer escalation cycles)
   • Timeline: start week 3, ongoing each 2 weeks
   • Owner: SRE lead + rotation of engineers
3. Postmortem knowledge-sharing + short micro-training (20–30min sessions) for recurring failure modes
   • Impact: 4–6%
   • Timeline: weekly sessions starting week 2
   • Owner: incident reviewer
4. Fast-access playbook UX: searchable KB, runbook templates, pager message templates
   • Impact: 3–5%
   • Timeline: 3–5 weeks
   • Owner: Documentation champion + infra engineering
5. On-call checklist + decision tree badges (visual aids)
   • Impact: 1–2%
   • Timeline: 1–2 weeks
   • Owner: On-call coordinator

Validation & causality strategy:

• Baseline: compute per-incident-type MTTR over prior 8–12 weeks and volume-weighted aggregate.
• Phased roll-out + control: deploy runbooks for a subset of services first (A group) leaving B group as control; run drills for teams A, not B initially.
• Metrics: median and P95 MTTR, time-to-detect, time-to-ack, number of escalations, successful runbook use rate.
• Statistical tests: use interrupted time series and difference-in-differences (A vs B) to estimate causal effect; bootstrap confidence intervals for median reductions.
• Instrumentation: add a "runbook used" tag and automated event markers when playbooks are executed to attribute fixes.
• Confounder checks: control for incident mix, deploys, traffic spikes; exclude major outages unrelated to interventions.
• Continuous feedback: weekly dashboard, rapid iteration on low-ROI runbooks, and retest.

Expected result: combining these interventions yields >30% MTTR drop within quarter, with validated attribution via A/B and ITS analysis.

For a brand-new customer-facing microservice, the process matters as much as the numbers: getting the right stakeholders in the room before committing to a target avoids the two most common failure modes of setting a target that nobody can actually deliver, or one that nobody actually needed.

Structured elaboration

Stakeholders to involve: the engineering team building the service (technical feasibility), product/business (what does the user-facing experience actually require, tying back to a real business goal), and whoever owns any existing customer-facing SLA the new service might roll up into. Candidate SLIs should be proposed from the golden-signals framework (latency, traffic, errors, saturation) adapted to this specific service's actual failure modes, not copied wholesale from an unrelated service. SLO targets and error budgets should start deliberately conservative given no historical telemetry (see the bootstrap-a-new-service discussion), and the validation process after launch should include a defined observation period (2-4 weeks of real traffic) before the initial target is either confirmed or tightened.

Worked example

For a new "order-status" microservice: stakeholders are the owning team, the product manager who defined the "customers should see accurate order status within seconds" requirement, and the platform team whose existing customer SLA this service's reliability rolls up into. Candidate SLIs: request success rate and p95 latency, both instrumented from day one with no enforced target initially. After a 3-week observation period showing p95 stabilizing around 120ms and 99.8% success, the team sets the actual SLO at p95 < 200ms / 99.7% success, both with real headroom below observed performance, and documents the validation process (what was observed, what target was chosen and why) so the decision is auditable later.

Trade-offs and pitfalls

Skipping the stakeholder step and letting engineering set the target unilaterally often produces a technically-reasonable but business-irrelevant number (precisely calibrated to what's easy to achieve, not to what users or the business actually need); skipping the observation period and committing to a target immediately at launch, as covered elsewhere, risks locking in a target the service can't yet sustain. It's worth being explicit, in writing, about the SLO's provisional status during the observation period, so nobody downstream treats the placeholder number as a final commitment before it actually is one.

Direct answer

Notice it because you are genuinely poking at something out of curiosity, not because you were assigned to look, quantify how big the issue actually is before raising it, fix or flag it before it becomes visible to stakeholders as a bigger problem, and share the finding so it becomes a repeatable check rather than a one-off catch.

Structured elaboration

1. What made you look: usually a small, mildly unusual thing noticed while doing something else, browsing a dashboard, exploring data for an unrelated task, not a formal audit.
2. Confirm it is real and size it: check whether the odd thing is a genuine issue and roughly how big before spending more time or raising it, so you do not chase noise.
3. Act inside your own access: a fix or a clear flag you can do without a formal ask, since it is still small at this stage.
4. Turn the one-time catch into a repeatable check where possible, so the next instance does not depend on someone happening to notice again.

Worked example

While casually checking a cloud billing dashboard out of curiosity, not part of any assigned task, compute spend appeared to be creeping up gently week over week for about a month, with no matching increase in traffic or usage. Tracing it further led to a set of autoscaled compute instances from a finished experiment that had never been torn down, since the experiment's shutdown script only removed the primary resources and missed a secondary group. Checking their utilization over the prior week confirmed the instances, four in total, had been idle for the same roughly three-week stretch the spend had been climbing, so they were shut down, cutting an estimated $450 of unnecessary spend for that stretch, based only on the observed idle window rather than a projected annual figure. A weekly automated check was then added that flags any compute tagged to a completed experiment still running, so this class of leftover resource gets caught going forward instead of by someone noticing a slow creep on a dashboard.

Trade-offs and pitfalls

A common wrong turn is raising an alarm before confirming the thing is actually a real, sized problem, which spends other people's attention on noise. Another is fixing it and moving on without turning it into a repeatable check, so the exact same leak recurs with the next experiment. Also watch for over-claiming the savings; report only what you can actually verify, instances found idle, roughly how long they had been running, rather than projecting a large annualized figure from a short observation window.

Direct answer
Quantitative evidence (metrics, logs, traces) tells you what happened and when with precision but can miss context and intent; qualitative evidence (engineer interviews, notes, chat logs) fills in the why and the human decision-making, but is subject to memory bias and self-justification. Weigh them together, and when they conflict, treat the disagreement itself as a finding worth investigating rather than picking whichever is more convenient.

Structured elaboration

• Quantitative evidence is precise and timestamped, which makes it the backbone of any timeline, but it can be silent on intent and context: a metric shows latency spiked at 14:03, but not why an engineer chose to deploy at that specific moment or what they believed was true when they did.
• Qualitative evidence captures reasoning and context that logs can't ("I deployed because the dashboard looked fine and I didn't know about the downstream dependency"), but human memory reconstructs events after the fact, often unconsciously smoothing over uncertainty or minimizing one's own role, so it should never override hard timestamped data when the two genuinely conflict.
• Correlating them into one timeline: anchor the timeline on quantitative events (deploys, alerts, metric changes) first, since those are objective and timestamped, then layer qualitative context alongside each event (what the engineer believed, what they were looking at, why they made a given call) as annotation, not as competing facts.
• When they conflict: if an engineer recalls checking a dashboard that logs show wasn't accessed, that's not necessarily dishonesty, memory under stress is genuinely unreliable, but it IS worth investigating why the gap exists: was there a different dashboard, a misremembered timestamp, or a real gap in what was actually checked before the decision was made. The conflict itself, not just its resolution, is often informative about where the process broke down.

Worked example
An engineer recalls seeing a warning-level alert before deploying and deciding it looked minor enough to proceed. Logs show no alert fired until four minutes after the deploy. Rather than concluding the engineer is simply wrong or dismissing the recollection, the investigation digs further and finds the engineer was actually looking at a stale, cached view of the dashboard that hadn't refreshed in several minutes, itself a real and separately worth-fixing gap (a dashboard that can silently show stale data during exactly the moment it matters most). The quantitative record established what actually happened; the qualitative account, once reconciled rather than dismissed, revealed a genuine, previously-unknown contributing factor that the logs alone would never have surfaced.

Trade-offs and pitfalls
The most common mistake is treating quantitative data as always authoritative and qualitative accounts as merely decorative color, which misses genuine contributing factors that only surface through human context. The opposite mistake, treating a confident personal recollection as more reliable than the logs when they conflict, risks building the postmortem's conclusion on a memory distortion. The discipline is to anchor on timestamped data but take conflicting qualitative accounts seriously enough to investigate the gap, not dismiss either source reflexively.

Direct answer

An enterprise-scale, highly-available Jenkins installation needs controller HA (so a single controller failure doesn't take down the whole platform), agent capacity that scales independently of the controller, a disciplined plugin-management policy (since the plugin ecosystem is both Jenkins' greatest strength and its biggest operational risk), and security hardening appropriate for a system many teams depend on. Separately, if the controller starts showing performance problems under load, that's a distinct operational-troubleshooting skill from the architecture itself.

Structured elaboration

Controller HA. Jenkins' controller is not naturally distributed the way many modern systems are; the practical HA options are active-passive failover (a standby controller that takes over if the primary fails, using shared, replicated storage for JENKINS_HOME) or a robust backup-and-restore process with a defined, tested recovery time objective. JENKINS_HOME (which holds job configuration, build history, and credentials) needs to live on durable, ideally replicated storage, since losing it is losing the platform's entire state, not just uptime.

Scaling build agents. Agents should scale independently of the controller: a Kubernetes-based agent pool (ephemeral pod-per-build agents, autoscaled by the Kubernetes cluster) or cloud autoscaling groups for VM-based agents, following the same autoscaling design principles (leading indicators, pre-warmed capacity) discussed elsewhere. This decoupling matters because agent capacity needs are driven by build volume, while controller capacity needs are driven by job configuration count and orchestration overhead, and conflating them means over- or under-provisioning one to compensate for the other.

Plugin management. Given the plugin ecosystem's size, a deliberate policy is required: vet new plugins before adoption (community support, maintenance activity, security history), pin specific versions rather than always auto-updating, test upgrades in a non-production Jenkins instance before rolling out broadly, and actively minimize plugin count, since every installed plugin is both a maintenance burden and a potential attack surface.

Security hardening. SSO and role-based access control for the controller (rather than Jenkins' own basic auth), script-approval sandboxing (so an untrusted Groovy script from a pipeline can't execute arbitrary code on the controller without explicit admin approval), agent isolation (containerized or otherwise sandboxed, so a compromised build can't reach the controller or other agents), network segmentation between the controller/agents and the rest of the internal network, and audit logging of administrative actions.

Diagnosing controller GC pauses and high CPU. This is a distinct, concrete troubleshooting skill: profile the JVM (heap dumps, GC logs, a profiler attached to the running controller) to identify whether the pressure comes from a specific plugin's memory usage, an excessive number of configured jobs each holding in-memory state, or genuinely undersized heap for the controller's load. Mitigations include JVM tuning (heap size, garbage collector choice), auditing and potentially removing or replacing a misbehaving plugin, offloading work that doesn't need to run on the controller to agents, and reducing per-job overhead by moving toward lightweight, pipeline-as-code patterns (multibranch pipelines defined in Jenkinsfiles) instead of many individually-configured freestyle jobs, which each carry more controller-side overhead.

Worked example

An enterprise Jenkins deployment: the controller runs active-passive with JENKINS_HOME on replicated network storage and a tested failover runbook targeting a defined recovery time; build agents run as ephemeral Kubernetes pods, autoscaled on queue depth; a plugin review board vets and pins plugin versions, testing upgrades in a staging Jenkins instance monthly; SSO via the organization's identity provider controls controller access with RBAC (role-based access control). When the controller later shows sustained high CPU and long GC pauses under peak load, profiling reveals a specific plugin holding excessive per-job in-memory state; the team downgrades that plugin pending a fix, converts the highest-job-count teams from freestyle jobs to multibranch pipelines to reduce controller-side overhead, and increases the controller's heap allocation as a near-term mitigation while the longer-term plugin and job-pattern changes roll out.

Trade-offs and pitfalls

The most common architectural mistake is coupling agent scaling to the controller (running agents as static, manually-provisioned machines the controller directly manages) instead of an independently-scalable pool, which means agent capacity can't grow without controller-side reconfiguration effort. The most common operational mistake is treating plugin updates as routine and low-risk, applying them directly to production without testing in a separate instance first, which is a frequent source of exactly the kind of GC-pressure or stability regression the troubleshooting scenario describes.

/etc/passwd — purpose: account metadata (not passwords). Sample line:
alice:x:1500:1000:Alice Example:/home/alice:/bin/bash
Fields:

1. username, 2) password placeholder (x or *) → real hashes in /etc/shadow, 3) UID, 4) GID (primary group), 5) GECOS/full name/comment, 6) home directory, 7) login shell.

etc/shadow — purpose: secure password hashes and aging. Sample:
alice:$6$abcd...$hash:18765:0:90:7:14:::
Fields:

1. username, 2) password hash (or !/* for locked), 3) lastchange (days since epoch), 4) min days between changes, 5) max days, 6) warn days before expiry, 7) inactive days after expiry, 8) expiration date (days since epoch), 9) reserved.

/etc/group — purpose: group definitions. Sample:
developers:x:1000:alice,bob
Fields:

1. groupname, 2) password placeholder (x), 3) GID, 4) comma-separated member list (supplementary groups).

Commands to create user alice with UID 1500, primary group developers, home /home/alice, set password, configure aging, lock/unlock:

• create group (if needed):
  groupadd developers
• create user with UID, primary group, create home, set shell:
  useradd -u 1500 -g developers -d /home/alice -m -s /bin/bash alice
• set initial password interactively:
  passwd alice
• set password aging (example: min 0, max 90, warn 7 days):
  chage -m 0 -M 90 -W 7 alice
  or set specific dates:
  chage -E 2026-12-31 alice
• inspect aging and status:
  chage -l alice
  getent passwd alice
  getent shadow alice (requires root)
• lock/unlock account:
  passwd -l alice # locks by prepending '!' to hash
  passwd -u alice # unlocks
  or
  usermod -L alice
  usermod -U alice

Where aging is stored: /etc/shadow stores lastchange, min, max, warn, inactive, expiry fields (columns 3–8). Use chage -l alice or read /etc/shadow directly to inspect numeric values (convert days-since-1970 to dates with date -d "@$((value*86400))").

Direct answer
Combining blue-green with a read-only database clone lets you validate a migration's effect on real, current data BEFORE committing to it on the live database: the clone gets the migration applied first, in isolation, so you catch a problem against production-representative data without any risk to the actual live system, and the blue-green switch itself still gives you fast rollback for the application layer once you do commit.

Structured elaboration

1. Clone the production database (read-only) into an isolated environment and apply the migration to the CLONE first, validating both that the migration runs successfully and that the resulting data is correct, against real data characteristics (volume, distribution, edge cases) that synthetic test data might miss.
2. If clone validation passes, apply the actual migration to production using the same discipline covered elsewhere (backward-compatible, expand-contract, batched for large tables), since the clone validated the LOGIC and DATA EFFECT, not the operational safety of running it against a live, concurrently-written system.
3. Blue-green for the application layer: once the schema is safely migrated (backward-compatible, so both old and new app code can run against it), deploy the new application version to the green environment, validate it, and cut over traffic, keeping blue as an instant application-level rollback path.
4. Minimizing data loss and handling replication lag: the production migration's write path needs an explicit boundary that prevents an in-flight write from landing in the gap between the old and new state. Concretely: take a fresh, final backup/snapshot immediately before the real migration begins (the earlier clone can be stale by the time the actual migration runs, so it isn't a substitute for this), run the migration as backward-compatible expand-contract in small, monitored batches so a failure partway through never forces discarding already-migrated data, and gate the blue-to-green traffic cutover on replication lag explicitly: define a maximum acceptable lag (for example, hold the cutover while green's replica lag exceeds a few seconds) and only cut traffic over once green has caught up to that threshold, rather than cutting over on a fixed timer regardless of lag. During the cutover moment itself, a brief write-quiesce or dual-write window (writes are accepted by blue and also applied to or replicated into green before green starts serving reads) closes the specific gap where a write landing in the last moments before cutover could otherwise be lost to whichever side ends up not serving traffic.
5. Rollback scope: if a problem emerges post-cutover, blue-green gives fast APPLICATION rollback (switch back to blue), which is why replication should keep flowing from green back to blue for a defined window after cutover, so blue doesn't fall behind and a same-day switch-back doesn't lose whatever writes landed on green in the meantime; but if the issue traces to the migration itself rather than the application code, that's a data-layer rollback with its own considerations (covered by the backward-compatibility discipline that made the migration safe to begin with), not something the blue-green switch alone fixes.

Worked example
A migration converting a JSON blob column into normalized relational fields: applied first to a read-only production clone, revealing that roughly 2% of real production rows have malformed JSON that the migration's parsing logic doesn't handle, a data-shape problem synthetic test fixtures never surfaced. The migration logic is fixed to handle that edge case, re-validated against the clone, and only THEN applied to the actual production database with the same batched, lag-monitored discipline: a fresh pre-migration backup is taken, batches are throttled to keep replica lag under a defined threshold, and the blue-to-green traffic cutover waits until that threshold is met, with a brief dual-write window bridging the cutover moment itself so no write is lost in the gap; the application's blue-green cutover happens afterward, once the schema itself is confirmed safely migrated.

Trade-offs and pitfalls
Cloning a large production database is itself a real operational cost (storage, time to create the clone, and it can go stale relative to live production if there's a meaningful delay between cloning and actually running the real migration), so this technique earns its cost specifically for migrations complex or risky enough that catching a data-shape problem before it hits live data is worth the overhead; a simple, well-understood migration probably doesn't need it. The common mistake is treating clone validation as a substitute for the real migration's own operational safety discipline (batching, lag monitoring, a fresh pre-cutover backup, and an explicit lag threshold gating cutover) rather than as a complementary, earlier-stage check.

Requirements & scope:

• Multi-region service, 10M monthly users, SLO: 99.95% availability. Incident comms must be automated, role-specific, low-noise, and auditable.

Automated triggers (ordered by severity):

• Sev 1 (major outage): sustained SLO breach > 5m OR global error-rate spike > 5x baseline for 3m OR >50% region traffic loss.
• Sev 2 (partial degradation): latency p99 > 2x target for 5m OR error-rate 2–5x baseline.
• Sev 3 (minor/feature): localized errors, degraded metrics but no SLO breach.
• Telemetry-loss trigger: monitoring pipeline down -> notify on-call + monitoring owner.

Delivery channels:

• On-call engineers: PagerDuty (push + SMS + phone escalation), dedicated incident Slack channel, incident runbook link.
• Internal stakeholders (product, comms, legal, exec): Email + Slack briefings + private status page.
• External customers: Public status page + Twitter/X / in-app banner + email for major incidents.

Owners:

• Initial on-call message: primary on-call SRE (automated, owned by monitoring playbook).
• Internal stakeholder brief: Incident Commander (IC) — typically senior SRE or engineering manager.
• External customer comms: Communications lead (PM/Comms) owned; draft auto-populated by IC and sent after IC approval (auto-publish for Sev1 if IC unavailable for 10m).

Message templates (shortened; fields auto-filled: incident id, severity, regions, ETA):

• On-call engineers (Pager/Simple):
  "INC-{id} Sev {1|2|3}: {symptom} detected in {regions}. Key metrics: error-rate ↑ {x}x, p99={ms}. Runbook: {link}. IC: {name}. Acknowledge within 3m."
• Internal stakeholders (Email/Slack):
  "INC-{id} Sev {level}: Impact: {user%/regions}. Current hypothesis: {brief}. Actions in progress: {actions}. Next update: in {minutes}. IC: {name} (contact {phone})."
• External customers (Status page / in-app):
  "Service degradation (INC-{id}): Some users in {regions} may experience errors or delays. We're investigating and working to restore service. Next update: in {minutes}. Subscribe for updates: {link}."

SLAs (automated enforcement):

• Initial notification:
  • Sev1: On-call pager and IC notification within 1 minute of trigger.
  • Sev2: On-call pager within 3 minutes.
  • Sev3: Alert to on-call Slack within 15 minutes.
• First internal stakeholder update:
  • Sev1: within 10 minutes.
  • Sev2: within 30 minutes.
  • Sev3: next business hour.
• External customer initial notice:
  • Sev1: auto-publish within 15 minutes (IC can edit).
  • Sev2: publish within 60 minutes if customer impact confirmed.
• Update cadence:
  • Sev1: every 15 minutes until mitigated; then hourly during recovery; postmortem within 72 hours.
  • Sev2: every 30–60 minutes; postmortem within 7 days if SLO impacted.
  • Sev3: updates only if status changes; postmortem as needed.
• Escalation automation:
  • If IC not acknowledged within 5 minutes for Sev1, escalate to on-call manager and exec pager.
• Audit & rollback:
  • All messages logged to incident system (ID, timestamps, sender, edits). External messages have rollback templates and "all-clear" template sent when resolved.

Implementation notes / best practices:

• Use alert deduplication, throttling, and anomaly detection to avoid noise.
• Templates stored in incident system; fields auto-filled from monitoring events.
• Integrate multi-region health and CDN telemetry for accurate region-level messaging.
• Runbooks link to automated remediation scripts where safe (auto-mitigation only under explicit test coverage).

Purpose:
Rotate the production API key used by Service-A to remove compromised keys or implement scheduled key rotation with zero or minimal downtime and no degraded customer impact.

Prerequisites:

• Access to secrets store (Vault), CI/CD pipeline, and service deployment pipeline
• Runbook approvals from Service Owner and Security Lead
• Maintenance window (if required) and communication to stakeholders
• Backup of current key and audit logs enabled
• Automated health checks and monitoring dashboards available

Exact step-by-step actions:

1. Notify on-call, product owner, and affected teams; open a rotation ticket.
2. Create new key in secrets store (Vault) with metadata: created_by, expires, rotation_id.
3. Stage new key to non-prod environment; run smoke tests.
4. Deploy config change to Service-A to load new key from Vault using side-by-side strategy:
   • Update config to read PRIMARY_KEY and SECONDARY_KEY (keep old as secondary).
   • Deploy canary (5% traffic) with new key as PRIMARY.
5. Monitor canary for 15 minutes (errors, latency, auth failures).
6. If stable, progressively increase rollout (25%, 50%, 100%) via deployment pipeline.
7. After 100% success, retire old key in Vault by setting disabled=true but not deleting for 24h.
8. After 24h of stable metrics, delete old key and update rotation record.

Verification queries:

• API-level: curl -i -H "Authorization: Bearer <new_key>" https://api.prod.example.com/health
• Logs: search for auth failures: query: severity=ERROR service=Service-A "invalid API key"
• Metrics (Prometheus): sum(rate(http_requests_total{service="Service-A",code=401}[5m]))
• Traces: check distributed traces for increased latency/error rate for requests using key

Rollback procedure:

• If increased 5xx/401s or latency spike >2x baseline within monitoring window:
  1. Immediately flip deployment to previous revision or set PRIMARY_KEY back to old key in Vault.
  2. Scale down canary and stop rollout.
  3. Investigate logs/traces; keep old key active until root cause resolved.
  4. Document incident and schedule reattempt after remediation.

Required approvals:

• Service Owner (functional impact)
• Security Lead (compliance & key lifecycle)
• On-call SRE (operational readiness)

Monitoring to watch after the change (first 24-72 hours):

• Authentication failures (401/403) rate
• Error rate (5xx) and latency SLOs
• Traffic drop / client retry spikes
• Secrets access audit logs for Vault
• Alerting: PagerDuty if 5xx rate > SLO breach or 401 sum > threshold

Owner:
Runbook owner: SRE Team — rotation lead: Jane Doe (jane.doe@example.com). Last reviewed: 2025-06-01. Rotation frequency: every 90 days.