Production Incident Management Questions

Production/service incident response: how an on-call engineer detects, triages, and resolves outages or reliability degradation. Covers detection via monitoring metrics, logs, and distributed traces; mitigation via rollbacks, circuit breakers, feature flags, or network ACLs; incident communication and stakeholder updates; root-cause analysis; and blameless postmortems. No adversary, no malware, no legal evidence chain: the concern is system failure and reliability, not intrusion or malicious activity.

HardTechnical

65 practiced

You are incident commander and two high-priority incidents occur simultaneously: a global API outage and a suspected data exfiltration from an internal data store. Describe how you would prioritize, delegate responsibilities, set communication cadence, and escalate for legal or exec involvement. Include how to keep both incidents moving without overloading responders.

Sample Answer

Situation: Two P1s arrive at once — a global API outage impacting customers and a suspected data exfiltration from an internal store.

Task: As Incident Commander (IC) I must triage, prioritize, delegate, maintain momentum on both incidents, protect evidence, and escalate appropriately without burning out responders.

Action:- Rapid triage & prioritization (first 5–10 minutes): - Confirm impact: outage affects all customers → highest immediate customer and revenue risk; exfiltration is high-severity for security/compliance but may be limited scope. Prioritize customer-facing availability while starting containment for exfiltration in parallel.- Delegate ownership: - Appoint two Incident Leads: Reliability Lead (API outage) and Security Lead (exfiltration). Each gets an incident channel, dedicated Scribe, and a small response team (2–3 engineers + SME). - IC focuses on cross-incident coordination, resource allocation, stakeholder communication, and escalation decisions — not hands-on debugging.- Immediate goals per lead: - Reliability: runbook for rollback/traffic diversion, identify blast radius, apply mitigations to restore service within SLO targets. - Security: preserve evidence (forensic snapshot, logs), isolate affected hosts/data store, disable compromised credentials, begin containment.- Communication cadence: - IC runs 10-minute standups for first hour (rolling updates from both leads), then move to 30-minute cadence until stable. Post a one-line status every 15 minutes to execs via pre-agreed channel and a dedicated exec summary every 30–60 minutes depending on severity. - Maintain a “customer-facing” comms owner to draft status updates and throttle release timing with execs.- Escalation for legal/exec: - Immediately notify Legal and InfoSec Leadership about suspected exfiltration after initial containment steps and evidence preservation; provide scope, affected data types, and next steps. If PII or regulated data involved, follow breach policy and trigger external counsel/incident response retainer. - Notify CTO/Head of Ops about outage impact and mitigation ETA; escalate to CEO if outage risk to revenue/brand exceeds predefined thresholds.- Prevent responder overload: - Limit rotation shifts: 2–3 hour focused shifts per responder, bring in secondary on-call to expand capacity. - Pause non-essential alerts/automation noise for affected systems; funnel all work through incident leads to avoid context switching. - Use runbooks and playbooks to let junior engineers execute well-defined tasks; reserve senior SMEs for decisions.- Coordination & artifacts: - Maintain a shared timeline, decisions log, and action tracker (who/what/ETA). IC arbitrates resource conflicts (e.g., need to reboot DB vs. preserve forensic snapshot) based on impact and legal guidance.- Result & next steps: - Once services restored and containment confirmed, move security work into a focused forensic/eradication incident with scheduled deep investigation. Commit to a joint post-incident review, capture remediation, and update runbooks and breach response plans.

This approach balances customer impact and legal risk, keeps both incidents progressing via clear ownership and cadence, conserves responder energy with rotations and focused runbooks, and ensures timely executive and legal engagement when required.

MediumTechnical

85 practiced

A Kubernetes pod appears compromised and is exfiltrating data. As the SRE on-call, list step-by-step containment and evidence collection actions specific to Kubernetes and cloud storage. Include commands or API calls you would run to preserve pod logs, container filesystem, persistent volumes, and cluster audit logs while minimizing disruption to other pods.

Sample Answer

1) Triage & quick isolation (minimal blast radius)- Don’t delete the pod. Create a NetworkPolicy that denies all egress/ingress for that pod (match by pod label) so you stop exfil while leaving other pods running.

bash

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-suspect
  namespace: suspect-ns
spec:
  podSelector:
    matchLabels:
      app: suspect-pod-label
  policyTypes: ["Ingress","Egress"]
  ingress: []
  egress: []
EOF

2) Preserve Pod metadata & runtime state- Capture pod YAML, events, describe, and node info:

bash

kubectl get pod suspect-pod -n suspect-ns -o yaml > suspect-pod.yaml
kubectl describe pod suspect-pod -n suspect-ns > suspect-pod.describe
kubectl get events -n suspect-ns --sort-by='.lastTimestamp' > suspect-events.log
kubectl get pod suspect-pod -n suspect-ns -o jsonpath='{.status.hostIP}' > suspect-node.txt

- Capture process list and open connections (if allowed) by exec’ing a read-only command; if exec is risky, skip and rely on node-level collection.

bash

kubectl exec -n suspect-ns suspect-pod -- ps aux > pod-ps.txt
kubectl exec -n suspect-ns suspect-pod -- ss -tunap > pod-netstat.txt

3) Preserve logs- Pull current and previous container logs with timestamps:

bash

kubectl logs suspect-pod -n suspect-ns --timestamps > suspect-logs.current
kubectl logs suspect-pod -n suspect-ns --previous --timestamps > suspect-logs.previous || true

- If cluster uses centralized logging (Stackdriver/Cloud Logging/ELK), export relevant log slices immediately:GCP:

bash

gcloud logging read 'resource.type="k8s_container" AND resource.labels.pod_name="suspect-pod"' --limit=10000 --format=json > gcp-logs.json

AWS CloudWatch:

bash

aws logs filter-log-events --log-group-name "/aws/eks/cluster/..." --filter-pattern 'suspect-pod' > cw-logs.json

4) Capture container filesystem & artifacts- If allowed, copy filesystem via kubectl cp:

bash

kubectl cp suspect-ns/suspect-pod:/ /artifacts/suspect-pod-fs -c suspect-container

- If you cannot exec/cp (safety), perform node-level snapshot of container runtime files. On the node (requires access): - With containerd/crictl: pause container then export/containerd snapshot or use cri-tools to export container filesystem tar. - Example (node):

bash

sudo crictl ps --name suspect-pod  # find container id
sudo crictl imagefs <container-id> # vendor-specific; or use runc checkpoint/export

(Implement per runtime; preserve runtime socket logs)

5) Preserve Persistent Volumes (cloud storage)- DO NOT delete PVC. Immediately create a snapshot of underlying disk:AWS EBS (get volume id via PVC annotation or describe PV):

bash

kubectl get pv <pv-name> -o jsonpath='{.spec.awsElasticBlockStore.volumeID}'
aws ec2 create-snapshot --volume-id vol-xxxx --description "forensic suspect-pod $(date -u +%FT%T) " > ebs-snapshot.json

GCP PD:

bash

gcloud compute disks snapshot DISK_NAME --project=PROJECT --zone=ZONE --snapshot-names=suspect-snapshot-$(date +%s)

For GCS-backed PVCs (CSI snapshot support): create VolumeSnapshot CR.

bash

kubectl apply -f volumesnapshot.yaml

6) Preserve cluster-level audit & control-plane logs- Export Kubernetes audit logs from cloud provider:GKE:

bash

gcloud logging read 'resource.type="gce_instance" AND logName="projects/PROJECT/logs/cloudaudit.googleapis.com%2Factivity" AND protoPayload.request.resourceName:"pods/suspect-pod"' --format=json > kube-audit.json

EKS/CloudTrail:

bash

aws cloudtrail lookup-events --LookupAttributes AttributeKey=ResourceName,AttributeValue=suspect-pod > cloudtrail.json

- If you have built-in kube-apiserver audit sink, fetch the audit log files from their storage (GCS/S3/ELK) and snapshot them.

7) Node-level evidence (if pod is on node you control)- Cordon the node to avoid scheduling but keep pods running:

bash

kubectl cordon <node>

- Snapshot node disk if permitted, capture container runtime logs (/var/log/containers, /var/log/pods, /var/log/messages/journal).

bash

sudo tar czf /artifacts/node-logs-$(date +%s).tgz /var/log/containers /var/log/pods /var/log/syslog

8) Prevent restarts/side effects- Patch the pod's owner (Deployment/ReplicaSet) to avoid immediate replacement: scale down deployment or add an annotation to pause controllers. Scaling down impacts other replicas; if application has multiple replicas, prefer isolating only suspect pod via NetworkPolicy and patching the pod to `spec.activeDeadlineSeconds=1` is destructive—avoid unless necessary.

bash

kubectl scale deploy suspect-deploy -n suspect-ns --replicas=0   # only if acceptable

9) Chain of custody & storage of artifacts- Store all artifacts in an immutable, access-controlled bucket (S3/GCS) with versioning and server-side encryption; note timestamps/collector identity.AWS:

bash

aws s3 cp suspect-logs.current s3://forensic-bucket/suspect/ --storage-class STANDARD_IA

GCP:

bash

gsutil cp suspect-logs.current gs://forensic-bucket/suspect/

10) Notify security/forensics and escalate- Share collected artifacts, snapshot IDs, node names, and all commands run. Freeze further automated remediation until security confirms.

Notes/Reasoning:- NetworkPolicy isolates without deleting artifacts or affecting unrelated pods.- Snapshot PVs immediately; deleting PVC/PV loses data.- Pull logs from centralized logging where possible - it’s less intrusive.- Avoid restarting or deleting the pod; that can remove memory artifacts and logs.- Node-level collection may be required for deeper forensics but carries higher impact — cordon first.- Keep an audit trail of every forensic action (who ran what, when).

Edge cases: If pod uses hostPath or shared volumes, prioritize node disk snapshots. If exec is blocked (attacker removed shell), rely on node/runtime-level artifacts and central logs.

HardTechnical

75 practiced

Describe how you would define SLOs and error budget policies that integrate with incident response workflows. Include thresholds and automated actions such as pausing deployments, limiting feature rollouts, or triggering postmortems as error budgets are consumed. Explain how those policies help balance reliability and velocity.

Sample Answer

Start by defining clear, measurable SLOs tied to user-visible outcomes (e.g., 99.95% successful requests latency < 200ms per month, or 99.9% payment success rate). Derive an error budget = 1 - SLO (e.g., 0.05% downtime per month). Instrument with reliable observability (service metrics, distributed tracing, synthetic checks) and compute burn rate over rolling windows (e.g., 7d and 30d).

Define tiered error-budget policy (example):- Green (<= 25% budget used): normal velocity. Full CI/CD, can run canaries and wide feature rollouts.- Yellow (25–50% used): caution. Require additional approval for risky changes; limit rollout % to 25% via feature flags; increase monitoring frequency and create a temporary incident dashboard.- Orange (50–80% used): restrict. Pause non-critical deployments automatically in CI unless emergency; force canary-only releases at 5–10%; require SRE review and automated smoke-tests to proceed; throttle background jobs.- Red (>80% or high burn rate): protective. Automatically pause all standard deployments; roll back recent changes flagged by deployment correlator; open an incident (pager/Slack) and a triage runbook; schedule a blameless postmortem and freeze feature launches until remediation and approval.

Automations to implement:- CI/CD hooks that query current error-budget state before allowing merges or promotions.- Feature-flag service integrations to programmatically cap rollout percentages or disable features when in orange/red.- Deployment orchestration rules (Argo/Spinnaker) that pause pipelines and run automated rollback if error-budget thresholds crossed.- Alerting playbooks that generate incidents and attach relevant logs/traces, and auto-create postmortem templates when red threshold triggered.- Burn-rate monitors that evaluate short-term vs long-term windows and trigger different policies.

How this balances reliability and velocity:- The tiered approach lets teams move quickly while containing risk: small, gradual consumption (green/yellow) doesn’t block delivery, preserving velocity. Rapid budget consumption causes progressive safeguards (orange/red) that reduce blast radius and force focus on reliability. Automation enforces policy consistently and reduces human latency in decision-making. Requiring SRE reviews and postmortems ensures learning and prevents repeat burns, aligning long-term velocity with reliability goals.

HardTechnical

78 practiced

A production outage was caused by a sequence of cascading failures across services and configuration changes. Describe an approach and toolchain to perform root cause analysis that correlates metrics, logs, and traces into a timeline, identifies contributing factors, and distinguishes root cause from symptoms. Include query strategies and visualization techniques to support the RCA.

Sample Answer

Approach (framework)- Triage → Contain → Collect → Correlate → Hypothesize → Validate → Remediate → Postmortem.- Goal: build a single timeline that shows when config/deploy changes, metric anomalies, logs/errors, and traces occurred; use that to identify causal chain and separate root cause from downstream symptoms.

Toolchain (example)- Metrics: Prometheus + Grafana- Logs: Loki/Elasticsearch + Kibana/Grafana Explore- Traces: OpenTelemetry → Jaeger/Tempo + Grafana- Orchestration/CI: Kubernetes events, Argo/CD or Git commit history- Incident DB: PagerDuty/Incident.io + an RCA doc template

Step-by-step RCA with queries & visualizations1) Ingest anchors: mark exact deploys/config-change timestamps (CI/CD, ConfigMap changes, feature flags) as annotations in Grafana.2) Global timeline panel: Grafana dashboard with aligned panels (top to bottom): deployments/events, error-rate (5m), latency P99/P95, request rate, CPU/mem, retries, downstream error rates. Use dashboard time sync to zoom.3) Metrics-first queries (PromQL examples): - Error spike: increase(http_requests_total{job="svc",status=~"5.."}[5m]) - Latency shift: histogram_quantile(0.99, sum(rate(http_server_request_seconds_bucket[5m])) by (le))4) Logs pivot (Loki/ES): - Find logs around spike: {app="svc"} |= "error" |= "timeout" | json | sort by @timestamp - Query for correlated trace_id or request_id in logs: request_id="XYZ" - Use structured logs to aggregate failure types and top callers.5) Trace correlation: - Search traces for high-latency or error traces: span.duration > 1s AND status = error - Filter by service and the request_id found in logs to jump to full trace. - Visualize span waterfall to see which downstream call first timed out/retried causing backpressure.6) Cross-linking: - Always include request_id/trace_id in logs/metrics; use Grafana Explore "trace" links from log lines. - Use SQL/ES aggregation to compute error counts by host/pod/node to find blast radius.7) Identify contributing factors vs root cause: - Temporal precedence: root cause must precede symptom. - Minimality: removing root cause (rollback or fix) should stop cascade. - Reproducibility: can you reproduce by simulating that change/failure in staging? - Use "change windows" to test hypothesis: roll back config or scale one component and observe metrics/trace behavior.8) Visualization techniques: - Annotated, synchronized timeline with colored bands for deploys. - Heatmap for error rate by service over time. - Waterfall trace maps with critical path highlighted. - Dependency graph showing increased latencies/traffic.9) Query strategies & shortcuts: - Top-down: service-level error/latency → drill to pod/node → logs → traces. - Bottom-up: alerting trace IDs or unique request IDs → find metric aggregations around those IDs. - Use rate(), increase(), histogram_quantile() in PromQL to detect shifts; use percentile diffs vs baseline. - Use Loki/ES aggregation to count distinct error messages and sort by frequency.10) Document and validate: - Record hypotheses, tests performed (rollbacks, throttles), and results. - Capture exact root cause (e.g., misconfigured timeout introduced in config X at T caused retries to queue, causing downstream service Y to exhaust connections → 5xx). - Postmortem: remediation (fix config checks, circuit breakers, retries, SLO changes), metrics/alerts to detect earliest signal.

Example short scenario- Observation: sudden P99 latency and 5xx spike at 10:12.- Annotated deploy at 10:10 that changed client timeout from 3s→10s.- Traces show client kept connections open longer, causing connection pool exhaustion in downstream service → cascade.- Validation: rolling back timeout restored previous error rates; root cause = faulty config change; symptoms = downstream timeouts and CPU pressure.

Best practices- Standardize request_id/trace_id in logs/headers.- Annotate dashboards with CI events automatically.- Keep dashboards for “blast-radius” and “dependency latency”.- Alert on relative change from baseline (not absolute thresholds) to catch regressions quickly.

HardSystem Design

130 practiced

Design a tamper-evident evidence archival pipeline for forensic artifacts at scale. Include signing of artifacts, use of immutable/WORM storage, key management (HSM or KMS), audit logging, access controls, and retention policies. Explain how to prove evidence authenticity and how to rotate or revoke access safely.

Sample Answer

Requirements & constraints:- Functional: ingest forensic artifacts (files, hashes, metadata), produce tamper-evident, queryable archive, chain-of-custody proofs.- Non-functional: high availability, scalable to millions of artifacts/month, low latency for verification, compliance (WORM, retention, auditability).

High-level architecture:Ingest -> Preprocessing -> Signing Service (HSM/KMS) -> Immutable/WORM Storage -> Audit Log + Index DB -> Verification API

Components & responsibilities:1. Ingest layer (API/gateway, message queue): accepts artifacts, performs validation, generates ingest-id, stores raw to temporary safe staging.2. Preprocessing: compute canonicalized artifact (normalize metadata), compute cryptographic digest (SHA-256/512), capture provenance metadata (uploader, timestamp, source).3. Signing Service: runs in secure environment. Private keys stored & used inside HSM (FIPS 140-2/3) or cloud KMS with HSM-backed keys. Sign canonical digest + provenance -> produces signed manifest (signature, key-id, sig-alg, timestamp).4. Immutable/WORM Storage: write-once object store (S3 Object Lock Governance/Compliance mode or on-prem WORM appliances). Store artifact + signed manifest together and set retention/lock based on policy.5. Audit logging: append-only tamper-evident audit log (e.g., backed by append-only ledger service or write hashes into a blockchain-like anchoring service; also replicate to remote syslog/SIEM). Log: ingest-id, operations, signer key-id, storage object-id, actors.6. Index & Retrieval DB: read-only indexed metadata for searches; store pointer to object and manifest.7. Verification API/Tools: verify signature using public key, validate storage object checksum, check audit log entries and retention/lock status.

Proving authenticity:- Recompute digest from retrieved artifact, verify signature against public key with certificate chain rooted at organization CA.- Check manifest timestamp and signer key-id; verify key was valid/not revoked at signing time via key transparency/CRL/OCSP logs.- Validate object immutability by asking storage for object-lock/retention metadata and compare audit log entries. Optionally verify anchoring record stored in an external timestamping service (RFC 3161) or public blockchain for non-repudiation.

Key management, rotation & revocation:- Keys are HSM-protected; signing keys are non-exportable. Use KMS envelope encryption for artifacts if needed.- Key lifecycle: create key (with metadata), use for signing, record key-id in manifest & audit log, rotate by creating new key and publishing new public cert.- To rotate: retire old key for new signatures; keep old key available in HSM for verification of past signatures (do not delete private material until retention expiry). Publish key metadata and validity windows in key registry.- To revoke: publish revocation record in key registry + CRL/OCSP; record revocation in audit log and anchor revocation to external timestamping. For artifacts signed before revocation, preserve signatures and use timestamp to prove validity at signing time. If private key compromise requires re-signing, follow policy: mark affected artifacts as "re-signed" by new key, retain original manifest, and record re-sign operation with justification/audit.

Access control & separation of duties:- Principle of least privilege: RBAC & ABAC for ingest, retrieval, admin.- Signing service access restricted to service account with MFA and approval workflow for emergency access to HSM.- Immutable audit trail for all admin actions; require dual control for destructive operations (policy change, retention reduce).- Network isolation: signing in private subnet, HSM/KMS in secure network.

Retention & legal hold:- Define retention classes (e.g., legal-hold, standard forensic retention). Apply object-lock with appropriate retention mode and expiry. For legal hold, use indefinite lock triggered by case management system; only extensible, not reducible.- Automated policy engine enforces lifecycle transitions; alerts before expiry for review.

Observability, SRE & operations:- Monitor signer latency, HSM health, S3/Object Lock errors, queue depth. SLOs: signing uptime, verification latency.- Backups: keep encrypted replicas of manifests and audit logs to geographically separate read-only stores.- Disaster recovery: HSM backups via split knowledge backup templates; test recovery drills.- Chaos & compliance testing: periodic end-to-end verification tests (recompute+verify) and external attestation.

Trade-offs & alternatives:- Use cloud KMS HSM for ease vs on-prem HSM for maximum control.- Anchoring to public ledger increases non-repudiation but adds cost/latency.- Re-signing vs key-archival trade-off depends on max retention and legal requirements.

Edge cases:- Partial ingestion failures: treat as atomic transaction—retain staging until signed & stored or tombstone with audit entry.- Key compromise: follow rotation/re-signing policy and notify stakeholders; use offline forensics to determine affected artifacts.

This design yields tamper-evident archives with provable authenticity, auditable chain-of-custody, safe key lifecycle handling, and operational practices to maintain reliability at scale.

Unlock Full Question Bank

Get access to hundreds of Production Incident Management interview questions and detailed answers.

Join thousands of developers preparing for their dream job.