Infrastructure and Deployment Troubleshooting Questions

Covers a systematic approach to diagnosing and resolving infrastructure and deployment failures across cloud and on premise environments. Topics include collecting and interpreting logs, metrics, and traces; isolating failures and performing root cause analysis; verifying network connectivity, identity and access management, and resource configuration; debugging containerization and operating system level issues; diagnosing continuous integration and continuous delivery pipeline failures across build, test, and deploy stages; addressing infrastructure as code drift and service limits; applying rollback, canary, and incremental deployment strategies; deciding when to escalate versus handling directly; and conducting incident response and post incident learning to prevent recurrence.

EasyTechnical

87 practiced

Describe the differences between logs, metrics, and traces in cloud observability. For each artifact explain: 1) typical schema and retention trade-offs, 2) when you would use it during a production incident, and 3) one concrete example problem best solved by that artifact (give a short example for each).

MediumTechnical

74 practiced

RDS instance creation is failing with a quota/service-limit error in a busy account. Propose immediate mitigations to restore functionality (workarounds that can be executed quickly) and longer-term architectural changes to prevent hitting the same service limits again.

Sample Answer

**Immediate mitigations (fast, actionable)**

- Confirm the exact quota using Service Quotas / AWS CLI (aws service-quotas get-service-quota) to know which limit is hit.- Request an emergency quota increase via Service Quotas console or Support (choose Business/Enterprise for faster SLAs).- Free capacity now: stop or delete noncritical RDS instances, snapshots, or automated snapshots; scale down oversized dev/test instances to a smaller class; or recycle idle instances.- Reuse existing databases: restore from an existing snapshot into an available instance class or promote a read-replica instead of creating new instances.- Create the DB in another Region or another AWS account (if multi-account org available) as a fast temporary workaround.- Fall back to a lightweight alternative: deploy a single EC2-hosted database or use Aurora Serverless v2 (if compatible) to restore app functionality quickly.- Communicate: notify stakeholders, push short maintenance windows and feature flags to reduce load.

**Longer-term architectural changes (prevent recurrence)**

- Implement quota monitoring and automation: export Service Quotas to CloudWatch, alert before thresholds, and automate quota increase requests via API where possible.- Adopt multi-account pattern and regional distribution to spread resource usage; enforce quotas per environment via SCPs.- Move to more flexible DB options: Aurora Serverless v2 or Aurora clusters with autoscaling for connections/compute; use RDS Proxy to reduce instance count by multiplexing connections.- Enforce lifecycle policies: automated cleanup of snapshots, unused instances, and test environments via Infrastructure-as-Code (Terraform) with destroy policies.- Capacity planning & runbooks: maintain capacity forecasts, playbooks for emergency scale, and regular Trusted Advisor/Cost Explorer audits.- Governance: tagging, cost/usage chargebacks, Service Quotas owner, and periodic quota review with AWS support.

These steps restore service quickly and reduce future risk by combining monitoring, governance, flexible DB choices, and multi-account/regional distribution.

MediumTechnical

77 practiced

During a high-severity incident, two engineering teams strongly disagree on the immediate containment action versus the long-term fix. As an incident coordinator, explain the steps you would take to make a decision quickly, keep stakeholders aligned, and ensure the incident is contained while preserving evidence for post-incident analysis.

Sample Answer

Situation & goalI’m the incident coordinator for a high-severity cloud outage where Team A wants immediate containment (isolate affected instances) and Team B wants a longer diagnostic-first approach. My goals: contain impact fast, preserve forensic evidence, and keep stakeholders informed.

Immediate steps (decision-making)- Triage fast: ask both teams to state immediate risks/benefits (30–60s each). Use runbook criteria (safety, data loss, blast radius) to score options.- Choose containment-first if user impact or data exfiltration risk is high; choose investigation-first if containment would destroy volatile evidence and impact is moderate. If unclear, prefer a short-lived containment-with-reversible-rollback plan.- Call the tie-breaker (on-call engineering lead or SRE manager) only if runbook doesn’t resolve it.

Containment actions (cloud-specific)- Isolate resources via security groups/NACLs or move instances to a quarantine subnet rather than terminate.- Snapshot disks (EBS snapshots, GCP persistent disk snapshot) and export memory dump if possible.- Turn on/collect CloudTrail, VPC Flow Logs, ASG/activity logs, and copy logs to an immutable S3 bucket with object lock.- Freeze deployments (pause CI/CD) and revoke affected credentials.

Stakeholder alignment & communication- Immediately post a concise incident update: action taken, rationale, owner, ETA for next update (every 15 min).- Assign clear RACI: who contains, who investigates, who communicates externally.- Schedule brief syncs for decisions that require longer discussion; escalate blockers to incident commander.

Post-containment- Verify service stability, validate rollback plan, continue evidence collection.- Transition to post-incident analysis with preserved artifacts, timeline, and agreed next steps (hotfix vs. long-term fix) and ensure both teams contribute.

Why this works- Uses playbook + measurable criteria for quick, defensible decisions.- Preserves forensic evidence by snapshotting and quarantining rather than destroying resources.- Keeps stakeholders informed and ownership clear so containment and long-term fixes proceed in parallel.

HardTechnical

79 practiced

Flaky integration tests in your CI/CD pipeline are causing frequent false failures that block deployments. Design a prioritized plan that includes immediate mitigations to unblock deployments and a long-term plan to reduce flakiness. Include concrete tactics for quarantine, test reliability metrics, and automation.

Sample Answer

**Immediate objective — unblock deployments (0–48h)**

- Triage & fast quarantine - Automatically tag failing tests in CI with “quarantine/candidate” and remove them from the blocking gate pipeline within a rollback-safe feature flag. Keep them in a non-blocking nightly pipeline. - Add an automated rerun-once policy for transient failures (with jitter) to filter intermittent network/timeouts before quarantine.- Short-term stability tweaks - Increase timeouts for known eventual-consistency APIs (S3, DynamoDB, IAM propagation) and add exponential backoff where applicable. - Use blue/green or canary releases for deployments while CI investigatory work continues.

**Medium-term (1–4 weeks) — root-cause & measurement**

- Metrics & dashboards - Track test-level Flakiness Rate = flaky runs / total runs; Mean Time To Flake (MTTF); Failure Signal-to-Noise. - Build a dashboard (CloudWatch/GCP Monitoring/Prometheus + Grafana) showing flaky tests, services implicated (network, storage, auth), and owners.- Ownership & process - Assign test owners, establish SLA (e.g., <1% flakiness), require root-cause within 7 days for high-priority tests. - Implement quarantine review workflow: auto-unquarantine after passing X runs or manual sign-off.

**Long-term (1–3 months) — reduce flakiness at source**

- Test design & infra improvements - Move integration tests to isolated ephemeral environments using IaC (Terraform/CloudFormation) per test or per-suite to eliminate resource contention. - Replace brittle timing assertions with event-driven checks, idempotent setup/teardown, and mocked endpoints where appropriate. - Use stable test data strategies: dedicated test accounts, deterministic seeding, and per-run namespaces.- Automation & CI engineering - Create a “flaky-test bot” that files issues, annotates runs, and enforces quarantine rules. - Run mutation and chaos testing in pre-prod to surface race conditions. - Integrate test health gating: only tests with recent health passes are required for blocking merges.

**Trade-offs & expected outcomes**- Short-term quarantine unblocks deploys quickly at the cost of temporarily reduced test coverage; medium/long-term investments (ephemeral infra, ownership, tooling) reduce operational toil and restore confidence in the pipeline.

MediumTechnical

78 practiced

You have CloudWatch logs showing repeated 500 responses and an attached latency metric that spikes at the same time. Outline how you would correlate logs, metrics, and traces to identify which microservice and which code path introduced the fault. Include specific correlation keys and queries you would use conceptually.

Sample Answer

**Approach (high level)**1. Triangulate: start with metrics (latency / 5xx spikes) → find time window → pull logs with correlation keys → open distributed traces for those requests to find service & code path.

**Step-by-step**- Identify window from CloudWatch metric graph (e.g., latency and 5xx alarm at 2026-02-28T12:03–12:06).- Query logs with common correlation keys: traceId / X-Request-ID / awsRequestId / container_id / instance-id.

Example CloudWatch Logs Insights to find 5xx and request ids:

sql

fields @timestamp,@message
| filter status >= 500
| filter @timestamp >= '2026-02-28T12:03:00Z' and @timestamp <= '2026-02-28T12:06:00Z'
| parse @message /traceId=(?<traceId>[\w-]+)/
| stats count() by traceId, bin(1m)
| sort by count() desc

- Use traceId or X-Amzn-Trace-Id to fetch distributed trace in X-Ray (or Jaeger/Zipkin): - In X-Ray console search trace ID found above; inspect segment timeline to see which microservice introduced long latency or error exceptions.- Cross-check logs for the specific microservice (e.g., auth-service) using container/instance tags:

sql

fields @timestamp, service, container_id, @message
| filter traceId = "TRACE-XYZ"
| sort @timestamp asc

**What to look for in traces/code path**- Longest segment (high subsegment duration) and exception/throw stack- Retries or downstream 500 responses- Resource saturation (CPU, DB latency) correlated via CloudWatch metrics for that instance / RDS

**Correlation keys to standardize**- X-Request-ID (UUID) propagated across services- X-Amzn-Trace-Id / trace-id (for X-Ray)- container_id / ecs_task_id / instance-id- userId or sessionId (business context)

**Actions & follow-up**- Patch the code path causing exception, add more granular metrics and structured logs (JSON with traceId/requestId), run synthetic tests for the failing path, and create alert playbook mapping metric → logs query → trace lookup.

This workflow quickly narrows service and exact code path by combining time-windowed metrics, request-scoped logs, and full distributed traces.

Unlock Full Question Bank

Get access to hundreds of Infrastructure and Deployment Troubleshooting interview questions and detailed answers.

Join thousands of developers preparing for their dream job.