Infrastructure Testing and Validation Questions
Covers testing, validation, and safety practices for infrastructure changes and infrastructure as code. Topics include infrastructure unit and integration testing, smoke tests, load and performance testing, chaos engineering basics, validating deployments and rollbacks, testing infrastructure changes safely in staging, scaling and recovery validation, trade offs and coverage strategies for infrastructure testing, and monitoring and observability for deployed infrastructure.
MediumTechnical
45 practiced
Design a performance profiling test to detect memory leaks in a long-running service. Describe the workload profile, duration, tools to collect heap/alloc profiles (pprof, jemalloc), metrics to monitor, and acceptance thresholds that would indicate a leak.
Sample Answer
Situation / goal: Prove whether a long‑running service exhibits a memory leak under realistic load and quantify its rate so SREs and engineers can prioritize fixes.Workload profile- Use a production‑like synthetic workload: traffic mix (e.g., 70% reads, 30% writes), realistic QPS, connection churn, background jobs (cron, batch), and occasional peak spikes (burst for 5–10m).- Include exercises that exercise uncommon code paths (file IO, large payloads, authentication flows).- Baseline warm‑up: ramp from 0 → target QPS over 5–15 minutes, run steady state for the main window, then a cooldown + spike periods.Duration- Minimum: 24 hours to capture slow leaks and periodic GC/cleanup cycles.- Preferable: 72 hours for confidence in low‑rate leaks and memory growth patterns.- Collect profiles at high fidelity in first hour (every 1–5min), then every 15–30min during steady state; keep on‑demand snapshots during anomalies.Tools & data collection- pprof (for Go/C++ builds with symbols): collect heap/alloc profiles, goroutine stacks, CPU profiles. Use periodic HTTP pprof endpoints or trigger runtime/pprof writes.- jemalloc (if used): enable jemalloc stats and background sampling; collect mallctl/malloc_stats_print output, opt into stats_print and epoch to get accurate counters.- OS metrics: RSS, VSZ, swap usage via /proc or cAdvisor (containerized).- Allocator samplers: tcmalloc/hoard profiling if applicable.- Export metrics to Prometheus or similar; store raw profiles in object store for postmortem.- Use flamegraphs, heap diffing (pprof top/peek), and retained size analysis.Metrics to monitor- Absolute: RSS (MB), heap_alloc/heap_sys (MB), jemalloc allocated/active/dirty (MB).- Rates: d(RSS)/dt (MB/hour), allocation rate (MB/s), malloc/free count ratio.- Retention: retained size by stack (pprof retentive size), count of long‑lived objects.- GC/collector signals: GC pause latency, GC cycles per hour, free/reclaimed ratio.- Anomaly metrics: growth after GC (i.e., RSS immediately after GC vs before).- Application‑level: in‑flight requests, connection count, open file descriptors.Acceptance thresholds (examples you can tune per service)- After warm‑up (2× cycle of typical GC), RSS growth must be <= 1% per hour OR <= 5 MB/hour sustained for 24h.- Heap_alloc should plateau (variance <5%) for at least 12 hours under steady state.- Rate condition: sustained positive d(RSS)/dt for 24+ hours indicates leak — fail test.- Retained objects: no single stack trace should hold >5% of total retained memory unless expected; top 5 retainers combined should be <30% of heap.- GC reclaim validation: GC should reduce heap by >50% of transient allocations; if GC reclaims little while allocations continue, investigate retention.- Safety caps: RSS > configured service limit (e.g., container memory limit × 0.8) or FD growth without recycle → fail.Procedure & analysis- Run baseline test with only monitoring to understand system noise.- Run workload and collect periodic pprof + jemalloc stats + Prom metrics.- Post‑run: compute linear regression of RSS over time; compute per‑hour and per‑GC growth.- Heap diffs: compare pprof heap profiles at t0, t12h, t24h to find growing allocation sites; produce flamegraphs and top retained frames.- Validate with synthetic small tests to reproduce the suspect path, add targeted instrumentation, and verify fix reduces growth below thresholds.Remediation signal- If test fails, produce: leak rate (MB/hr), top 10 retaining stacks with retained size, triggerable reproduction steps, and recommended fixes (close handles, cache eviction, fix goroutine leaks, free C allocations).This plan balances realistic load, long enough duration to spot slow leaks, automated metric thresholds, and actionable profiling output (pprof/jemalloc) to locate root causes.
EasyTechnical
33 practiced
What is test data management in the context of infrastructure testing? Describe two strategies to manage secrets and credentials safely in test environments, e.g., ephemeral credentials and secrets redaction, and explain trade-offs.
Sample Answer
Test data management for infrastructure testing is the practice of provisioning, protecting, and maintaining the data and credentials used by CI/CD pipelines, integration tests, chaos experiments and staging environments so tests are reliable, repeatable, and safe. For an SRE this means ensuring tests exercise realistic conditions without leaking sensitive production secrets or creating long-lived attack surfaces.Two safe strategies1) Ephemeral credentials (short-lived, scoped tokens)- What: Issue credentials dynamically (e.g., via Vault/OIDC/STSEndpoint) with narrow scopes and TTLs; tests request, use, then revoke them.- Benefits: Minimizes blast radius, reduces risk from leaked tokens, supports auditability, and aligns with least privilege.- Trade-offs: Adds system complexity (credential broker, rotation logic), potential flakiness if issuance fails or TTLs expire mid-test; needs reliable network/PKI and retry logic.2) Secrets redaction and token fencing- What: Store sensitive values in secrets manager and never include them in logs/artifacts; redact or mask outputs, and replace real secrets with synthetic or hashed values where full fidelity isn't required.- Benefits: Simple to adopt, reduces accidental exfiltration via logs or test artifacts, and allows post-test retention of traces without secrets.- Trade-offs: Redaction can hide useful debug info; synthetic data may not exercise full production paths (e.g., encryption-specific behaviors). Requires discipline in tooling to enforce redaction and secure access controls.Best practice: Combine both—use ephemeral, scoped credentials for tests that must hit real services and redact any outputs; automate issuance/revocation, bake retries, and include auditing and monitoring to detect misuse.
MediumSystem Design
26 practiced
Design a CI pipeline stage that runs Terraform integration tests in an ephemeral AWS account. Describe steps for account provisioning, secure credential handling, resource provisioning, running assertions, safe teardown, and guardrails to avoid leftover resources and hitting provider rate limits.
Sample Answer
Requirements:- Create ephemeral AWS accounts per CI job, run Terraform to provision infra, execute integration assertions, then fully teardown within a bounded window. Secure creds, prevent runaway costs, and avoid provider rate-limit issues.High-level flow:1. Provision ephemeral account (AWS Organizations)2. Acquire short-lived credentials (STS/OIDC)3. Run Terraform (with unique names/tags and isolated backend)4. Execute integration tests (Terratest / pytest + boto3)5. Teardown (terraform destroy), enforce cleanup guards6. Post-checks and force cleanup if neededSteps and choices:- Account provisioning: - Use AWS Organizations CreateAccount API or an Account Factory (Control Tower) to spin an account. Tag account with CI job id, owner, created_at, ttl. - Emit account metadata to a central DB (DynamoDB) with TTL for lifecycle tracking.- Secure credential handling: - CI uses OIDC identity provider (GitHub Actions/GitLab) to assume a short-lived role in the new account through a central bootstrap role in the management account. No long-lived AWS keys. - Use least-privilege IAM role scoped to Terraform actions and test execution. Store any sensitive test secrets in Vault or AWS Secrets Manager; fetch via short-lived session.- Resource provisioning: - Execute Terraform in the ephemeral account. Use a dedicated S3 backend bucket per-account or a namespaced prefix (with SSE and versioning). Lock state with DynamoDB table scoped per-account. - Always apply a unique prefix/tag (ci-job-id, account-id) on all resources so they can be found and cleaned. - Limit sizes: enforce Terraform variable constraints (small instance types, limited replicas).- Running assertions: - Use Terratest (Go) or integration tests that call AWS SDK to validate resources (endpoints respond, DNS resolves, IAM policies attached). - Run smoke tests first, then longer tests. Fail fast and run most destructive checks last.- Safe teardown: - Run terraform destroy automatically in CI, retry with exponential backoff on transient AWS errors. - If destroy fails or times out, mark account for forced cleanup and invoke a central cleanup controller (Lambda/Step Function) that enumerates resources by tag and deletes them, escalating to account termination if necessary. - Implement a hard TTL: accounts older than X hours automatically suspended/closed via scheduled job.- Guardrails to avoid leftovers and rate limits: - Service Control Policies (SCPs) applied at account creation to prevent creation of high-cost services (e.g., block certain regions, high-end instance families, or public IP allocation). - Quotas: set per-account service quotas and enforce in Terraform plan checks. - Concurrency limits: CI orchestrator enforces a global parallelism cap to avoid hitting AWS API rate limits. - Terraform provider rate-limiting: set provider max_retries and provider meta config (parallelism flag) to a low value (e.g., -parallelism=10). - Tagging + inventory: central inventory checks for orphaned resources each hour; send alerts and auto-clean if beyond grace period. - Cost guardrails: budget alerts for ephemeral accounts; block accounts exceeding $X.Observability & metrics:- Emit metrics (account lifecycle events, test pass/fail, teardown success) to central monitoring (CloudWatch/Prometheus).- Alert on failed teardowns, high error rates, or quota exhaustion.Example tools:- Orchestration: GitHub Actions/GitLab CI, Step Functions, Lambda- Tests: Terratest / pytest + boto3- State/security: S3(encrypted)+DynamoDB locks, Vault/AWS Secrets Manager, OIDC+STS- Cleanup controller: Lambda with IAM role to enumerate+delete by tagWhy this works:- Short-lived accounts and creds reduce blast radius and credential leakage risk.- Tags + centralized inventory + automatic TTL ensure fast detection and safe teardown.- SCPs, quotas, and limited CI parallelism protect from runaway costs and provider rate limits while enabling repeatable, isolated integration testing.
EasyTechnical
34 practiced
List five minimal observability checks you would include as part of a post-deploy smoke test for a stateless microservice running on Kubernetes. For each check, explain why it's important and realistic thresholds you'd use (examples: readiness probe, 5xx rate, request latency, log error patterns, trace propagation).
Sample Answer
1) Readiness probe status Why: Verifies the pod is accepting traffic and passed in-app initialization (avoids 503s from kube-proxy). Threshold: >= 99% pods Ready within 30s of start; no pods in CrashLoopBackOff. Fail if any single replica remains NotReady after 60s.2) 5xx error rate (per-minute) Why: Indicates server-side failures introduced by new code/config. Catchs regressions quickly. Threshold: 5xx rate < 0.5% of requests over 1 minute AND <= 1 absolute 5xx per minute for low-traffic services.3) Request latency (p95 and p99) Why: Detects performance regressions that degrade user experience or cause timeouts upstream. Threshold: p95 < 300ms, p99 < 1s (adjust by service SLA); fail if p99 increases >2x vs pre-deploy baseline.4) Log error/fatal patterns Why: Some failures don't surface as 5xx but appear as exceptions in logs (resource leaks, startup errors). Threshold: Zero new ERROR/FATAL logs containing “panic”, “stacktrace”, or “unhandled” in the last 5 minutes; fail on >3 unique ERRORs in 5m.5) Trace/span propagation and sampling check Why: Ensures distributed tracing context is preserved end-to-end for debugging and latency attribution. Threshold: Successful trace propagation >= 95% for sampled requests over 5 minutes; fail if critical downstream spans missing for >=10% of traces.Bonus operational checks: Pod restart count (no unexpected restarts), resource usage (CPU/RAM < 80% of request limits).
HardTechnical
29 practiced
Monitoring shows increased errors when a third-party dependency is called only if infrastructure changes were applied in a specific order. How would you investigate order-dependent failures, design regression tests to catch them, and prevent such order-sensitive deployment sequences in the future?
Sample Answer
Situation: Production alerts showed a spike in errors calling a third‑party API, but only when recent infrastructure changes were applied in a particular sequence during deployment. This pointed to an order-dependent failure.Investigation & troubleshooting:- Reproduce deterministically in staging: capture the exact changes (IaC commits, feature flags, config diffs) and replay the sequence that triggers the failure.- Collect evidence: enable request/response logging, trace IDs, full stack traces, and sidecar/container logs. Correlate timestamps with deployment steps.- Hypothesize common causes: race conditions (services start before dependent config or secrets), version mismatches, stale caches, network rules (firewall/NAT) applied after service start, initialization ordering in orchestration (init containers, readiness probes).- Binary search the change set: apply changes incrementally to isolate the minimal order that reproduces the bug.- Example finding: service A started before the updated network policy was applied, causing it to cache unreachable endpoint metadata and retry with wrong host until restart.Root-cause confirmation:- Once reproduced, run controlled experiments (restart order variations, config reloads) to confirm which ordering and what fix (retry/backoff, init checks) removes errors.Design regression tests:- Add an automated integration test that simulates permutations of applying changes/rollbacks in CI: - Orchestrate deployment steps in varying orders (randomized and targeted sequences). - Validate end-to-end success of third‑party calls after each step.- Implement an idempotent “start/readiness” test: after each infra change, assert service health, third‑party connectivity, and that cached metadata is refreshed.- Example test snippet (pseudo-python):- Integrate into PR pipeline as a gated test for infra/IaC changes and into nightly regression with randomized sequences (property-based testing).Prevention & long-term fixes:- Make deployments order‑insensitive: - Use readiness checks and block other changes until dependent services are healthy. - Apply configuration and network changes via atomic transactions or orchestration sequences that ensure connectivity before traffic is routed (e.g., update network policies, then rollout). - Ensure components are resilient to transient failures: exponential backoff, retries, circuit breakers, and idempotent initialization. - Avoid implicit state by externalizing config/version info (feature flags, dynamic config service) so services refresh rather than cache immutable assumptions.- CI/CD & orchestration improvements: - Enforce deployment policies in pipeline (policy-as-code) that validate ordering safety. - Use canary/blue‑green to apply infra and rollout in safe phases; validate third‑party calls on canary before full rollout. - Add automated preflight checks: connectivity, DNS resolution, certificates, and dependency version compatibility tests before progressing.- Observability & guardrails: - Add targeted SLOs/alerts for third‑party error rate changes during deployments and for configuration drift detection. - Add deployment-time dashboards and traces to detect order-related anomalies quickly.- Process/culture: - Require an IaC rollback plan and post-deploy validation steps in runbooks. - Run periodic chaos/chaos engineering exercises that randomly reorder infra actions to surface order-sensitive issues.Result: Combined immediate mitigations (restarts, config refresh) with automated order-permutation tests and CI gating removes recurrence risk — deployments become resilient to ordering, and incidents are detected and remediated faster.
python
# orchestrate deployment steps in different orders and assert calls succeed
for order in permutations(steps):
apply_steps(order)
wait_for_readiness(all_services)
assert third_party_call() == expected_response
teardown()Unlock Full Question Bank
Get access to hundreds of Infrastructure Testing and Validation interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.