Team Infrastructure Challenges and Priorities Questions

Understand the specific infrastructure problems the team is facing, current technical priorities, and the direction of ongoing projects. Topics include the team's roadmap, high priority infrastructure improvements, common operational pain points, technical debt, team bandwidth constraints, and metrics for early success in the first six to twelve months. Candidates should be able to discuss likely trade offs, propose pragmatic first steps, and show awareness of organizational and operational factors that affect infrastructure work.

MediumSystem Design

24 practiced

You must migrate Terraform-managed infrastructure from AWS to GCP with minimal downtime. The team is small and bandwidth-constrained. Provide a pragmatic, phased migration plan: state handling and mapping of resources, validation strategy, cutover steps, testing, and rollback procedures. Explain three practical trade-offs you might accept to reduce risk and scope.

Sample Answer

Requirements & constraints:- Minimize downtime (near-zero for critical services), small team, limited bandwidth, maintain IaC (Terraform), preserve monitoring/SLAs.

Phased migration plan1) Discovery & mapping (1–2 weeks)- Inventory AWS resources, Terraform state files, dependencies, networking, IAM, data sizes.- Create resource map: AWS resource → GCP equivalent (e.g., EC2 → GCE/Cloud Run, S3 → GCS, RDS → Cloud SQL/Spanner).- Prioritize by criticality and data transfer complexity.

2) Prepare GCP infra in parallel (2–4 weeks)- Create separate Terraform code/modules for GCP; reuse logical modules (networking, IAM, logging, monitoring).- Use a new backend for state (GCS + Terraform Cloud/remote state locking).- Implement service accounts & least-privilege IAM.

3) State handling & dual-run strategy- Keep AWS Terraform state untouched during prep. Create fresh GCP state.- For resources that can be run in parallel (stateless services, read-only replicas), deploy on GCP and run dual-write/dual-read where feasible.- For stateful/datastore migration, create replicas in GCP (e.g., cross-region DB replicas, storage copy using gsutil / Storage Transfer Service) and keep sync until cutover.

4) Validation & testing- Automated smoke tests, integration tests, canary traffic routing via load balancers or traffic-splitting.- Observability parity checks: metrics, logs, alerts must match SLO thresholds before cutover.- Dry-run terraform plan comparisons; terraform import for any pre-existing GCP resources to ensure state alignment.

5) Cutover steps (night/weekend low-traffic)- Freeze writes if necessary (short maintenance window) or use DNS/traffic-split for progressive shift.- Switch DNS / load balancer targets to GCP in stages (canary → 10% → 50% → 100%) with health checks.- Monitor metrics and rollback triggers actively.

6) Rollback procedures- Predefine automated rollback thresholds (error rate, latency, saturations).- Keep AWS infra intact for rollback window. Reverse DNS/LB targets to AWS.- Use Terraform to reapply AWS state if needed; for DB, use last consistent snapshot or failback replication.

Validation strategy- End-to-end test suite, synthetic traffic, user acceptance testing, compare production metrics (error rate, p95 latency, throughput).- Data integrity checksums for migrated data; reconcile counts.

Three practical trade-offs to accept1) Accept temporary dual-running of some services (cost increase) to avoid risky big-bang cutover.2) Keep some managed AWS services (e.g., specialized AWS-only features) for a transition period instead of full rewrites to save time.3) Use lift-and-shift VMs (GCE) for complex apps initially rather than full refactor to Cloud-native—reduce scope, then refactor post-migration.

This plan minimizes risk with staged validation, keeps rollback simple, and matches a small SRE team's bandwidth by prioritizing automation, parallel prep, and incremental cutover.

HardSystem Design

38 practiced

Design an approach to build and maintain a service-level dependency map for a complex system with hundreds of services. Explain data sources for generating dependencies, how you'd keep the map up-to-date, and how SREs should use it to prioritize infrastructure improvements, root cause analysis, and SLO alignment.

Sample Answer

Requirements & constraints:- Accurate, near-real-time directed dependency graph of ~hundreds of services- Trustworthy provenance, change history, and confidence scoring- Queryable for blast radius, SLO impact, and prioritization- Low operational overhead and privacy-safe

High-level architecture:- Ingest layer → Normalization & enrichment → Graph store → API + UI + Jobs

Data sources (priority order):1. Distributed tracing (OpenTelemetry): most reliable for runtime call edges and latency.2. Service mesh telemetry / Envoy metrics: upstream/downstream relationships, success rates.3. Network flow (eBPF/sFlow) for infra-level dependencies and non-instrumented services.4. Service registry / discovery (Consul, Kubernetes API): declared dependencies, ports.5. CI/CD manifests and infra-as-code: declared configs, team ownership, deploy pipelines.6. Logs (call patterns) & database access logs for async or indirect dependencies.7. Human-curated overrides from owners (for critical business relationships).

Core components & responsibilities:- Ingest workers: poll traces/mesh/k8s/CI, produce canonical edge records with timestamps and confidence.- Normalizer: dedupe, merge edges, map service identifiers to canonical service IDs and teams.- Graph database: Neo4j or DGraph for queries (or scalable property graph on Bigtable).- Enrichment jobs: attach SLOs, error budgets, ownership, deployment frequency, and change events.- Change detector: compares snapshots, flags new/missing edges, auto-notifies owners.- Validation UI: allow owners to accept/reject edges; store provenance & confidence.

Keeping the map up-to-date:- Near-real-time stream from tracing/mesh; fallback batch sync from k8s/CI hourly.- CI/CD hooks: on deploy/update, trigger re-scan and lineage update.- Canary for noisy changes: require multiple corroborating observations before flipping high-confidence edges.- Automated tests: synthetic transactions that validate critical paths and detect drift.- Governance: periodic owner review, SLA for owners to confirm changes, automatic rollback of disputed auto-changes.

How SREs use it:- Root cause analysis: given an alerted service, traverse upstream/downstream with filter by error-rate spikes and recent deploys to find probable root causes; rank by confidence and blast-radius.- Prioritization of infra improvements: compute risk score per service combining fan-in/fan-out, SLO violation frequency, owner response time, and business criticality. Target high-risk, high-impact services.- SLO alignment & change management: map SLOs to dependency subgraphs to see which downstream/upstream SLOs must be coordinated for changes; use graph to simulate error budget impact of dependency failure.- Automation: tie dependency map to runbooks and incident playbooks so responders see validated next steps and owners.

Trade-offs & pitfalls:- False positives from sampling; mitigate via confidence scoring and corroboration.- Privacy/security for tracing—mask sensitive data and enforce RBAC.- Scalability: store aggregated edges with time-series metadata, prune old low-confidence edges.

Metrics to track map quality:- Coverage (percent services with >X confidence edges)- Drift rate (changes/day)- Owner validation rate and time-to-ack- Accuracy measured via periodic synthetic tests

This approach balances automated discovery with owner validation, gives SREs an actionable graph for RCA, prioritization, and SLO-aware change management.

MediumTechnical

19 practiced

A VP is pushing for multi-cloud to avoid vendor lock-in, but production runs 100% on AWS. As the SRE, evaluate trade-offs (engineering cost, operational complexity, security, latency) and propose a minimum viable multi-cloud approach with three concrete steps and acceptance criteria to validate the strategy.

Sample Answer

High-level evaluation (trade-offs)- Engineering cost: Porting systems to another cloud requires redesign (IAM, networking, storage semantics, infra-as-code), duplicate pipelines, CI/CD changes — significant upfront and ongoing cost.- Operational complexity: Multi-cloud increases runbooks, observability integration, incident playbooks, and skill requirements — elevates toil and MTTR initially.- Security: Attack surface grows (multiple IAM models, key management, compliance mapping). Consistent posture requires central policy automation (e.g., OPA/Gatekeeper).- Latency: Cross-cloud traffic adds egress cost and higher latency; data gravity means multi-cloud for stateful services is either complex (replication) or prohibitsively slow.

MVP multi-cloud approach — goals: reduce lock-in risk, keep production reliability, minimize cost/complexity.Three concrete steps1) Abstract infra and CI/CD for portability - Action: Introduce an IaC and pipeline layer that can target AWS and one secondary cloud (GCP/Azure) using Terraform modules + environment-specific variables; make container images cloud-agnostic. - Acceptance: Deploy a non-production workload (stateless service + infra) from same pipeline to both clouds within 1 hour; successful health checks and synthetic transactions passing SLA.

2) Implement cross-cloud disaster recovery for a subset - Action: Choose a critical, stateless customer-facing service and configure failover: replicate images/registry, sync config, and route traffic via DNS + health checks. - Acceptance: Simulated region outage in primary cloud triggers DNS failover and restores >= 95% of baseline throughput within RTO target (e.g., 15 minutes) with error budget intact.

3) Centralize observability and security policy - Action: Forward logs/metrics/traces from both clouds to a single observability plane; enforce CIS baseline via automated scans and policy-as-code. - Acceptance: Unified dashboards show metrics from both clouds; automated security scan reduces policy violations to zero for deployed MVP service.

Validation criteria (decision points)- Cost delta: annualized run cost for MVP path < 15% premium vs single-cloud projected — otherwise pause expansion.- Operational readiness: On-call can run failover/playbooks in ≤ RTO during drill with <2 post-drill runbook edits.- Business value: VP satisfied that we reduced vendor-lock risk for critical service and can expand incrementally.

If acceptance criteria met, expand to more services; else keep focus on portability and DR while avoiding full multi-cloud production until benefits justify cost/complexity.

MediumTechnical

33 practiced

Design a capacity-planning process for a service expected to grow 5x in 12 months. Include telemetry sources, headroom assumptions, autoscaling strategies, cost trade-offs between reserved vs spot/ondemand capacity, and how you would validate the forecast in production.

Sample Answer

Requirements & assumptions: target 5x traffic growth over 12 months, maintain SLOs (p99 latency, 99.9% availability). Plan for seasonal spikes (peak factor 1.5) and safety headroom (30% baseline) unless business provides stricter SLAs.

Telemetry sources:- Application: request rate, error rate, p50/p95/p99 latency, payload size- Infrastructure: CPU, memory, disk, network, socket counts, thread pools, GC metrics- Orchestration: pod/container counts, node utilization, pod start times, OOM/kube evictions- Business: feature flags, user signups, marketing campaigns, daily active users- Cost: hourly billing, reserved/spot usage, termination events

Capacity forecast process:1. Baseline: compute current peak RPS and resource-per-RPS (CPU, mem, I/O) from historical telemetry (rolling 90d), using linear regression and bucketed percentiles.2. Growth projection: apply 5x growth uniformly and include peak factor & 30% headroom: required_capacity = 5 * peak_RPS * (1+0.5) * (1+0.3).3. Translate to infra: convert resource needs into number/type of nodes (instance types), pods, and storage IOPS.

Autoscaling strategy:- Multi-layer autoscaling: - Horizontal Pod Autoscaler (HPA) based on custom metrics (RPS per pod, p95 latency) with target utilization and stabilization windows. - Cluster Autoscaler (for k8s) with node pools per workload; use pod priority/taints to prefer scaling certain pools. - Use predictive autoscaling: short-term (5–30 min) trend forecasting (exponential smoothing) to pre-scale before predictable increases.- Scaling policies: conservative cooldowns for scale-down, aggressive for scale-up up to defined burst capacity.

Cost trade-offs:- Reserved instances (or committed-use) for baseline steady-state (cover ~60–70% of forecasted steady usage) to minimize cost.- Spot/preemptible for ephemeral/batch or excess burst capacity (use node pools that tolerate interruptions, use PodDisruptionBudgets and checkpointing).- On-demand for critical control-plane or stateful components where termination is unacceptable.- Run TCO analysis: cost per vCPU/GB and failure risk; choose instance families with best price-performance and binpack-friendly sizing.

Validation in production:- Canary capacity increases: provision incremental capacity and route 5–10% of traffic, validate latency/error/SLOs, then ramp.- Chaos tests for spot interruptions and node failures; verify graceful degradation and fallback to on-demand pools.- Continuous monitoring: compare real-time utilization vs forecast; alert when divergence >15% for 24h.- Monthly reforecast with actuals, update reserved commitments before renewal windows.Metrics to track: forecast error, SLO burn rate, cost per RPS, time-to-scale for autoscalers.

This process balances reliability and cost while providing measurable feedback to refine forecasts.

MediumTechnical

23 practiced

Teams are performing frequent rollbacks after deployments due to regressions. Describe an investigation and stabilization plan, including how you'd collect deployment evidence, triage implicated changesets, introduce deploy-time safeguards (canaries, feature flags, progressive rollouts), and short-term mitigations to reduce rollback frequency while we fix root causes.

Sample Answer

Investigation & stabilization plan (SRE perspective)

1) Immediate evidence collection (first 1–3 hours)- Gather deployment metadata: deployment ID, Git SHAs, CI build number, image tags, Kubernetes rollout timestamp, helm/terraform diffs.- Pull runtime signals: error rates, latency P50/P95, saturation (CPU/mem), request rates, traffic logs, and recent alert history from Prometheus/Grafana, ELK, or Datadog.- Capture traces and logs for affected requests (Jaeger/Zipkin, structured logs) and sample user sessions. Export a short reproducible request trace if possible.- Record rollout topology: which regions/nodes/users saw the change (labels/taints, ingress routing).

2) Triage implicated changesets (next 2–6 hours)- Correlate time-series anomalies to deployment timestamps and individual commits via deployment metadata.- Prioritize commits by risk: infra/config changes, DB migrations, third-party upgrades, auth/serialization code.- Reproduce in a staging environment by promoting the same image + config. If not reproducible, narrow by toggling recent commits via git bisect or feature flags.- Assign owners: one lead SRE + relevant dev for each high-risk change.

3) Deploy-time safeguards to introduce ASAP- Canary deployments: direct 1–5% of traffic to new version for 15–30 min with automated health checks (error rate, latency, user-facing SLOs). If thresholds breached, auto-roll back.- Feature flags: ensure new behavior is behind flags that can be toggled without deploy. Use targeting (internal users, regions).- Progressive rollouts: exponential increase (1%, 5%, 25%, 100%) with manual gates and automated guardrails.- Automated rollback policy in CI/CD: if canary health checks fail, CI triggers rollback and alert to on-call.- Circuit breakers & bulkheads: limit blast radius (per-user quotas, connection pools).

4) Short-term mitigations to reduce rollback frequency- Require deploy checklist: DB migration preview, backward compatibility checks, schema versioning, contract tests.- Enforce stricter canary-by-default for all risky services for a 2-week window.- Add quick “kill switch” endpoints and feature-flag toggle playbook in runbook.- Increase observability around deploys: add deploy-correlated dashboards (deploy tag annotation), synthetic tests exercising critical paths immediately post-deploy.- Freeze non-essential changes until root cause fixed for teams with repeated rollbacks.

5) Root cause & long term fixes- Post-incident RCA with owner, timeline, contributing factors (testing gaps, alert thresholds, rollout config).- Invest in contract tests, chaos testing for dependencies, better staging parity, and automated integration tests in CI that run before canary.- Track metrics: rollback rate, mean time to detect (MTTD), mean time to recover (MTTR), canary failure rate; report to engineering leadership and adjust SLOs/error budgets accordingly.

Why this works- Fast evidence collection narrows scope and enables targeted mitigation.- Canary + feature flags reduce blast radius and allow safe experimentation.- Short-term process controls and observability reduce human error and improve confidence while engineering fixes root causes.

Unlock Full Question Bank

Get access to hundreds of Team Infrastructure Challenges and Priorities interview questions and detailed answers.

Join thousands of developers preparing for their dream job.