Comprehensive understanding of cloud computing platforms and core infrastructure concepts. Candidates should know service models including Infrastructure as a Service, Platform as a Service, and Software as a Service, and be familiar with major providers such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure. Core technical knowledge includes compute models, storage systems, networking fundamentals such as domain name system and load balancing, virtual private networks and network segmentation, virtualization, containerization for example Docker, orchestration with Kubernetes, serverless architectures, and microservices. Candidates should be able to evaluate trade offs between managed services and self managed solutions with respect to cost, reliability, operational burden, scalability, performance, security, and vendor lock in, and reason about when to choose platform managed services versus building custom infrastructure. The topic also covers system design considerations for high availability and fault tolerance, capacity planning and autoscaling, monitoring and observability, deployment strategies, and operational practices such as infrastructure as code and continuous integration and continuous delivery. This knowledge is critical for backend engineers, site reliability engineers, and DevOps roles and is increasingly relevant across many engineering positions.
MediumTechnical
29 practiced
Provide a Kubernetes YAML manifest (Deployment + HorizontalPodAutoscaler) for a stateless web service named "catalog": 3 replicas, container image registry.example/catalog:1.2, CPU requests 200m, CPU limits 500m, liveness probe (HTTP /healthz) and readiness probe (HTTP /ready). The HPA should scale between 3 and 10 replicas targeting 60% CPU. You can show the manifest as YAML.
Kubernetes API server saturation is causing scheduling failures during large rollouts: describe root causes that can produce this behavior (e.g., controller loops, API request storms), how you would diagnose the control plane, and short/medium-term mitigations and long-term fixes.
Sample Answer
Situation: During large rollouts the Kubernetes API server becomes saturated causing scheduling failures — pods remain Pending, controllers time out, and kubectl/emergency ops are slow.Root causes:- API request storms: e.g., mass pod create/delete from rollout, CI loops retrying, or tens of thousands of watchers reconnecting (watch churn).- Controller loop hot-spots: custom controllers or operators performing expensive list/watch or frequent status updates (heavy writes).- Excessive informer resyncs or small-resync intervals across many controllers.- Etcd or apiserver resource limits: CPU, I/O, request concurrency, or etcd leader GC compaction pressure.- Admission webhook slowness or network latencies increasing apiserver request durations.- Misconfigured Horizontal Pod Autoscalers or jobs that create many objects at once.Diagnosis (how I'd investigate):- Observe apiserver metrics (apiserver_request_total, request_latencies_seconds_bucket, longrunning requests) and etcd metrics (etcd_disk_wal_fsync_duration_seconds, leader_changes).- Check apiserver logs for 429/500 responses, audit logs for high request sources and verbs (WATCH/LIST/CREATE/UPDATE).- Use kubectl top and node metrics to ensure scheduler not resource-starved; inspect controller-manager and scheduler metrics for queue depth and reconcile latency.- Trace admission webhooks and network (tcpdump, pprof) to find slow handlers.- Identify offending clients via source IP/user-agent in audit logs and kube-apiserver --audit-policy.Short/Medium-term mitigations:- Throttle or stagger rollouts: reduce parallelism (maxUnavailable), rollout pacing.- Rate-limit noisy clients (API server --max-requests-inflight, --max-mutating-requests-inflight) and add API priority and fairness configuration.- Temporarily increase apiserver and etcd capacity: vertically scale control plane nodes, increase etcd IOPS, add apiserver replicas behind LB.- Disable noncritical admission webhooks and reduce informer resyncs.- Evict or pause custom controllers/operators causing storms.Long-term fixes:- Implement API usage best practices: use watches instead of frequent list polling, exponential backoff on retries, efficient patching (server-side apply/strategic-merge patches).- Harden control plane autoscaling and capacity planning (SLO-driven): simulate rollouts in load tests; define apiserver, etcd SLOs and reserve headroom.- Adopt API Priority and Fairness to protect control-plane-critical traffic.- Refactor heavy controllers to reduce write amplification, batch status updates, and use leader election/sharding for scale.- Improve observability: structured audit logs, request tracing, automated anomaly detection to catch API storms early.Result: Combining immediate throttling and capacity increases with long-term client fixes and QoS for control-plane traffic prevents saturation and ensures rollouts complete reliably.
HardSystem Design
32 practiced
Design a multi-region, active-active architecture for a stateful service that must support global reads and writes (e.g., user profile store). Address data replication (sync/async), conflict resolution, DNS routing and failover, consistency trade-offs, and monitoring you would add to detect replication issues.
Sample Answer
Requirements & constraints:- Global reads and writes, low write latency (~50-200ms), high availability (99.99%), must tolerate region failures, bounded staleness acceptable (configurable), SRE ownership for observability and failover automation.High-level architecture:- Active-active across N regions. Each region hosts full service + local storage (distributed DB per region) and a global replication layer that ships updates to other regions.Data replication:- Hybrid: synchronous within a single region (replica set/RAFT) for local durability; asynchronous multi-region replication for cross-region propagation to avoid write latency spikes.- Use per-object change-log/append-only WAL with causal metadata (vector clocks or Lamport timestamps). Replication streams are ordered per-partition (shard) to simplify convergence.Conflict resolution:- Prefer CRDTs for mergeable state (counters, sets, maps) where possible.- For non-CRDT user-profile fields, use last-writer-wins augmented by vector clocks + application-level merge hooks (e.g., preserve most recent and allow user merge UI for ambiguous cases).- Record per-field metadata (last-modified timestamp, origin region, write-id) to aid automated merges and audits.Routing & failover:- Global traffic via Anycast or a managed Global Load Balancer (GCLB) + Geo-DNS for region affinity. Use health-checked endpoints with short TTLs (e.g., 60s) for fast failover.- Client libraries prefer local region; on write conflicts, clients optionally retry with causal context.- Automated region failover: health checks detect region outage, GCLB removes endpoints; replication resume/repair on region return with replays and anti-entropy.Consistency trade-offs:- Offer tunable consistency per API: strong (read-your-writes) via session tokens and read from write-quorum (may route to leader region or perform read-repair), or eventual for low-latency global reads.- Document SLOs: e.g., eventual replication <= 5s 99th percentile, strong reads within region.Monitoring & detection:- Replication lag metrics per shard & per-region (producer/consumer offsets), stream errors, retry/backoff rates.- Data divergence checks: periodic hash (Merkle tree) comparisons per shard across regions, and sampled end-to-end shadow reads comparing authoritative read vs replica.- Alerts: sustained replication lag > threshold, stream stall, checksum drift, conflict-rate spike, consumer rebalances.- Dashboards: per-region write/read QPS, latency P50/P95/P99, replication queue depth, last-applied LSN per region.- Runbooks & automation: automated repair (replay WAL), throttling on overload, safe-rollbacks, and postmortems; synthetic transactions to detect cross-region inconsistencies early.Operational considerations:- Capacity planning for cross-region bandwidth, backpressure handling, transactional boundaries minimized, encryption in transit, and strong auditing for conflict resolution decisions.
EasyTechnical
32 practiced
Explain what a CDN (Content Delivery Network) and edge caching provide. Give examples of resource types that benefit most from a CDN, explain cache invalidation strategies, and describe when a CDN would not help application performance.
Sample Answer
A CDN (Content Delivery Network) is a distributed network of edge servers that cache and serve content from locations closer to users to reduce latency, offload origin servers, and increase availability. Edge caching stores copies at POPs so requests hit the edge instead of the origin.Resource types that benefit most:- Static assets: HTML/CSS/JS, images, fonts — high cacheability and high request volume- Media files: video/audio, large binaries, software updates — reduce bandwidth and improve throughput- CDN-friendly API responses: read-heavy, mostly-identical responses (public blobs, product catalogs)- Immutable artifacts: versioned assets and container imagesCache invalidation strategies:- TTL-based caching (Cache-Control max-age) for automatic expiry- Versioned URLs / cache-busting (content fingerprinting) for instant invalidation- Explicit PURGE/INVALIDATE API for emergency removals- Stale-while-revalidate / stale-if-error to balance freshness and availability- Surrogate keys to invalidate related groups atomically- Use short TTLs for dynamic resources and longer for immutable onesWhen a CDN won't help:- Highly personalized or per-request dynamic content (user dashboards, SSE, websockets)- Low-latency origin-dependant operations where edge doesn’t reduce critical path (complex auth requiring origin roundtrips)- Small user base concentrated near origin (no geographic distribution gains)- Upload-heavy workloads (CDN helps distribution but not origin ingestion)SRE considerations: monitor cache hit ratio, purge latency, origin request rate, and TLS/HTTP handshake overhead. Use origin shielding to reduce origin load, and balance TTLs vs. freshness to meet SLOs.
HardTechnical
27 practiced
As a Staff SRE you need to convince executives to invest in formal SRE practices (error budgets, SLOs, runbooks, automation). Draft the key elements of your business case: metrics to show, cost vs benefit, risk quantification, and initial pilot plan with measurable KPIs.
Sample Answer
Executive summary: Investing in formal SRE (SLOs, error budgets, runbooks, automation) reduces outage cost, speeds feature delivery, and makes reliability decisions data-driven. Below are the key elements of a concise business case.Metrics to show- Current state: MTTD, MTTR, change failure rate, incident frequency/severity, customer-visible downtime (minutes/month), support ticket volume, on-call hours, engineering time spent on incidents (%).- Business impact: revenue lost per minute/hour of downtime, NPS/CSAT drops correlated with outages, developer cycle time.- Baseline and target SLOs (e.g., 99.95% availability → max 21.9 minutes/month).Cost vs benefit- Costs: tooling (SLO dashboards, runbook platform, automation tooling), training, 3–6 month FTE effort to define SLOs/runbooks, runbook maintenance.- Benefits: reduced downtime cost (use current outage cost * expected % reduction), lower on-call burn (salary savings/retention), faster recovery (MTTR reduction), faster deployments (reduced rollbacks), fewer support escalations.- Example ROI: If current outages cost $200k/year and SRE practices reduce outages by 50%, payback in <12 months after a $75k investment.Risk quantification- Likelihood and impact matrix for top 5 incident types (probability, downtime minutes, revenue impact).- Error budget model showing allowable risk vs. feature velocity tradeoffs.- Sensitivity analysis: how MTTR reductions map to revenue/CSAT improvements.Initial pilot plan (90–120 days)- Scope: one critical service with clear customer impact.- Activities: 1. Define 1–2 SLOs and error budget for the service (weeks 1–2). 2. Implement SLO/reliability dashboard and alerting (weeks 2–5). 3. Create runbooks for top 5 incident types; automate 2 manual recovery steps (weeks 3–8). 4. Run error budget governance for two release cycles and host 2 blameless postmortems (weeks 9–12).- Measurable KPIs: - MTTR reduction (%) target: e.g., from 60m → 20m (66%). - Incident frequency reduction (%) target: e.g., -30%. - Error budget burn rate within acceptable range over 2 cycles. - Reduction in on-call interrupts (alerts/actionable ratio improvement). - Time engineers spend on incidents (% reduction).- Success criteria to scale: meet at least 3 of 5 KPIs and positive stakeholder feedback; present ROI projection for organization-wide rollout.Ask: approval for pilot budget (tooling + 1.5 FTE for 3 months) and executive sponsorship to enforce error-budget tradeoffs.
Unlock Full Question Bank
Get access to hundreds of Cloud Platforms and Infrastructure interview questions and detailed answers.
