Scalability and Systems Resource Management Questions

Design and operational practices for managing compute and platform resources as systems scale. Covers autoscaling, resource pooling, orchestration, cost trade offs between always on versus on demand provisioning, and architectural choices that affect resource utilization and performance. Candidates should be prepared to discuss capacity planning for infrastructure, metrics and alerts for autoscaling, and cost versus performance decisions for high availability systems.

EasyTechnical

90 practiced

Explain the difference between horizontal and vertical autoscaling for compute resources. In your answer describe at least three operational considerations (statefulness, startup latency, cost impact, and failure modes) that would make you choose one approach over the other for a customer-facing web service that must maintain <100ms median latency.

Sample Answer

Horizontal autoscaling adds/removes instances (more machines/pods); vertical autoscaling increases/decreases resources on the same instance (CPU/memory). Both aim to meet load, but their operational profiles differ and those differences drive the choice for a customer-facing web service with a <100ms median latency SLO.

Key considerations:

- Statefulness: Horizontal scaling favors stateless services (easy to load‑balance, scale out/in). If the service holds session state in‑memory, vertical scaling avoids complex sticky sessions or session replication, so vertical may be chosen when refactoring state is infeasible. Best SRE practice: prefer stateless horizontally-scalable design (session stores, caches) to enable robust scaling.

- Startup latency: New instances take time to boot, initialize, JIT, or warm caches (pod start + app warmup). If cold-starts exceed budget and cause p95/p99 spikes above 100ms, relying solely on horizontal autoscale is risky unless you maintain a warm pool or predictive scaling. Vertical resizing (hot-add vCPU/memory) can be faster for small adjustments, reducing risk of latency spikes.

- Cost impact: Horizontal scaling can be cost-efficient at scale (many small instances) and offers better fault isolation; however, it may incur more per‑instance overhead (OS, replication). Vertical scaling can be cheaper for steady loads but hits steep cost jumps for large VMs and reduces multi-tenant packing efficiency. Consider cost vs. elasticity trade-offs and use autoscaling policies that minimize waste (scale-to-demand, scheduled scaling).

- Failure modes: Horizontal scaling provides redundancy—one instance failure affects a fraction of traffic; rolling failures are manageable. Vertical scaling creates single‑node risks: if that node fails, higher blast radius. For high availability and low-latency requirements, horizontal with health checks and multi-AZ deployment reduces risk.

Recommendation (SRE lens): For a customer-facing web service with <100ms median latency, prefer horizontal autoscaling combined with techniques to mitigate startup latency: keep instances stateless, use a warm pool or predictive/scheduled scaling, tune readiness probes so only warmed instances receive traffic, and use fast caches/CDNs. Use vertical scaling only for short, predictable bursts or when stateful legacy constraints make refactor impractical—while limiting blast radius via redundancy (small vertical steps, multi-node replication). Monitor latency SLOs, use autoscaler cooldowns, and run chaos tests to validate failure modes.

EasyTechnical

75 practiced

Compare cost and performance trade-offs between always-on (steady-state) capacity and on-demand (autoscaled/ephemeral) provisioning for a public API that has heavy daytime usage and light nights. Include numerical examples showing how to compute monthly cost difference given instance hourly price and average utilization.

MediumTechnical

83 practiced

You're responsible for defining SLOs and an error budget for a new service that must be highly available for customers. Given target availability 99.95% over 30 days, compute the allowed downtime. Describe how you would link autoscaling policies to error budget consumption and provide an example policy where autoscaling is restricted when error budget is near exhaustion.

Sample Answer

Calculate allowed downtime:- 30 days = 30 × 24 × 60 = 43,200 minutes.- Target availability = 99.95% → allowed unavailability = 100% − 99.95% = 0.05% = 0.0005 fraction.- Allowed downtime = 0.0005 × 43,200 = 21.6 minutes per 30-day window (≈21 minutes 36 seconds).

How to link autoscaling to error budget consumption (concept + rationale)- Goal: use error budget as a shared risk signal. When plenty of error budget remains, allow more aggressive automation (scale-outs, confident deployments). As error budget gets consumed, reduce risk by constraining automated changes that could destabilize the service.- Mechanism: continuously evaluate two inputs: (a) error budget remaining (percentage of window), (b) error budget burn rate (recent consumption vs. expected). Feed these into an autoscaling policy controller (or orchestration layer) that adjusts scaling aggressiveness and deployment behaviors.- Benefits: prevents automatic actions (rapid up/down scaling, frequent deployments) from pushing system past SLO, gives on-call time to investigate.

Example policy (practical, actionable)- Inputs: - EB = error budget remaining (%) over 30-day window - BurnRate = error budget consumption rate (e.g., last 1h vs expected)- Thresholds: - Normal: EB > 50% - Caution: 10% < EB ≤ 50% - Exhausted: EB ≤ 10% or BurnRate > 2× expected

- Rules: 1. Normal (EB > 50%) - Autoscaler: allow max replica count = cluster limit (e.g., up to 50 pods), scale-out step up to +5 pods per 1m, normal cooldowns. - Deployments: allow automated rollouts and canaries. 2. Caution (10% < EB ≤ 50%) - Autoscaler: cap scale-out rate (e.g., max +2 pods per 5m), enforce longer cooldowns, reduce target CPU threshold (be more conservative) to avoid oscillation. - Deployments: require canary with extended observation window (e.g., 30m). Pause non-essential feature flags. 3. Exhausted (EB ≤ 10% or BurnRate > 2×) - Autoscaler: disable automatic scale-out (only allow scale-in to reduce blast radius) OR restrict scale-outs to a small emergency window after on-call approval. Set minimum replicas to a safe baseline. - Deployments: block automated deployments; require manual approval and strict canary/testing. - Operations: page on-call, enable additional monitoring/traffic shaping (reduce non-critical traffic), consider rolling back recent changes.

Concrete Kubernetes-style example (numbers)- Normal: HPA target CPU 60%, max replicas 50, scaleUpPolicy: maxSurge +5 pods / 60s.- Caution: HPA target CPU 55%, max replicas 30, scaleUpPolicy: maxSurge +2 pods / 300s.- Exhausted: HPA scale-up disabled (kubectl patch to min/max replicas = current), set replicas = max(current, baseline 10), block CI/CD pipeline, send pager.

Tie to burn-rate alarms and automation- Implement automated runbook: - If BurnRate > 2×: immediately switch to Caution tier. - If EB ≤ 10%: immediately switch to Exhausted tier and notify SRE. - When EB recovers above thresholds for a sustained period (e.g., 24h), revert to more permissive tiers.- Ensure fail-safes: policy changes are logged, reversible, and limited by RBAC (ops can override with approval).

Trade-offs and considerations- Restricting autoscaling can increase short-term latency if demand spikes; mitigate by reserving a baseline capacity sized for typical peak or by pre-warming instances.- Tune thresholds to your risk tolerance and business impact; integrate with cost policies so cost vs. availability trade-offs are explicit.- Test the policy in staging and during game days to ensure it behaves as expected.

This approach protects the SLO by reducing change-induced risk as error budget depletes while still allowing automated responsiveness when safe.

HardTechnical

73 practiced

You are asked to design Terraform modules and a CI pipeline to provision autoscaling groups (or managed instance groups) across environments. Describe module interfaces, state management, promotion workflow, and how to test autoscaling behavior safely as part of the pipeline.

Sample Answer

Requirements & approach:- Goal: reusable Terraform modules to create ASGs/MIGs across envs, manage state safely, CI promotion pipeline, and safe autoscale testing.

Module design (interface):- Root module: environment wrapper (inputs: env, region, vpc_id, subnet_ids, tags, lifecycle_policy).- compute module (reusable): inputs: name, instance_template_id / launch_template, desired_count, min_size, max_size, scaling_policies (cpu, custom metric ARNs), health_check_type, cooldowns, update_strategy (rolling, surge), target_group_ids, autoscaling_notifications.- outputs: asg_id, autoscaling_policy_ids, target_group_arns, current_desired- Parameterize low-risk defaults and allow override via variables.tf with validation and descriptions.

State management:- Remote state per environment using a backend (e.g., S3 + DynamoDB lock, GCS + locking, or Terraform Cloud workspaces). One state per env and logical service to limit blast radius.- Use workspaces sparingly; prefer per-environment state files keyed by service.- Locking enabled to avoid concurrent applies.

Promotion workflow / CI:- Git flow: feature branch → PR → merge to staging branch → promote to prod.- Pipeline stages: 1. plan (on PR): terraform init/plan with -var-file=env.staging.tfvars, save plan artifact. 2. policy checks: run Sentinel / OPA / terrascan (security, cost caps). 3. integration tests in ephemeral sandbox (see testing below). 4. manual approval to deploy staging: apply plan artifact to staging. 5. after verification, create a promotion PR that uses same plan with prod variables, gated by approvals and deploy windows, then apply to prod.- Keep plan artifacts signed (hash) to ensure apply matches reviewed plan.

Testing autoscaling safely in pipeline:- Use a dedicated test environment that mirrors production but small scale and isolated.- Blue/green or canary pattern for rollout to limit impact.- Automated autoscale tests: - Deploy ASG with metric-based scaling hooked to a mock/custom metric (e.g., CloudWatch custom metric or Prometheus alertmanager) that CI can push synthetic load signals to. - Use load generators against instance health endpoints or apply synthetic metric increments (push custom metric values) to trigger scale-up/down without stressing production systems. - Verify behavior: scale-up occurs within expected timeframe, new instances pass health checks and register with load balancer, scale-down respects cooldowns and drains connections gracefully.- Safety controls: - Test environment IAM scoped tightly; use quotas to prevent runaway scaling. - Cost and instance count caps in module defaults and CI variable overrides. - Rollback on failure: detect unmet conditions and revert to previous desired_count via IaC or force-decrease policies.- Observability & assertions: - Pipeline asserts: metric change observed, ASG desired capacity changes, target group healthy count increases, no failed lifecycle hooks. - Collect logs/metrics for the run and attach to the CI job.

Operational considerations & tradeoffs:- Use lifecycle policies and update_strategy to avoid capacity loss during updates.- Prefer infrastructure-as-code promotion (plan -> apply artifact) to maintain immutability and audit.- Balance test realism vs cost: synthetic metric injection is low-cost and safe; full load tests reserved for periodic performance validation.

This approach yields modular, auditable Terraform modules, safe state handling, controlled promotion, and automated, low-risk autoscaling validation as part of CI.

EasyTechnical

123 practiced

Why are stateful services generally harder to autoscale than stateless ones? Provide three strategies for handling state when scaling (e.g., sharding, sticky-sessions, externalize state) and describe the operational trade-offs of each.

Sample Answer

Stateful services are harder to autoscale because instances hold per-client or in-memory data (sessions, caches, leader state). That creates tight coupling between clients and specific instances, makes instance lifecycle (start/stop) riskier, and complicates consistency, failover, and load redistribution compared to stateless services where any request can go to any instance.

Three strategies and trade-offs:

1) Sharding (partition state across nodes)- Description: Split data by key (user ID, range, hash) so each instance owns a subset.- Operational trade-offs: + Scales horizontally and balances load predictably. − Requires re-sharding when scaling or recovering (data movement), adds routing logic and cross-shard transactions complexity. Needs monitoring for hot shards and automation for rebalancing.

2) Sticky sessions / session-affinity- Description: Load balancer routes a client to the same instance (cookie/IP affinity).- Operational trade-offs: + Simple to implement; minimal app change. − Reduces effective elasticity (some instances can be overloaded while others idle), harder to do rolling updates or failover without session migration, and increases blast radius if an instance fails. Need health checks and session replication for resilience.

3) Externalize state (central store or distributed cache)- Description: Move state to durable/shared services (Redis, databases, object stores); make app instances ephemeral.- Operational trade-offs: + Enables true stateless scaling, simpler deployments, easier failure recovery. − Introduces dependency on external storage availability and latency; you must provision, scale, and HA the store, and handle consistency, TTLs, and increased network traffic. Monitoring and capacity planning shift to the state store.

As an SRE, choose based on workload: low-latency per-request needs favor in-memory with sharding + careful rebalancing; simple web apps may use sticky sessions temporarily; for long-term reliability and autoscaling, externalizing state is usually preferable with strong monitoring, capacity automation, and tested failure scenarios.

Unlock Full Question Bank

Get access to hundreds of Scalability and Systems Resource Management interview questions and detailed answers.

Join thousands of developers preparing for their dream job.