Cost Optimization at Scale Questions

Addresses cost conscious design and operational practices for systems operating at large scale and high volume. Candidates should discuss measuring and improving unit economics such as cost per request or cost per customer, multi tier storage strategies and lifecycle management, caching, batching and request consolidation to reduce resource use, data and model compression, optimizing network and input output patterns, and minimizing egress and transfer charges. Senior discussions include product level trade offs, prioritization of cost reductions versus feature velocity, instrumentation and observability for ongoing cost measurement, automation and runbook approaches to enforce cost controls, and organizational practices to continuously identify, quantify, and implement savings without compromising critical service level objectives. The topic emphasizes measurement, benchmarking, risk assessment, and communicating expected savings and operational impacts to stakeholders.

MediumSystem Design

44 practiced

Design CI/CD and IaC safeguards that prevent accidental creation of high-cost resources. Include policy-as-code (e.g., OPA), pre-merge cost estimates, resource quotas, automated PR checks and developer UX considerations so velocity isn't unduly impacted.

Sample Answer

Requirements & constraints:- Prevent accidental high-cost infra while keeping deploy velocity.- Enforce as-code policies, surface pre-merge cost impact, apply quotas, block dangerous PRs automatically but allow fast approved exceptions.- Integrate with Terraform/Cloud providers, CI (GitHub Actions/GitLab), and developer UX tools.

High-level architecture:- Dev PR → CI pipeline runs: static checks, policy-as-code, cost estimate, tests → merge gate enforces quotas/policies → Terraform plan/apply in controlled workspace.

Key components & flow:1. Policy-as-code: OPA/rego (or Conftest) policies stored in repo; evaluate terraform plan JSON. Policies include: deny certain instance sizes, deny public IPs on prod, require tags, and cost thresholds per env.2. Pre-merge cost estimates: In CI run Infracost against the terraform plan to produce diff and a summarized cost change (daily/monthly). Post a comment on PR with clear red/yellow/green signal and cost breakdown.3. Automated PR checks: CI jobs implement: - terraform fmt/validate - plan -> save plan file - OPA policy eval against plan; fail PR on violations - Infracost run -> fail if delta > threshold OR label PR as “expensive” requiring approval4. Resource quotas & guardrails: - Cloud quotas: use provider-side quotas (AWS Service Quotas, GCP quota) and enforce via central account policies (SCPs, IAM). - Kubernetes: ResourceQuota & LimitRange per namespace. - Terraform layer: tag resources with billing code and attach budget alerts; use provider-specific caps (e.g., enforce disk type/size limits).5. Approval & exception flow: - Allow an “expensive change” approval workflow: PR requires one or two approvers from finance/SRE; approval recorded via GitHub protected branch or GitHub Checks re-run with an approved flag. - Automated short-lived escalations: create change request ticket and enable apply only after ticket linked.6. Deployment control: - Use Terraform Cloud/Enterprise or Atlantis to centralize applies and run in isolated CI runners with least privilege. - Apply jobs check final runtime policies and budgets before apply; abort if cloud budget thresholds exceeded.7. Developer UX: - Fast local feedback: provide a thin CLI wrapper (dev-tool) that runs terraform plan -> infracost locally and prints quick delta. - PR feedback is actionable: single-line badge (PASS/WARN/FAIL), cost breakdown, remediation hints (smaller instance, spot/preemptible). - Templates and catalog: curated module registry with pre-approved instance types and sizes per env.8. Observability & metrics: - Log policy violations, cost deltas, approval durations. - Alert when repeated overrides by same team. - Track mean time to approve expensive changes and %blocked merges.9. Scalability & reliability: - CI jobs parallelized per repo; OPA evaluations are fast and cacheable. - Keep policies modular to avoid long eval times; precompile bundles for CI.10. Trade-offs: - Strict denies reduce risk but may slow innovation — mitigate via expedited approvals and pre-approved module catalog. - Relying on Infracost depends on accurate pricing — sync price data and validate with billing.

Example policies (conceptual):- Deny EC2 instance types > m5.4xlarge in non-approved projects.- Fail if monthly cost delta > $500 for staging or > $5000 for prod unless approved.- Enforce ResourceQuota for k8s namespaces and deny new LoadBalancers in dev.

This design balances safety and velocity by failing fast with clear remediation, centralizing apply authority, providing self-serve cost visibility, and offering a documented, expedited approval path for legitimate high-cost needs.

HardSystem Design

46 practiced

Design an end-to-end framework to continuously measure, attribute, and report cost-per-feature and cost-per-customer across a microservices platform handling 10 billion requests/day. Describe telemetry you would collect (and where), how to propagate attribution through service calls, sampling strategies to control pipeline cost, aggregation windows, and how to present uncertainty in reports.

Sample Answer

Requirements & constraints:- 10B req/day (~115k req/s) → telemetry must be low-overhead, highly aggregated, and streamable.- Produce cost-per-feature and cost-per-customer with known uncertainty for operational and billing uses.

High-level architecture:- Instrumentation (sidecar/agent + app): lightweight counters & traces → ingestion (Kafka) → realtime attribution pipeline (Apache Flink/Beam) → OLAP store (ClickHouse/BigQuery) + BI/dashboard.

Telemetry to collect (where):- Per-request (application): trace_id, parent_span, span_duration, service, route, feature_flag_ids, customer_id (if available), response_size, HTTP status.- Host/infra (node agent / cloud telemetry): CPU_seconds, memory_bytes, ephemeral_storage_io, network_bytes, pod/container id, instance_type, price_rate ($/CPU-sec, $/GB-month, $/IO).- Downstream costs: DB ops (RUs / queries), third-party API costs (metered events).- Business mapping: feature metadata (feature_id → owning team) in config DB.

Attribution model & propagation:- Carry trace_id + tags: feature_ids (allow multiple), customer_id, sampling_rate. Tag spans at the entry point with feature/customer context.- Compute per-request cost = sum of: - Compute cost = sum(span_cpu_ms * cpu_rate + span_memory_avg * mem_rate) - Network + IO charges = attributed from span metrics - Downstream metered costs = allocate to caller trace/span proportionally (e.g., by number of related calls or latency) - Shared infra (load balancers, infra reserved): apportioned using proportional usage (requests, CPU-seconds) over the aggregation window.- For multi-feature requests, use configurable allocation rules: exclusive (primary feature gets full cost), proportional (based on time spent in feature-tagged code paths), or fractional equal split. Store allocation policy with feature metadata.

Sampling strategies to control pipeline cost:- Two-tier telemetry: 1. Always-on aggregated counters (per-minute): request_count, error_count, sum_latency, sum_cpu_ms, per-service-feature-customer counters — pre-aggregated at sidecar/agent to reduce cardinality. 2. Traces: head-based probabilistic sampling (very low p) + adaptive tail-based sampling for slow/error/high-cost flows; ensure guaranteed sampling for high-value customers/features (sample rate override).- Maintain and propagate sampling_rate tag to enable unbiased estimators during aggregation.- Use reservoir sampling for high-cardinality customer/feature combinations for exploratory analytics.

Aggregation windows & processing:- Real-time operational window: 1m ingest, 5m rollups for SLOs/alerts.- Cost reporting window: hourly rollups computed from minute-level aggregates; daily and monthly rollups for billing.- Store raw sampled-span events for 7–30 days to support re-attribution; keep aggregated metrics long-term.- Use windowed joins in Flink to combine infra cost rates (per-second) with resource usage counters to compute $.

Handling sampling & uncertainty:- When traces are sampled, use Horvitz-Thompson estimator: weight each sampled event by 1 / sampling_rate to produce unbiased totals.- Compute variance and produce confidence intervals for each aggregated metric using sampling theory (e.g., for weighted sums Var = sum((1/p - 1)*w^2*var_event)).- Propagate uncertainty from multiple sources (sampling, shared-cost allocation assumptions, price-rate volatility). For downstream metered costs, include meter resolution error.- Display: point estimate ± 95% CI, sample rate, contribution of uncertainty sources, and provenance (what telemetry / allocation policy used). For features/customers with high uncertainty, flag and provide recommended actions (increase sampling, instrument more, or use deterministic billing).

Pipeline cost control & accuracy trade-offs:- Pre-aggregate at edge to reduce cardinality (minute buckets, hashed customer groups for low-value customers).- Use eBPF or runtime cgroups accounting for low-overhead CPU/memory attribution at span granularity where possible.- Allow configurable SLAs for accuracy: e.g., produce billing-quality reports with stricter sampling or full capture for high-value customers only.

Operational considerations:- Alert on sudden cost attribution anomalies (e.g., feature cost spike), sampling rate drops, or storage/backpressure.- Periodic reconciliation: compare platform-level cloud bill vs aggregated attributed cost; use residual allocation method to reconcile remaining un-attributed costs and feed back to allocation policies.- Runbacktesting: re-run attribution on stored sampled/raw data to validate models and tighten uncertainty estimates.

Example: computing cost-per-customer (hourly)- Sidecar emits per-minute: requests, sum_cpu_ms, sum_latency per customer_id (or hashed bucket), sampling_rate.- Flink multiplies sum_cpu_ms by cpu_rate ($/ms) and adds apportioned DB $ (from DB metering partitioned by caller_count).- If sampling_rate <1, each sampled trace contributes weight = 1/sampling_rate; variance computed and stored.- Dashboard shows customer X: $120 ± $8 (95% CI), sample_rate=0.01, provenance: 70% aggregated counters, 30% sampled traces.

This framework balances low overhead at 10B req/day with statistically sound attribution, clear provenance, and actionable uncertainty so SREs and finance teams can trust and iterate on cost models.

HardTechnical

44 practiced

You're given 10 proposed cost-savings projects with different expected savings, effort (person-weeks), and operational risks. Define an objective prioritization framework (a scoring model) to rank projects. Explain how to quantify uncertainty and how you'd present the recommended order to engineering and product leadership.

Sample Answer

I’d build a transparent, quantitative scoring model that balances savings, effort, operational risk, and reliability impact, then surface uncertainty and recommended sequencing clearly for engineering and product leadership.

Scoring model (steps):1. Normalize inputs to 0–1: - Expected annual savings S (USD) → S_norm = (S - min)/(max - min) - Effort E (person-weeks) → E_norm = (E - min)/(max - min) - Operational risk R (0–10) from risk assessment → R_norm = R/10 - Reliability impact I (-1 to +1): negative for increased risk, positive for reliability gains → map to 0–1 - Strategic alignment A (0–1) from stakeholder scoring2. Composite score (higher = better): Score = w_s*S_norm + w_e*(1 - E_norm) + w_r*(1 - R_norm) + w_i*I_norm + w_a*A Example weights (adjustable): w_s=0.35, w_e=0.20, w_r=0.20, w_i=0.15, w_a=0.10

Quantifying uncertainty:- Capture uncertainty for S and E as distributions (e.g., P10/P50/P90 estimates from engineering estimates or historical variance).- Run Monte Carlo simulation (10k runs) per project sampling S and E distributions and varying R within a range. Produce median expected savings, 90% confidence intervals, and probability of positive net benefit within X months.- Compute sensitivity: which inputs (savings, effort, risk) drive score variance.

Presenting recommended order:- Deliver a ranked table with columns: project, median annual savings, P90/P10 range, expected effort, composite score, expected payback (weeks), reliability impact, risk summary, dependencies.- Visuals: bar chart of median savings with error bars (uncertainty), waterfall showing cumulative monthly savings if executed in recommended order, and a risk vs reward scatter with size = effort.- For leadership, highlight 3 recommended tranches: - Quick wins (high score, low effort, low risk) — execute immediately with owners and SLAs. - Strategic investments (high savings, medium effort/risk) — schedule over next quarter; include pilot plan. - Low priority/risky (low score or negative reliability impact) — defer or re-scope.- Include execution details: canary rollout plan, monitoring SLOs & error budget impact, rollback criteria, test plan, and required approvals.

This framework makes prioritization reproducible, shows trade-offs, quantifies uncertainty so leaders see upside and downside, and provides an operational plan that SREs can own for safe delivery.

EasyTechnical

42 practiced

In Kubernetes, explain resource requests and limits and how they affect scheduling, bin-packing efficiency, and cloud costs. Propose a pragmatic default policy for setting requests and limits for new services to balance utilization and risk.

MediumTechnical

46 practiced

A single SQL query consumes 40% of your database CPU. Describe step-by-step how you would analyze and optimize it: profiling (EXPLAIN ANALYZE), indexing, query rewrite, caching approaches, denormalization, sharding, and how you'd estimate and validate cost savings after changes.

Sample Answer

1) Triage & baseline- Capture current metrics: top SQL by CPU (pg_stat_statements or MySQL slow log), total DB CPU, query frequency, p95/p99 latency.- Snapshot EXPLAIN ANALYZE and collect a few actual runtime traces.

2) Profiling (EXPLAIN ANALYZE)- Run EXPLAIN (ANALYZE, BUFFERS, VERBOSE) <query>; capture actual vs planner estimates, node-level times, and buffer/io counts.- Look for sequential scans, large nested-loop joins, high-decode time, or repeated sorts/hash spills.Example:

sql

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT ... FROM orders JOIN customers ON ... WHERE ...;

- Action: identify hottest operators (e.g., Seq Scan on orders = 80% of time).

3) Indexing- Add supporting indexes for WHERE, JOIN, ORDER BY. Prefer covering indexes to avoid lookups.- Test with hypothetical indexes (Postgres pg_hint_plan or CREATE INDEX CONCURRENTLY in staging).- Re-run EXPLAIN ANALYZE to confirm index is used and reduces rows processed.

4) Query rewrite- Simplify: remove SELECT *, push predicates earlier, rewrite correlated subqueries to joins, replace DISTINCT with window functions if appropriate.- Materialize expensive subquery once (WITH ... MATERIALIZED in Postgres or temp tables).

5) Caching approaches- Short-term: result caching in app or Redis for idempotent reads with TTL based on staleness tolerance.- DB-level: set up prepared statements, statement_plan_cache; use pg_prewarm for frequently used pages.- Validate cache hit rate and eviction effects.

6) Denormalization & precomputation- Create summary tables, aggregate rollups, or maintain incremental materialized views updated via triggers or background jobs (logical replication/CDC to analytics store).

7) Sharding / scaling- If single-query load is due to data volume, consider horizontal partitioning (range/hash partitioning) to prune partitions on query.- For read-heavy scale: add read replicas and route read traffic; ensure replicas can serve necessary data (replica lag considerations).- As last resort, application-level sharding for hottest tables.

8) Estimating & validating cost savings- Quantify baseline: measure avg CPU-seconds per query = (query_time * avg_concurrency). Multiply by qps to get CPU core-seconds per minute.- After change, run load test or A/B rollout to measure new CPU-seconds, latency, and error rates.- Calculate savings = (baseline CPU-seconds - new CPU-seconds) / baseline.- Validate on staging with representative data and on canary production (5-10% traffic) while monitoring SLOs, replica lag, cache hit rates, and rollback safety.

9) Operationalize- Add monitoring dashboards and alerts for query latency, DB CPU, slow-query count, cache hit rate, replica lag.- Document changes, migration steps (index creation concurrently), and rollback plan.

Example validation metric: if query was 40% DB CPU and optimization reduces per-query CPU by 5x while qps unchanged, expected DB CPU ~40%/5 = 8% → frees ~32% CPU capacity; confirm with 24–72h of production telemetry and cost accounting before full rollout.

Unlock Full Question Bank

Get access to hundreds of Cost Optimization at Scale interview questions and detailed answers.

Join thousands of developers preparing for their dream job.