Cost Optimization at Scale Questions

Addresses cost conscious design and operational practices for systems operating at large scale and high volume. Candidates should discuss measuring and improving unit economics such as cost per request or cost per customer, multi tier storage strategies and lifecycle management, caching, batching and request consolidation to reduce resource use, data and model compression, optimizing network and input output patterns, and minimizing egress and transfer charges. Senior discussions include product level trade offs, prioritization of cost reductions versus feature velocity, instrumentation and observability for ongoing cost measurement, automation and runbook approaches to enforce cost controls, and organizational practices to continuously identify, quantify, and implement savings without compromising critical service level objectives. The topic emphasizes measurement, benchmarking, risk assessment, and communicating expected savings and operational impacts to stakeholders.

MediumTechnical

54 practiced

You run a 10,000 QPS text generation API with a 100ms P95 latency SLO. Describe a batching strategy that improves GPU utilization while respecting the SLO. Include batch sizing policy, dynamic batching triggers, latency compensation, and how you would simulate/evaluate tradeoffs in staging.

Sample Answer

Approach (goal): maximize GPU throughput by forming large enough batches to reach high GPU utilization, while ensuring 95th-percentile end-to-end latency ≤100ms. Use a dynamic, deadline-aware batching policy that adapts to traffic and per-request latency budget.

Batch sizing policy- Hard caps: max_batch_size determined by GPU memory and measured throughput sweet spot (e.g., where tokens/sec saturates).- Soft target: prefer batch_size_target that yields good efficiency (from perf curves). Maintain min_batch_size = 1 to avoid starvation.

Dynamic batching triggers- Two triggers combined: 1. Size trigger: flush immediately when current queue reaches max_batch_size. 2. Time trigger (max_wait): flush when oldest request has waited max_wait_ms.- Adaptive max_wait: continuously tuned so that observed P95 ≈ target. Start with conservative max_wait (e.g., 5ms) and increase if GPU underutilized, decrease if latency drifts up.- Priority / deadline buckets: requests annotated with remaining latency budget; those near deadline bypass batching or use tiny max_wait.

Latency compensation (deadline-aware admission)- Estimate batch processing time: T_proc(batch_size, avg_tokens) from microbenchmarks (linear + offset model: T = α + β * batch_size * avg_tokens).- For each incoming request compute remaining_budget = SLO - network_overhead - observed queueing_estimate. If estimated T_proc for including this request would exceed remaining_budget for its percentile target, mark it “urgent” and either send immediately (no-batch) or place in an urgent bucket with tiny max_wait.- Early-release policy: when oldest request in batch reaches its personal deadline threshold, flush even if batch smaller than target.- Backpressure: when system is overloaded, shed low-priority requests or respond with graceful degradation (smaller model / cached responses).

Monitoring feedback loop- Continuously measure P50/P95/P99, GPU utilization, queue lengths, batch size distribution.- Controller adjusts max_wait and batch_size_target every few seconds using PID or simple proportional control to track both utilization and P95.

Simulation / staging evaluation- Build a discrete-event simulator using real arrival traces (Poisson + bursts, diurnal patterns) and measured per-batch latency model from perf runs (α, β).- Scenarios: - Steady 10k QPS, variable token lengths - Bursts (2–5x) and flash spikes - Tail-latency stress tests- Metrics to collect: P50/P95/P99 latency, GPU occupancy (%), average batch size, request queue size, dropped/served.- Sweeps: vary max_wait, max_batch_size, priority thresholds to plot Pareto frontier (GPU utilization vs P95).- Validate in staging with traffic replay or shadowing: run policy in mirror mode to measure impact without affecting production, then progressive rollout (canary) observing SLO.

Trade-offs and tuning- Larger max_wait increases batch size and GPU utilization but adds queueing latency—use adaptive controller to balance.- Bypass for urgent requests sacrifices some utilization for latency guarantees.- Use mixed strategies: small-model fast path for latency-critical requests; large-model batched path for high-throughput noncritical requests.

This approach provides a practical, measurable way to improve GPU utilization while guaranteeing the 100ms P95 SLO through deadline-aware batching, adaptive timers, and staged simulation-driven tuning.

MediumSystem Design

55 practiced

Design telemetry and instrumentation to attribute cloud cost to specific product features and experiments for an AI platform. Specify events, labels, aggregation windows, and an approach to join cost billing data with feature usage at daily granularity.

Sample Answer

Requirements:- Attribute daily cloud costs to product features and experiments (A/B tests) for an AI platform.- Accuracy within engineering tolerance (~day-level), scalable to millions of events, GDPR-safe (no PII).- Low overhead on inference/training pipelines.

High-level architecture:- Instrumentation -> Event collector (Kafka) -> Enrichment & feature lookup -> Aggregation pipeline (Spark/Flink) -> Cost joiner -> Data warehouse (BigQuery/Snowflake) -> BI + alerting

Events & labels (sent from service at runtime):- event_type: {inference_call, batch_job_start, batch_job_end, model_train_start, model_train_end, experiment_assignment}- timestamp (UTC)- request_id / job_id (UUID)- feature_id (string) — product feature using the model (e.g., "recommendation_v2")- experiment_id (nullable) — assigned experiment variant- model_id, model_version- resource_tags: {cluster, node_pool, gpu_type}- compute_meta: {cpu_seconds, gpu_seconds, memory_gb_seconds} — optional sampled counters for fine granularity- cost_unit_hint (optional): estimated units consumed

Aggregation windows:- Primary: daily UTC window (aligned with billing)- Secondary: hourly for anomaly detection and debugging- Session-level: for multi-call flows, aggregate by request_id/job_id before daily rollup

Join approach to billing:1. Ingest cloud billing/export files (line items) with resource identifiers and hourly/daily cost and resource tags.2. Normalize resource_tags from events to match billing tags (enforce tag schema).3. Produce two rollups per day: - Usage rollup: sum compute_meta (cpu/gpu/memory seconds) grouped by date, feature_id, experiment_id, model_id. - Billing rollup: sum cost by date and resource tag combinations.4. Join on (date, resource_tag) to allocate cost to features proportionally: - For each resource_tag group, compute feature_fraction = usage_of_feature / total_usage_on_resource. - Attributed_cost_feature = billing_cost_resource * feature_fraction. - Where compute_meta absent, fall back to equal-share by request counts weighted by estimated per-call cost hints.5. Store results in warehouse table: date, feature_id, experiment_id, model_id, attributed_cost, raw_usage, allocation_method.

Key considerations & trade-offs:- Accuracy improves with detailed compute_meta; if impossible, use probabilistic sampling.- Ensure strict tag governance across infra and instrumentation.- Handle cold-start GPU allocation: include job-level start/end to capture idle allocation.- Privacy: avoid user identifiers; aggregate before analytics.

Outcome:Daily dashboards of cost per feature/experiment, CI alerts when experiment cost exceeds thresholds, and drill-down to model/version and resource type.

HardSystem Design

38 practiced

Design a cost-optimized inference platform for a 70B-parameter LLM that must serve 1M requests/day with a 200ms P95 SLO and a target cost of under $0.50 per 1k requests. Discuss model serving strategy (sharding, tensor-slicing, quantization), caching, autoscaling, and hardware selection. Provide estimated trade-offs.

Sample Answer

Requirements clarification:- 70B-parameter LLM, 1,000,000 requests/day (~11.6 RPS avg, but traffic spikes → design for P95 200ms), cost target <$0.50 per 1k requests → daily budget ≤ $500 → hourly budget ≈ $20.8.

High-level approach:- Use aggressive quantization (4-bit/GPTQ or AWQ) + operator fusion to fit a full model replica on a small number of GPUs.- Serve with model-parallel replicas (tensor-parallel + pipeline where needed) and a stateless front layer that handles tokenization, caching, and batching.- Add a request cache (short-term response cache + KV cache for recent contexts) and autoscaling focused on replica warm pools.

Model serving strategy- Quantization: Convert to 4-bit (or hybrid 8/4-bit, keep first/last layers in 16-bit if necessary) — reduces memory ~4x vs fp16, minimal perplexity hit if using good calibration. Trade-off: slight accuracy drop, big cost/throughput gain.- Sharding / tensor-slicing: Use tensor parallelism (TP) across 2 GPUs per replica rather than large pipeline stages; for 70B in 4-bit many 40–48GB GPUs (L4/L40S/RTX 6000 Ada/LANL instances) can host full model across 1–2 GPUs. If using larger GPUs (A100/H100 80GB) you can run single-GPU replicas (simpler, lower communication). Trade-off: more GPUs per replica increases network overhead and latency; single-GPU replicas reduce inter-GPU comm but cost more per GPU-hour.- Mixed strategy: prefer single-GPU replicas on 80GB-class GPUs if budget allows; otherwise 2-way TP on cheaper 48–64GB GPUs.

Caching- LRU response cache for identical prompts (important if many repeats) — returns in <5ms.- KV-cache reuse for recent sessions: keep per-session cached keys in memory to avoid recomputing past token keys (reduces decode cost heavily). Evict by TTL/size.- Bloom filter or hashed prompt detection to quickly route repeated requests to cache.

Autoscaling & batching- Maintain a small warm pool (N warm replicas) to satisfy P95 latency; scale additional replicas based on queue length and CPU/GPU utilization.- Use tiny dynamic batching (batch size 1–8) to maximize utilization without exceeding 200ms SLO. For generation-heavy requests, cap tokens-per-request and apply backpressure.- Fast scale-up: keep pre-warmed containers or use checkpointed model snapshots on local SSD to reduce cold-starts to ~30–60s.

Hardware selection & deployment- Preferred: A100/H100 80GB single-GPU replicas for simplest low-latency path. If too costly, target 48–64GB inference GPUs (L4, L40S) with 2-GPU TP replicas.- Network: NVLink or equivalent intra-node with RDMA for multi-GPU shards to keep inter-GPU latency low.- Storage: model on local NVMe for fast load; central object store for cold backups.

Estimated numbers & cost trade-offs (example scenario)- Option A (low-latency higher-cost): Run 4-bit 70B on single A100/H100 80GB instances. Assume each replica handles ~6–10 RPS at P95 with 150–200ms latency. Need ~2–3 replicas to handle peaks + warm pool → say 4 replicas. If GPU cost is high, this may exceed budget but minimizes latency and complexity.- Option B (cost-optimized): 2-GPU 48–64GB replicas with 4-bit: each replica ~3–6 RPS. Need ~6–8 replicas including warm pool → more GPUs but cheaper per-GPU-hour. Communication overhead increases latency but still can meet 200ms with tuned code/fusion.- Cost math framing: daily budget $500 → hourly $20.8. If a GPU costs $2/hr, you can run ~10 GPUs within budget (2–5 replicas depending on GPUs/replica). If GPU is $0.5/hr (cheaper inference cards), you can run more replicas and keep latency low. Choose hardware where (replica throughput * uptime) meets RPS targets within $20.8/hr.

Trade-offs summary- Quantization: huge cost/throughput wins; small accuracy loss.- Single-GPU replicas: lower latency, easier autoscale, higher per-hour cost.- Multi-GPU TP: lower per-replica hardware cost, more network IPC overhead, more complex scheduling.- Caching/KV: very high ROI — reduces decode operations and cost significantly for repeated or long-context sessions.- Autoscaling: warm pool is essential for P95; cold-start reduction strategies critical.

Recommendation- Start with 4-bit quantized model running on 80GB-class GPUs if possible to minimize TP comm. Implement KV caching and short-term response cache. Size warm pool to cover sudden spikes. Monitor tail latency and model quality; if cost exceeds target, move to cheaper 48–64GB GPUs with 2-way TP and increase caching to reduce required replicas.

EasyTechnical

44 practiced

You are asked to compute unit economics for an AI inference service. Given: total monthly bill = $180,000, total inference requests = 30,000,000, active monthly users = 500,000. Calculate: (1) cost per request, (2) cost per active user, and (3) sensitivity: if requests increase 20% but fixed costs remain constant, how do the metrics change? State assumptions and rounding.

MediumTechnical

39 practiced

You have telemetry tables: requests(request_id, user_id, endpoint, latency_ms, timestamp) and costs(node_id, start_time, end_time, cost_usd). Write a SQL query (standard SQL) to compute average cost per request for June 2025 by joining request time to node billing windows. Assume each request maps to the node serving it via node_id available in requests table.

Sample Answer

Approach: join each request to the billing window of the node that contains the request timestamp, compute cost-per-request for that window (cost_usd / requests_in_window), then average across all June 2025 requests. This prorates a window's cost uniformly across requests that occurred inside that window.

SQL implementation:

sql

WITH june_requests AS (
  SELECT *
  FROM requests
  WHERE timestamp >= TIMESTAMP '2025-06-01 00:00:00'
    AND timestamp <  TIMESTAMP '2025-07-01 00:00:00'
),
requests_per_window AS (
  -- count requests per node billing window
  SELECT
    c.node_id,
    c.start_time,
    c.end_time,
    c.cost_usd,
    COUNT(r.request_id) AS req_count
  FROM costs c
  LEFT JOIN june_requests r
    ON r.node_id = c.node_id
   AND r.timestamp >= c.start_time
   AND r.timestamp <  c.end_time
  GROUP BY c.node_id, c.start_time, c.end_time, c.cost_usd
),
attrib AS (
  -- assign cost-per-request to each request by joining to its window
  SELECT
    r.request_id,
    r.user_id,
    r.endpoint,
    r.latency_ms,
    r.timestamp,
    w.cost_usd,
    w.req_count,
    CASE WHEN w.req_count > 0 THEN w.cost_usd::numeric / w.req_count ELSE 0 END AS cost_per_request
  FROM june_requests r
  JOIN requests_per_window w
    ON r.node_id = w.node_id
   AND r.timestamp >= w.start_time
   AND r.timestamp <  w.end_time
)
SELECT
  COUNT(*) AS total_requests,
  SUM(cost_per_request) AS total_cost_usd,
  AVG(cost_per_request) AS avg_cost_per_request_usd
FROM attrib;

Key points:- Filter requests to June 2025 first to limit data scanned.- Use LEFT JOIN when counting so windows with zero requests are included (req_count = 0).- If a request falls outside any window it will be excluded by the final JOIN — you can switch to LEFT JOIN + COALESCE to include and attribute zero cost if desired.

Alternatives / trade-offs:- Prorate by window duration (cost_usd / seconds) and multiply by a request weight if requests considered instantaneous — simpler but assumes uniform time-weighted usage.- If requests can overlap multiple windows (unlikely for instant timestamps), ensure consistent tie-breaking if boundaries touch (use < end_time as above).

Edge cases:- Missing costs for a node/time => request will not join; decide whether to treat cost as 0 or exclude.- Very large windows with few requests may skew per-request cost; consider smoothing or monthly allocation strategies.

Unlock Full Question Bank

Get access to hundreds of Cost Optimization at Scale interview questions and detailed answers.

Join thousands of developers preparing for their dream job.