Performance Cost Optimization & Resource Efficiency Questions

Approach

Cost-per-request here comes from two independent components that scale differently: compute cost, driven by how much of the accelerator's real, achieved throughput one request consumes, and egress cost (the cost of data leaving the provider's network to reach the client), driven purely by response size, independent of compute. The function estimates achieved throughput from peak FLOPS times a realistic achieved fraction (never the peak number itself), sizes a fleet to hit the target request rate under a utilization cap, and adds the two cost components together.

import math

def cost_per_request(model_flops_per_inference, instance_flops, instance_hourly_price,
                      expected_gpu_utilization, network_bytes_per_request,
                      egress_price_per_gb, target_requests_per_second,
                      utilization_cap=0.7):
    """
    model_flops_per_inference: FLOPs needed for ONE forward pass.
    instance_flops: the instance's PEAK FLOPs/sec (vendor spec-sheet number).
    expected_gpu_utilization: fraction of PEAK flops actually achieved in practice (0-1).
        Real workloads rarely hit peak because of memory-bandwidth limits, kernel
        launch overhead, and batch-size effects, so this is the key input that
        translates spec-sheet numbers into real throughput.
    utilization_cap: headroom cap so instances aren't sized to run at 100% busy.
    """
    effective_rps_per_instance = (instance_flops * expected_gpu_utilization) / model_flops_per_inference
    usable_rps_per_instance = effective_rps_per_instance * utilization_cap

    instance_count = max(1, math.ceil(target_requests_per_second / usable_rps_per_instance))
    fleet_hourly_cost = instance_count * instance_hourly_price

    seconds_per_hour = 3600
    compute_cost_per_request = fleet_hourly_cost / (target_requests_per_second * seconds_per_hour)

    bytes_per_gb = 1024 ** 3
    egress_cost_per_request = (network_bytes_per_request / bytes_per_gb) * egress_price_per_gb

    return {
        "effective_rps_per_instance": effective_rps_per_instance,
        "instance_count": instance_count,
        "fleet_hourly_cost": fleet_hourly_cost,
        "compute_cost_per_request": compute_cost_per_request,
        "egress_cost_per_request": egress_cost_per_request,
        "total_cost_per_request": compute_cost_per_request + egress_cost_per_request,
    }


if __name__ == "__main__":
    # Illustrative inputs only, not a published benchmark for any real accelerator.
    result = cost_per_request(
        model_flops_per_inference=2e11,     # 200 GFLOPs per forward pass
        instance_flops=1e14,                # 100 TFLOPs/sec peak (spec-sheet number)
        instance_hourly_price=3.00,
        expected_gpu_utilization=0.30,      # 30% of peak achieved in practice
        network_bytes_per_request=50_000,   # 50 KB response payload
        egress_price_per_gb=0.09,
        target_requests_per_second=500,
        utilization_cap=0.7,
    )
    print(f"Effective throughput per instance: {result['effective_rps_per_instance']:.1f} rps")
    print(f"Recommended instance count: {result['instance_count']}")
    print(f"Fleet hourly cost: ${result['fleet_hourly_cost']:.2f}/hr")
    print(f"Compute cost per request: ${result['compute_cost_per_request']:.8f}")
    print(f"Egress cost per request:  ${result['egress_cost_per_request']:.8f}")
    print(f"Total cost per request:   ${result['total_cost_per_request']:.8f}")
    print(f"Total cost per 1M requests: ${result['total_cost_per_request'] * 1_000_000:.2f}")

Output:

Effective throughput per instance: 150.0 rps
Recommended instance count: 5
Fleet hourly cost: $15.00/hr
Compute cost per request: $0.00000833
Egress cost per request:  $0.00000419
Total cost per request:   $0.00001252
Total cost per 1M requests: $12.52

Key points

• expected_gpu_utilization is doing the real work here: it is what separates a naive (and wrong) calculation using peak spec-sheet FLOPS from a realistic one, and it should come from an actual benchmark of this model on this hardware, not a guess.
• Compute cost and egress cost are computed independently and only added at the end, matching how they are actually billed (one is a compute-hours charge, the other a bytes-transferred charge), which keeps each easy to audit on its own.
• instance_count uses the same over-provisioning pattern as any capacity-planning function: round up, and cap planned utilization below 100% for burst headroom.

Complexity

O(1); every step is a fixed arithmetic calculation with no loops over the input.

Edge cases

• If expected_gpu_utilization is 0 or effectively so, usable_rps_per_instance becomes 0 and the division would raise a divide-by-zero error; a production version should validate that the utilization inputs are strictly greater than 0 before dividing.
• Very small network_bytes_per_request values make egress_cost_per_request negligible relative to compute cost, realistic for lightweight responses (a classification label) but would flip for a workload returning large payloads (embeddings, images), where egress could dominate the total instead.
• instance_count still floors at 1 for a very low target_requests_per_second, so the function does not recommend zero capacity even for near-idle traffic.

Set up the unit economics first. With CPU at 20 ms/token, one CPU instance processes 1000/20 = 50 tokens/sec, unbatched. With GPU at 5 ms/token, one GPU instance processes 1000/5 = 200 tokens/sec, unbatched: a 4x speed advantage. But the GPU instance costs 5x as much per hour. Compare those two ratios directly, because that comparison, not either number alone, is what decides cost per token.

cost per token = (price per hour / 3600 sec) / (tokens per sec per instance)

Using an illustrative CPU rate of $1.00/hr (so GPU is $5.00/hr):

CPU: (1.00/3600) / 50  = $0.00000556 per token
GPU: (5.00/3600) / 200 = $0.00000694 per token   (unbatched)

Unbatched, GPU is actually about 25% more expensive per token here, not cheaper: the 4x speed gain doesn't cover the 5x price premium. That's the first thing worth surfacing, because "GPU is faster so it must be cheaper per unit" is the wrong intuition without checking the ratio.

Batching changes this. GPUs get most of their advantage from parallelism: many sequences can often be processed within roughly the same per-step latency, so throughput scales with batch size much better than cost does. If a batch of size B keeps the GPU's 5 ms/token step time roughly flat, effective throughput becomes 200 * B tokens/sec at the same $5.00/hr:

GPU cost per token at batch B = (5.00/3600) / (200*B)

At B=1 (unbatched) GPU costs more per token than CPU, as shown above. At B=2, GPU cost per token drops to about $0.00000347, already cheaper than CPU. The crossover is right around B ~ 1.25: any batch size above that flips the decision to GPU, and realistic serving batch sizes (often well into the tens) push GPU far ahead of CPU on cost per token.

Meeting the workload target. At 200 tokens/sec demand and unbatched throughput, you'd need 4 CPU instances (4 * $1.00 = $4.00/hr) versus 1 GPU instance (1 * $5.00 = $5.00/hr): CPU wins unbatched, matching the per-token math above. Once batching is in play, a single GPU instance can absorb the same 200 tokens/sec with headroom to spare, at a lower total hourly cost than 4 CPU instances, so the batched comparison flips the fleet-level decision too.

Latency SLO (service level objective) consideration. Batching isn't free: a request may sit waiting for a batch window to fill before it's processed, adding queueing latency on top of the per-token generation time. If the SLO is tight (say a strict per-request p99, 99th percentile), the achievable batch size is capped by how long you can afford to wait for the batch to fill, which caps how far the GPU's cost advantage can go. This is the real trade-off: GPU batching buys cost efficiency at the price of some added tail latency, and the decision depends on how much of that latency budget you actually have.

Utilization. All of the above assumes the chosen instance is kept busy. A GPU sitting idle between bursts still bills the full $5.00/hr regardless of batch size, so if traffic is spiky and low-volume, a fixed GPU can be the worse choice even after accounting for batching, unless it can be autoscaled down or shared across workloads during trough periods. The CPU fleet, being cheaper per unit and easier to scale in small increments, wastes less money sitting idle.

Decision. For steady, high-volume traffic where batching headroom exists and the SLO tolerates modest queueing delay, serve on GPU with batching. For low-volume, bursty, or latency-SLO-constrained traffic where batching can't be used aggressively, CPU (or an unbatched/low-batch GPU) can genuinely be the cheaper and safer choice, the opposite of the usual assumption.

Setting up the monthly basis. All figures below are converted to a 30-day month (2,592,000 seconds) so compute, egress, and storage share one basis; mixing per-second and per-month numbers is exactly where these estimates usually go wrong.

Compute. At 500 RPS steady average and 100 RPS per instance at a 70% utilization target, instances needed = ceil(500/100) = 5.

compute cost = 5 instances * $2.50/hr * 24 hr * 30 days = $9,000/month

Egress (the fee charged for data leaving the cloud provider's network to the internet or another region). Requests/month = 500 RPS * 2,592,000 sec = 1,296,000,000. Each response is 10 KB, so total response volume = 1,296,000,000 * 10 KB / 1,000,000 = 12,960 GB/month. 30% of that (3,888 GB) crosses to the EU; the rest (9,072 GB) stays domestic. Using illustrative egress rates of $0.09/GB domestic and $0.12/GB cross-region to the EU (not any specific vendor's real published rate):

egress cost = (9,072 GB * $0.09) + (3,888 GB * $0.12) = $816.48 + $466.56 = $1,283.04/month

Storage. 200 GB at $0.02/GB-month:

storage cost = 200 GB * $0.02 = $4.00/month

Total baseline.

total = $9,000.00 + $1,283.04 + $4.00 = $10,287.04/month

Compute is about 87% of this bill, egress about 12%, storage under 0.1%. Any reduction plan has to attack compute first; shaving egress or storage alone can't move the total by much.

Plan to cut total cost by roughly 30%

1. Raise sustained throughput per instance from 100 to 125 RPS through batching image-classification requests on the GPU, at the same utilization target. Instances needed drops from 5 to ceil(500/125) = 4, cutting compute to $7,200/month, a $1,800/month saving (about 17.5% of the baseline total).
2. Trim the response payload from 10 KB to 7 KB by returning compact classification output (top-k label and confidence score) instead of a more verbose payload, cutting total response volume proportionally by 30%:

new domestic egress = 9,072 GB * 0.7 * $0.09 = $571.54/month
new EU egress        = 3,888 GB * 0.7 * $0.12 = $326.59/month
saving = $1,283.04 - ($571.54 + $326.59) = $384.91/month

3. Route the EU-bound 30% of traffic over the cloud provider's CDN or private backbone instead of standard public internet egress, cutting that tier's rate from an illustrative $0.12/GB to $0.08/GB on the already-trimmed EU volume:

EU egress at new rate = 2,721.6 GB * $0.08 = $217.73/month
additional saving over lever 2's EU line = $326.59 - $217.73 = $108.86/month

4. Move the now-right-sized 4-instance baseline to a 1-year reserved or committed-use plan. Since this is described as a steady average load rather than a spiky one, it's a good fit for commitment pricing; an illustrative 15% discount off on-demand brings compute to $6,120/month.

new total = $6,120.00 (compute) + $789.26 (egress, after levers 2 and 3) + $4.00 (storage)
          = $6,913.26/month
reduction = ($10,287.04 - $6,913.26) / $10,287.04 = ~32.8%

Why this hits the target. The plan deliberately spends most of its effort on the compute line (87% of the bill) through two independent levers, batching-driven right-sizing and a purchasing-model change, since a 10-15% improvement on a line that's 87% of the total moves the needle far more than an equivalent percentage improvement on egress. The egress and payload changes are worth doing but are secondary by construction, matching where the money actually is.

Approach

The function needs to do two things per candidate instance type: check feasibility (can this hardware answer one request within the latency SLO at all, since more instances never reduce a single request's own compute time) and, for whichever candidates pass that check, compute how many instances are needed to sustain the target request rate without running any single instance past the utilization cap. Then pick the cheapest feasible option.

import math

def select_cheapest_instance(configs, request_rate_rps, latency_slo_ms, utilization_target=0.7):
    """
    Pick the cheapest instance type (and how many of them) that can serve
    request_rate_rps while respecting latency_slo_ms.

    configs: list of dicts, one per candidate instance type:
        name, throughput_rps (requests/sec ONE instance serves at 100% util),
        single_request_latency_ms (compute latency for one request), hourly_cost.
    utilization_target: headroom cap (0.7 = never plan to run an instance
        hotter than 70% busy). Running near 100% utilization causes queueing,
        which blows up tail latency even if raw compute latency is fine.
    """
    feasible = []
    for cfg in configs:
        # More instances give more PARALLEL capacity, but never reduce the
        # latency of any single request. If one instance's own compute time
        # already exceeds the SLO, this candidate is disqualified outright.
        if cfg["single_request_latency_ms"] > latency_slo_ms:
            continue
        usable_rps = cfg["throughput_rps"] * utilization_target
        # Round UP: a fractionally short fleet will queue and miss the SLO
        # under real traffic variance, so we over-provision to whole instances.
        count = max(1, math.ceil(request_rate_rps / usable_rps))
        feasible.append({
            "name": cfg["name"],
            "instance_count": count,
            "hourly_fleet_cost": count * cfg["hourly_cost"],
            "single_request_latency_ms": cfg["single_request_latency_ms"],
        })

    if not feasible:
        raise ValueError("No candidate instance type meets the latency SLO on its own.")

    feasible.sort(key=lambda c: c["hourly_fleet_cost"])
    return feasible[0], feasible


if __name__ == "__main__":
    configs = [
        {"name": "cpu_general",   "throughput_rps": 50,  "single_request_latency_ms": 80, "hourly_cost": 0.10},
        {"name": "gpu_inference", "throughput_rps": 400, "single_request_latency_ms": 20, "hourly_cost": 1.20},
    ]
    winner, all_feasible = select_cheapest_instance(configs, request_rate_rps=2000, latency_slo_ms=100, utilization_target=0.7)

    print("Target: 2000 rps, SLO <= 100 ms, utilization cap 70%")
    print()
    for c in all_feasible:
        print(f"  {c['name']:13s} latency={c['single_request_latency_ms']:5.1f}ms  "
              f"instances={c['instance_count']:3d}  fleet_cost=${c['hourly_fleet_cost']:.2f}/hr")
    print()
    print(f"Cheapest feasible choice: {winner['name']} -> {winner['instance_count']} instances, ${winner['hourly_fleet_cost']:.2f}/hr")

Output:

Target: 2000 rps, SLO <= 100 ms, utilization cap 70%

  cpu_general   latency= 80.0ms  instances= 58  fleet_cost=$5.80/hr
  gpu_inference latency= 20.0ms  instances=  8  fleet_cost=$9.60/hr

Cheapest feasible choice: cpu_general -> 58 instances, $5.80/hr

Key points

• The latency feasibility check happens first and independently of cost, since it is a hard pass or fail: a cheaper option that cannot meet the SLO at all is not actually a candidate.
• utilization_target is a deliberate over-provisioning knob, not a bug: sizing to exactly 100% utilization looks cheaper on paper but is unrealistic under real traffic variance, where any burst above the average would immediately start queueing.
• The result in this example is intentionally counter-intuitive: the GPU has both lower per-request latency and higher raw throughput per instance, but the CPU fleet is still cheaper overall, because CPU's much lower hourly cost outweighs needing far more instances to hit the same request rate.

Complexity

O(n) in the number of candidate instance types; each candidate does a constant amount of arithmetic.

Edge cases

• If request_rate_rps is very small, count still floors at 1 (via max(1, ...)), so the function never recommends zero instances even for near-zero load.
• If every candidate's single_request_latency_ms exceeds the SLO, the function raises rather than silently returning the least-bad option, since recommending a fleet that cannot meet the SLO under any circumstances is worse than surfacing the gap.
• A utilization_target of 1.0 (no headroom) is allowed by the code but should be flagged in review as unrealistic, since it removes all burst tolerance.

Framing

FLOPS (floating point operations per second) measures raw computational throughput, but the number printed on a spec sheet is peak FLOPS, the best case under ideal conditions the vendor could construct. What actually matters for a cost decision is achieved FLOPS on your real model and workload, divided by what that hardware costs per hour, since a chip that is twice as fast on paper but only reaches a quarter of its peak on your workload (due to memory bandwidth limits, small batch sizes, or an immature software stack) can lose to a slower-looking chip that reaches a higher fraction of its own peak.

How I'd calculate FLOPS-per-dollar

1. Run the actual model, at a realistic batch size and precision, on each candidate and measure real throughput (inferences or tokens per second), not the vendor's advertised peak.
2. Convert that measured throughput into achieved FLOPS using the model's known FLOPs-per-inference (from its architecture).
3. Divide achieved FLOPS by the instance's hourly price to get FLOPS-per-dollar, or more practically, divide dollars-per-hour by inferences-per-hour to get a direct dollars-per-inference figure, which is usually the number people actually compare.

What I'd benchmark

• Throughput at saturation (the batch size where adding more work no longer meaningfully raises throughput) and latency at that same operating point, since a batch size chosen only for throughput can blow past a latency SLO (service level objective, the internal latency target the team commits to).
• Whether the candidate supports lower-precision execution (INT8/FP16) for this model, since that changes achieved FLOPS-per-dollar substantially and some accelerators support it more natively than others.
• Cold-start and warm-up time, since a hardware option cheaper per hour once running can still be more expensive overall if it takes minutes to become ready under autoscaling.

Worked example (illustrative figures, not published vendor benchmarks)

For a model needing about 250 GFLOPs (250 billion floating point operations) per inference:

• CPU sustaining roughly 500 GFLOPs/sec effective throughput at an illustrative $0.05/hour gives about 2 inferences/sec, or roughly $0.0000069 per inference.
• An A10-class GPU sustaining roughly 8 TFLOPs/sec effective at an illustrative $1.00/hour gives about 32 inferences/sec, or roughly $0.0000087 per inference.
• An A100-class GPU sustaining roughly 40 TFLOPs/sec effective at an illustrative $4.00/hour gives about 160 inferences/sec, or roughly $0.0000069 per inference.

In this illustrative example, CPU and the A100 land at almost the same cost-per-inference, while the A10 is the most expensive per inference despite being a GPU, because its achieved FLOPS-per-dollar happens to be worse than either alternative here. That is the point: the decision cannot be made from the spec sheet alone, it has to come from measuring achieved throughput on the real workload.

Decision criteria beyond raw FLOPS-per-dollar

• Latency fit: the A100 wins decisively if the workload has a tight single-request latency SLO, since raw cost-per-inference parity with CPU ignores that CPU inference is far slower per request, which may simply disqualify it regardless of cost.
• Software maturity: a purpose-built inference chip may have excellent FLOPS-per-dollar on paper but require a real porting and compilation effort, a specialized SDK or toolchain, before you can even measure it, an engineering cost the raw metric does not capture.
• Capacity and availability: high-demand GPU types are sometimes hard to get in a given region at a given time, which is an effective cost (delay, or paying more for a spot or on-demand mix) that a per-hour price alone does not reflect.
• Headroom for future models: choosing hardware only for today's model can force a second migration later if model size grows past what the chosen hardware's memory can hold.