Case Studies

Why the Cheapest H100 Rental May Cost More

Public H100 rental rates span $2.69 to $12.29 per hour, but the sticker price tells only part of the story. A practical comparison must account for I/O delays, queueing, preemptions, billing granularity, and workload requirements.

Font size

The same NVIDIA H100 can rent for anywhere from $2.69 to $12.29 per hour. But the final bill depends on more than the hourly rate: I/O, queueing, preemptions, and the shape of the workload all matter.

GPU rental pages usually lead with the price per hour.

The more useful question is: What will it cost to complete one experiment? How much will it cost to process a batch of requests? And how much will a failed job add when it has to run again?

According to the 109-gpu-pricing directory from datareceipts/data-receipts, public hourly rates for the same NVIDIA H100 range from $2.69 to $12.29, a difference of roughly 4.57 times.

The figures show a wide gap in advertised prices. They do not, by themselves, prove that the cheaper option will cost less in practice.

Why the Same H100 Costs 4.57 Times More

Start with the basic calculation:

  • Lowest rate: $2.69/hour
  • Highest rate: $12.29/hour
  • Multiplier: 12.29 / 2.69 ≈ 4.57

This comparison covers the same H100 configuration, not a mix of H100, H200, or other models.

Price is not determined by GPU SKU alone. Location, machine configuration, provider strategy, and transaction timing can all affect the final quote. Identical GPUs do not necessarily mean identical computing environments.

Another overlooked issue is that a listing's “per card per hour” price usually covers GPU rental only.

If two machines bill by the second and by the hour, respectively, the bill for the same 20-minute job can differ sharply. With preemptible instances, teams must also account for interrupted jobs, rewritten checkpoints, and retransmitted data.

Therefore, 4.57 is not the real cost difference across every scenario. It first demonstrates something more fundamental: an advertised GPU rate is not a stable, standardized product price.

Do Not Evaluate the GPU Rate Alone

When comparing options, I would break the total cost into these components:

total cost = GPU usage time
          + disk and network fees
          + queueing and job startup overhead
          + rerun costs from preemptions or failures
          + data preparation and transfer costs

1. Disk and I/O

Training datasets are usually much larger than model checkpoints. Object storage traffic, temporary disk capacity, read and write speed, and initialization time can all leave GPUs waiting for data.

Suppose a cheaper instance adds 30 minutes of data preparation to every experiment. At $2.69 per hour, that adds:

2.69 × 0.5 = 1.345 dollars

For a job that runs for two hours, data preparation has already erased part of the lower hourly rate's advantage.

2. CPU, Memory, and Network

Tokenizing data, cleaning datasets, and preparing data parallelism before training all require more than GPUs. CPU core count, memory bandwidth, and inter-node connectivity determine whether a multi-GPU job can keep the H100s fully utilized.

Inference has similar dependencies. Model loading, time to first token, request queueing, dynamic batching, and KV Cache consumption all affect how much work an instance can actually handle.

A machine with eight advertised GPUs does not necessarily deliver eight times the data throughput.

3. Preemption Policies

Spot instances work well for fault-tolerant, retryable batch workloads, such as generating embeddings in bulk, running offline evaluations, and sweeping model parameters.

Do not keep the only checkpoint on temporary disk and wait until the instance is reclaimed before handling it.

A safer approach is to save checkpoints regularly and build recovery logic into the job:

from pathlib import Path
import json

checkpoint_dir = Path("checkpoints")
checkpoint_dir.mkdir(parents=True, exist_ok=True)

payload = {
    "step": 12000,
    "loss": 0.184,
    "model_state": model.state_dict(),
}

checkpoint_path = checkpoint_dir / "step-12000.pt"
torch.save(payload, checkpoint_path)

Path("checkpoints/latest.json").write_text(
    json.dumps({"path": str(checkpoint_path)}),
    encoding="utf-8",
)

This code demonstrates only the basic checkpoint format. Production systems also need atomic writes, object storage uploads, and state validation. A file's existence does not guarantee that recovery will succeed.

4. Billing Granularity and Minimum Commitments

Before choosing a provider, ask four questions:

  • Is billing by the second, minute, or hour?
  • Does a partial hour count as a full hour?
  • Is there a minimum spend or commitment?
  • Are disks, traffic, and public IP addresses charged separately?

If an experiment takes only 12 minutes, the minimum charge may still be one hour. A rate that is five times lower does not make short jobs five times cheaper.

Training and Inference Need Different Measures

Both training and inference can use H100s, but they optimize for different outcomes.

Training: Measure Stable Throughput and Failure Loss

For training, the cost of one run can be estimated as follows:

cost per run = hourly GPU rate
             × number of GPUs
             × actual runtime hours
             × effective runtime ratio

The effective runtime ratio should exclude initialization, data waits, checkpoint saves, and reruns after failures.

Sustained throughput after several hours matters more than peak FLOPS. If a cheap instance is frequently preempted, the team loses not only the current step but also the time spent restarting data loading and communication buffers.

A simple threshold can help determine whether switching providers is worthwhile:

cheap_rate = 2.69
expensive_rate = 12.29

cheap_overhead_hours = 0.5
expensive_overhead_hours = 0.05
cheap_runtime_hours = 2.0
expensive_runtime_hours = 2.0

cheap_cost = cheap_rate * (cheap_runtime_hours + cheap_overhead_hours)
expensive_cost = expensive_rate * (
    expensive_runtime_hours + expensive_overhead_hours
)

assert expensive_cost < cheap_cost

In this hypothetical calculation, the $12.29 instance may actually cost less because it incurs less overhead. These numbers demonstrate the calculation method; they do not represent a judgment on any provider's pricing.

Inference: Measure Latency and Concurrency

For inference, it is more useful to calculate the cost “per million tokens” or “per thousand requests”:

cost per million tokens
= (total instance cost + additional resource costs) / effective tokens processed

Three variables are especially easy to overlook:

  1. Whether batch size remains stable;
  2. Whether time to first token affects how long users must wait;
  3. Whether the GPU remains idle for long periods because traffic is insufficient.

Online inference must optimize for more than minimum latency; it also needs to meet p95, p99, and concurrency targets. Training, by contrast, is more concerned with whether a job can run to completion without interruption.

The same low-cost GPU provider may be suitable for offline batch inference but unsuitable for an online service.

How to Use a Public Price List

The 109-gpu-pricing directory is useful for an initial screening process, but it should not be treated as a purchasing contract.

I usually follow five steps:

  1. Normalize the units: Convert everything to dollars per GPU per hour and record whether disk, network, and public IP addresses are included.
  2. Normalize the specifications: Confirm the GPU model, memory, CPU, RAM, node count, and network conditions.
  3. Add workload parameters: Enter the expected runtime, startup time, fault-tolerance requirements, and latency targets.
  4. Calculate the effective cost: Include queueing, preemption, data transfer, and reruns after failures.
  5. Run a small-scale load test: Execute a representative workload before committing to a long-term resource.

If the public data is available as CSV or JSON, the fields can first be normalized. The following Python script reads a CSV file and sorts the entries by GPU hour:

import csv
from pathlib import Path

path = Path("gpu_pricing.csv")
rows = []

with path.open(newline="", encoding="utf-8") as file:
    for row in csv.DictReader(file):
        if row.get("gpu") != "H100":
            continue

        rate = float(row["price_per_hour"])
        rows.append({
            "provider": row["provider"],
            "region": row.get("region", ""),
            "price_per_hour": rate,
        })

for item in sorted(rows, key=lambda item: item["price_per_hour"]):
    print(
        f'{item["provider"]:20} '
        f'{item["region"]:12} '
        f'${item["price_per_hour"]:.2f}/h'
    )

The script expects the CSV to contain the fields gpu, price_per_hour, and provider. If the repository uses different field names, map them first. Do not hard-code prices merely to make the script run.

The final comparison should retain three columns: advertised rate, estimated effective cost, and risk-adjusted cost. Only the last column should drive the decision.

Editor's Perspective

The significance here goes beyond the 4.57-times gap in H100 prices. GPU rental is emerging as a tiered market. The hourly rate appears standardized on the listing, but what the customer actually receives is compute capacity that can be used continuously. Training customers buy stability, inference customers buy latency and throughput, and batch workloads buy interruptible compute that can recover.

Going forward, quotes are likely to include queueing time, preemption rates, instance specifications, and data transfer fees. A platform that advertises only the lowest hourly rate without explaining the cost of completed work is likely to lose some larger customers. Providers may also introduce pricing based on completion time, effective tokens, or job success rates.

This differs from simply comparing cloud provider instance catalogs. A public pricing snapshot makes price differences easier to identify, but it cannot account for live inventory, every region, or all additional fees. It is useful for finding candidates, not for replacing load tests or contract terms. Whether an option is genuinely cheaper can only be established by testing it with the intended workload.

Source: https://github.com/datareceipts/data-receipts/tree/main/109-gpu-pricing