H200 vs H100 for LLM inference: which GPU to buy, and where the B200 fits

AI infrastructure & software

If you are about to sign a purchase order for GPUs to run LLMs on your own infrastructure, the trap is this: the H100 and H200 have identical compute specs on paper, so vendors will happily ship you whichever is in stock. Inference throughput is decided mainly by memory capacity and bandwidth, not TFLOPs, and that is exactly where the two cards differ. This article maps model size and context length to a specific card, shows the tokens-per-second arithmetic using published benchmarks, and gives you a cost-per-million-tokens calculation you can copy, including where the Blackwell B200 earns its price premium and when renting beats owning.

Every engineering leader who commits to self-hosted LLM inference eventually sits in the same purchase meeting. The vendor quotes an 8-GPU HGX node, the price difference between H100 and H200 looks like a rounding error next to the total, and the spec sheets are nearly identical. Then six months later the serving stack is GPU-memory-starved at a batch size of 8, and the business case you wrote is off by a factor of two.

This is a hardware decision guide for that meeting. It pairs with two earlier questions we cover separately: whether to self-host at all (buy versus rent from an API), and which serving engine to run (vLLM versus SGLang versus TensorRT-LLM). This one answers a narrower but expensive question: which silicon, and how many cards.

The decision on one page

Three NVIDIA data centre GPUs matter for enterprise LLM inference right now: the H100 SXM (Hopper), the H200 SXM (Hopper with a memory upgrade) and the B200 (Blackwell). The headline specs, from NVIDIA's own pages and the SemiAnalysis InferenceX chip database:

Sources: NVIDIA H200 specifications, InferenceX H100, InferenceX H200, InferenceX B200. The dollar rates come from the SemiAnalysis AI Cloud TCO model: they are amortised cost per chip-hour, not list prices, and are the figures to use when comparing owned capacity with rented capacity.

The short answer, before the working:

• 70B-class models at real concurrency, or any 32k-plus context window: buy H200. The 80 GB on an H100 forces either a tiny batch or multi-GPU tensor parallelism for a 70B FP8 model, and both cost you more than the H200 premium saves.
• 8B to 14B models, short prompts, tight budget, spare H100 capacity already in the building: H100 is fine. On small memory-bound workloads the H200 advantage shrinks to about 13 per cent.
• Greenfield cluster with a 3-year horizon and a modern serving stack: get a B200 quote alongside the H200 one. Per dollar it is the better chip if you can serve FP4, but you pay more per hour and per watt.
• Spiky demand, pilots, or a horizon under a year: rent. Public on-demand prices for H200 are currently $3 to $5 per GPU-hour.

Why inference throughput is a memory problem

Prefill (processing the prompt) is compute-bound. Decode (generating tokens) is memory-bandwidth-bound: every generated token requires reading the model weights and the KV cache out of HBM. A useful approximation for decode:

output tokens/sec ≈ batch_size × memory_bandwidth / (weight_bytes + batch_size × kv_bytes_per_token)

Two things follow immediately. First, batch size amortises the weight read, so throughput scales with concurrency until the KV cache dominates. Second, at a fixed batch size, tokens per second track memory bandwidth almost linearly.

That is why the H100 and H200 behave so differently despite identical FLOPs. Both deliver 1,979 dense FP8 TFLOP/s (H200 spec table). But the H200 has 141 GB of HBM3e at 4.8 TB/s against the H100's 80 GB at 3.35 TB/s: 76 per cent more capacity and 43 per cent more bandwidth. For decode, that is the whole story.

What do measured numbers look like?

• NVIDIA's TensorRT-LLM H200 launch benchmark measured 11,819 output tokens/sec per GPU on Llama2-13B (FP8, batch 1,024, 128-token prompts) and reports up to 1.9x H100 throughput on LLM workloads.
• The same table shows Llama2-70B on one H200 at 3,014 tokens/sec (batch 512, 128 in, 128 out) and 341 tokens/sec in a summarisation shape (64 concurrent requests, 2,048-token prompts, 128-token outputs). For Llama2-70B, NVIDIA's footnote notes the H100 ran batch 8 where the H200 ran batch 32. That batch headroom, not raw bandwidth, is why the gap reaches 1.9x.
• An independent vast.ai comparison on a small model (DeepSeek R1 Distill Llama 8B, vLLM, 16-token prompts, 128-token outputs) found 13,650 tokens/sec on H100 versus 15,435 on H200: only 13 per cent.

Both results are correct. On an 8B model with short prompts, the working set fits comfortably in 80 GB and the workload is not memory-starved, so the extra bandwidth buys little. On a 70B model, or long prompts, the H100 runs out of room for batch and KV cache, and the H200 pulls away. If your vendor quotes the 13 per cent figure to sell you H100s for a 70B deployment, or the 1.9x figure to sell you H200s for a chatbot over an 8B model, they are quoting the wrong benchmark.

One more honest caveat: those TensorRT-LLM numbers were measured on v0.5.0, and the blog itself notes later releases improved materially. Treat them as conservative.

Memory maths: weights, KV cache and a rule of thumb

Before comparing cards, budget the model. Weights are the easy part:

weights in GB ≈ parameter count in billions × bytes per parameter

FP16 or BF16 costs 2 bytes per parameter, FP8 costs 1, INT4 weight-only quantisation costs about 0.5. Add roughly 10 per cent for the engine, activations and fragmentation. That gives a practical minimum card count:

The KV cache is the part people forget. Per token:

kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

For a 70B-class model with grouped-query attention (80 layers, 8 KV heads, head dimension 128), that is 163,840 elements per token: 320 KiB at FP16, 160 KiB at FP8 KV cache. Multiply by sequence length and concurrency:

• 8k context: about 2.5 GiB per concurrent sequence at FP16 KV, 1.25 GiB at FP8.
• 32k context: about 10 GiB per sequence at FP16, 5 GiB at FP8.
• 128k context: about 40 GiB per sequence at FP16, 20 GiB at FP8.

So a single H200 serving a 70B FP8 model (70 GB weights, roughly 70 GB left for KV and overhead) handles around 25 concurrent 8k sequences at FP16 KV, but only 2 to 3 at 128k with an FP8 KV cache, and 1 at 128k with FP16 KV. On a single H100 the same 70B model has only a few GB of headroom for KV at all, which is exactly the "batch 8 versus batch 32" effect NVIDIA's footnote describes. If your product promise is 100k-token context on a 70B model with meaningful concurrency, your minimum viable purchase is two H200s, or KV-cache quantisation and paging in the serving engine.

A quick calculator to reuse when a vendor sends a model card:

def kv_bytes_per_token(num_layers, num_kv_heads, head_dim, bytes_per_element=2):
    """KV cache bytes per generated token (K and V)."""
    return 2 * num_layers * num_kv_heads * head_dim * bytes_per_element

def gpu_budget(param_b, bytes_per_param, ctx_len, concurrency, kv_bpt):
    weights_gb = param_b * bytes_per_param
    kv_gb = kv_bpt * ctx_len * concurrency / 1024**3
    return weights_gb, kv_gb, weights_gb + kv_gb * 1.1  # 10% runtime overhead

# 70B FP8, 32k context, 16 concurrent sequences, FP8 KV
print(gpu_budget(70, 1, 32768, 16, kv_bytes_per_token(80, 8, 128, 1)))
# (70, 80.0, 158.0) -> exceeds one H200 (141 GB); at this concurrency you need two

Worked example: cost per million tokens

Numbers, not vibes. Take a real workload: a 70B instruct model answering questions over internal documents, prompts around 2,048 tokens after retrieval, answers capped at 128 tokens, 64 concurrent requests, FP8 weights on a single H200.

From the TensorRT-LLM H200 table, the closest measured point (ISL 2,048, OSL 128, batch 64) is 341 output tokens/sec per GPU. With the SemiAnalysis TCO rates:

def usd_per_million_output_tokens(output_tok_s, usd_per_gpu_hour, gpus=1):
    tokens_per_hour = output_tok_s * 3600
    return usd_per_gpu_hour * gpus / tokens_per_hour * 1_000_000

print(usd_per_million_output_tokens(341, 1.22))   # $0.99 per million, owned H200
print(usd_per_million_output_tokens(341, 2.90))   # $2.36 per million, retail-tier rental
print(usd_per_million_output_tokens(341, 3.50))   # $2.85 per million, current market on-demand

The same card, the same model, and the arithmetic for three workload shapes:

That 14x spread on one piece of hardware is the single most useful takeaway here. Your prompt shape moves cost per token far more than the choice between H100 and H200 does, so measure your own traffic mix before you model anything.

For the H100 comparison, use NVIDIA's measured 1.9x claim for memory-bound 70B decode: 341 tokens/sec on H200 implies roughly 180 tokens/sec on an H100 at the same shape (an estimate derived from NVIDIA's ratio, not a measurement of your workload). At the $1.17/hr owned rate that is about $1.81 per million output tokens: the H200 delivers roughly 1.8x better cost per token at almost the same hourly rate. On the short-prompt 8B benchmark where the gap is 13 per cent, the same calculation says the H100 is the better buy. Again: match the benchmark to your model.

Energy follows the same logic. An H200 draws about 1.37 kW all-in per chip, so 341 tokens/sec works out at about 1.1 Wh per thousand output tokens, while the long-answer shape (4,750 tokens/sec) is about 0.08 Wh per thousand. In a region where data centre power and cooling are a real line item, tokens per joule and tokens per square metre of rack belong in the business case.

Where the B200 fits

The Blackwell B200 is a different proposition, not a faster H200. Per InferenceX's B200 entry: 180 GB of HBM3e at 8 TB/s, 4,500 dense FP8 TFLOP/s (2.3x the Hopper parts), 9,000 dense FP4 TFLOP/s, NVLink 5.0 at 900 GB/s each way, 1,000 W TDP and about 1.71 kW all-in per chip. The TCO model rates it at $1.73/GPU-hr owned and $3.70/GPU-hr retail: roughly 1.4x the H200's hourly cost for more than double the compute and 1.67x the memory bandwidth.

Three things decide whether that premium pays:

1. FP4 readiness. The B200's headline advantage is NVFP4 serving. Open-weight models increasingly ship FP4 checkpoints with accuracy within noise of FP8, and vLLM, SGLang and TensorRT-LLM all support Blackwell FP4 paths. If your engine stack is pinned to a version that predates that support, you are buying FLOPs you cannot use.
2. Workload shape. Decode-heavy serving with long contexts benefits from 8 TB/s and 180 GB. Compute-heavy prefill and training benefit from the doubled tensor throughput. Batch-classification workloads that already saturate Hopper need neither.
3. Scale. Above eight GPUs, Blackwell's interconnect story matters: NVLink 5.0 doubles per-chip bandwidth, and rack-scale systems such as DGX GB200 wire 72 Blackwell GPUs and 36 Grace CPUs into one NVLink domain with up to 13.4 TB of HBM3e and 576 TB/s aggregate bandwidth. For a 405B-class model with high concurrency that is a genuine architectural step, not a refresh.

A practical rule: if your workload fits comfortably in 141 GB and runs well on H200 at today's rates, the B200 will improve your cost per token mainly where FP4 applies. If you are sizing a new platform for the model sizes you will serve in two to three years, price Blackwell now, because the facility requirements (about 1.7 kW per chip all-in, liquid cooling in the rack-scale SKUs) are not something you retrofit cheaply.

Buy, rent or reserve: the market today

Public on-demand prices, per GPU-hour, from live price aggregators: H100 SXM around $2.00 to $3.50 median across providers, cheapest listings near $1.33 to $1.99; H200 SXM around $3.00 to $5.34 median, cheapest near $2.63 to $2.99; B200 median about $8.50 with spot offers near $4.66 (GPUPerHour, Silicon Liquid, Kovara market). Spot and preemptible capacity runs 30 to 37 per cent below on-demand on H100 and H200.

The ownership decision is then arithmetic. At the SemiAnalysis owned rate of $1.22/GPU-hr for H200 versus $3.50/GPU-hr to rent, at a 60 per cent duty cycle (432 hours a month), renting costs about $1,512 per GPU-month and the owned equivalent is about $527: a gap near $985 per GPU per month. Two corrections, both in the same direction. The owned rate assumes hyperscaler-scale procurement, so your real rate is higher; and owning carries capex, colocation, spares and an operations team, which the rental price includes. Use the gap as a lower bound on what ownership can save, and run it over the period you will actually operate the hardware, typically three to five years.

Rules of thumb we use with clients:

• Rent when duty cycle is below about 40 to 50 per cent, the horizon is under 12 months, the model or quantisation strategy is still moving, or you need capacity this quarter.
• Own when load is steady and predictable, the data must remain in a specific jurisdiction, the same platform will serve several teams, or you are paying a premium for burst capacity you could hold yourself.
• Reserve or commit for the middle case: monthly reserved instances typically discount against on-demand without capex.

For GCC organisations the data residency point is often decisive. Regulated data, government workloads and much enterprise policy in the UAE and Saudi Arabia keep inference inside national borders, and once the cluster runs in a local colocation facility or your own data centre, that question simply disappears. The trade is operational: you now own the platform. That is precisely the layer we design, build and run for clients from our team in the UAE, from GPU servers and Kubernetes up to a single gateway for every model, and it is what products like our FastLLM Proxy are for: routing, budgets and access control across both your own LLM servers and hosted providers in one place.

Buying checklist

1. Fix the workload first. Model, precision, prompt length distribution, output length distribution, concurrency, and time-to-first-token SLA. Without these, every GPU comparison is marketing.
2. Do the memory budget (weights plus KV plus 10 per cent) using the calculator above. If it lands within 20 per cent of a card boundary, buy the larger card: KV cache and batch headroom are what fail in production, not weights.
3. Model cost per million output tokens for each candidate card using your own prompt mix and the snippet in the worked example, at both the owned and the rental rate. Compare against each other, not against peak TFLOPs.
4. Decide ownership with the duty cycle. Below roughly half utilisation, rent; above it and past a year, buy.
5. Size the facility, not just the server. Hopper draws about 1.37 kW per chip all-in, so an 8-GPU HGX node is around 11 kW before top-of-rack switching. Blackwell is about 1.7 kW per chip and the rack-scale systems are liquid-cooled.
6. Choose the form factor deliberately. HGX SXM gives full NVLink domains of 8 GPUs and the best multi-GPU inference. The H200 NVL (PCIe, air-cooled, up to 600 W, 141 GB at 4.8 TB/s, NVIDIA AI Enterprise included) fits standard enterprise racks and up to four GPUs on an NVLink bridge.
7. Pin the serving stack. Benchmark vLLM, SGLang or TensorRT-LLM at your exact model, precision and sequence lengths before purchase. The same hardware moved 341 to 4,750 tokens/sec across workload shapes in NVIDIA's own table; engine version and configuration matter as much.
8. Plan operations from day one. Spares and RMA paths in your region, cluster monitoring, an inference gateway with per-team budgets, and a quantisation policy (FP8 weights, FP8 or paged KV) documented in the runbook.

What to do next

If you remember one line: H100 versus H200 is a memory decision, and for 70B-class models with long context the answer is H200; the B200 wins on cost per token when your stack can serve FP4.

Do this in order: measure your real prompt and output length distribution for two weeks; run the memory budget for your largest intended model at your intended context window; price that configuration in tokens per second and dollars per million tokens at both owned and rented rates; then write the purchase order against the number, not against the TFLOPs.

If you want a second opinion on the sizing, the facility plan or the total cost model, our team in the UAE does this work for a living: we design, build and run the GPU and model platform on your own infrastructure under AI infrastructure, and our FastLLM Proxy gives you one OpenAI-compatible gateway with routing and budgets across your own servers and hosted providers while you decide how much to own.

https://www.azrty.com/blog/h200-vs-h100-for-llm-inference-which-gpu-to-buy-and-where-the-b200-fits

<a href="https://www.azrty.com/blog/h200-vs-h100-for-llm-inference-which-gpu-to-buy-and-where-the-b200-fits">H200 vs H100 for LLM inference: which GPU to buy, and where the B200 fits</a> (Azrty)