All products 
Renting an H100 server: SXM or PCIe, what fits in 80 GB and how not to overpay
NVIDIA H100 is the Hopper-generation accelerator for training and serving large models. People rent an H100 server when an A100 is no longer fast enough and the model or dataset needs 80 GB of fast memory per card. The table above lists cloud plans and dedicated servers with one, two or four cards; monthly prices in the catalogue run from roughly $1,350 to $5,400, with the middle around $3,000. Most plans can be billed by the hour.
SXM, PCIe and NVL: three different H100s under one name
Quite different cards are sold as H100, and plans often just say "H100 80GB".
- H100 SXM5, 80 GB HBM3. About 3.35 TB/s of memory bandwidth and up to 700 W. It sits on HGX boards where the cards are linked by NVLink at up to 900 GB/s. This is the fastest option for multi-GPU training.
- H100 PCIe, 80 GB HBM2e. About 2 TB/s and 350 W, a regular card in a PCIe slot. A pair can be joined with an NVLink bridge, but that is optional, not the rule. Slower than SXM, but cheaper and more common in the cloud.
- H100 NVL, 94 GB HBM3. A PCIe card with extra memory, designed to work in pairs over NVLink bridges. The additional 14 GB per card helps with long context in inference.
How to tell which one a plan has: look for SXM, PCIe or NVL in the name; a 700 W power figure points to SXM, 350–400 W to the PCIe version. For multi-card setups ask whether NVLink is present and between which cards: four PCIe cards without bridges exchange data only over the bus, which noticeably slows training with a split model. Once the server is up, check the card with nvidia-smi (name and power limit) and the links between cards with nvidia-smi topo -m.
What fits in 80 GB
Rule of thumb: in FP16 a model takes about 2 GB per billion parameters, in 8-bit about 1 GB, in 4-bit 0.5–0.6 GB, plus headroom for the KV cache, which grows with context length and the number of concurrent requests.
- 70B models in FP8 fit on a single 80 GB card with moderate context. For long context and many users an H100 NVL or two cards is more comfortable.
- The same 70B in FP16 takes about 140 GB and needs two cards with the model split between them (tensor parallel).
- In 4-bit a single card leaves plenty of room for context and parallel requests on models up to 70B.
- Full fine-tuning of 7–13B models fits on one or two cards; gradients and optimizer state take several times more memory than the weights.
- LoRA and QLoRA on 70B run on several cards, QLoRA with a small batch sometimes on one.
The main step up from the previous generation is the Transformer Engine with FP8: with frameworks that support it, transformer inference and training run noticeably faster than in FP16, and the model takes half the memory. For image generation and rendering the H100 is overkill: those jobs are cheaper on an L40S or RTX-class cards.
MIG, drivers and CUDA
The H100 supports MIG: the card can be split into up to seven isolated instances, each with its own share of memory and compute. Clouds sell fractional GPUs this way. That is economical for serving a small model, but training and large models need a full card, so check the VRAM figure in the plan, not just the name.
The H100 has compute capability 9.0. It needs CUDA 11.8 or newer, 12.x recommended, and a matching NVIDIA datacenter driver. In the image, check the driver, CUDA version and PyTorch build: old images made for A100 or V100 will start but without FP8 and with lost speed.
Hourly or monthly
The card is expensive, so the billing model affects the total more than with smaller GPUs. Hourly plans suit experiments, hyperparameter searches and one-off runs: start the server, train, save the checkpoint to external storage, delete the server. More on that in the hourly billing section. If you need an H100 for a service that answers users around the clock, or for training that lasts weeks, a monthly plan or a dedicated GPU server usually works out cheaper.
Training across several servers needs InfiniBand or another fast network between nodes. The catalogue does not always state it, so ask before paying. Also check the data centre location: for personal data of EU users, GDPR terms matter, and for interactive services latency to your users in Europe or the US does too.
If you do not need FP8 and the budget is tight, compare with the A100: it is cheaper and also has 80 GB, but it is slower and lacks FP8. The newer H200 offers 141 GB and makes sense when a model does not fit in 80–94 GB.