Your own language model on a server: how much memory and GPU you need
An 8-billion-parameter model compressed to 4 bits takes about 4 GB and fits any server card; at a slow pace it even runs on a CPU. A 70-billion model with the same compression needs 35 GB for the weights plus memory for the context, so a 48 GB card. Below: how to calculate memory for your model, which card for which size, and what it costs.
The memory formula
The video memory a language model needs has three parts.
memory = weights + context cache + overhead
Weights are the number of parameters times the size of one parameter. In 16-bit format (FP16 or BF16) a parameter takes 2 bytes, in 8-bit 1 byte, in 4-bit half a byte. NVIDIA gives an example: a 7-billion-parameter model in 16 bits takes about 14 GB. For the Llama 3.1 models Hugging Face publishes a ready table:
| Model | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8 billion parameters | 16 GB | 8 GB | 4 GB |
| 70 billion | 140 GB | 70 GB | 35 GB |
| 405 billion | 810 GB | 405 GB | 203 GB |
Compression to 8 and 4 bits (quantization) lowers answer quality a little but cuts memory several times. Test it on your own tasks: run the same questions through the compressed and the full version.
The context cache
While the model reads the request and writes the answer, it keeps intermediate data for every word (more precisely, every token): the key-value cache. It grows with the context length and with the number of requests the server handles at once. By NVIDIA's calculation a 7-billion model needs about 2 GB of cache for 4,096 tokens. For Llama 3.1 at 128 thousand tokens the cache takes 15.6 GB for the 8-billion model and 39 GB for the 70-billion one — more than the 4-bit weights themselves.
The practical conclusion: a long context and many concurrent users need as much memory as the model itself, or more. Ollama uses a 4,096-token context unless told otherwise, and the cache can be compressed to 8 bits with OLLAMA_KV_CACHE_TYPE=q8_0, which halves its memory. The vLLM server reserves almost all video memory for weights and cache up front; the share is set by the --gpu-memory-utilization parameter.
Overhead is the CUDA libraries and working buffers. Hugging Face notes separately that its table does not include it. Do not pick a card with no room to spare: leave a few gigabytes free.
Which card for which model
Prices are the market median per month in the accsly catalog on 13 November 2026, plans with one whole card unless stated otherwise. The catalog stores roubles, so the dollar figures are converted and rounded.
- 8 billion parameters in 4 or 8 bits (4–8 GB of weights). Any 16–24 GB card will do, with room left for a long context. RTX 3090 $421, L4 $532, RTX 4090 $559 a month.
- 8 billion in 16 bits (16 GB of weights). You need a 24 GB card; it will no longer fit into a 16 GB T4.
- 70 billion in 4 bits (35 GB). A 48 GB card: L40S, $929. The context will have to be limited to a few tens of thousands of tokens.
- 70 billion in 8 bits (70 GB). An 80–96 GB card: A100 with 80 GB about $1,180 (6 plans), RTX PRO 6000 $1,499, H100 $2,400. The 96 GB RTX PRO 6000 leaves more room for the cache.
- 70 billion in 16 bits (140 GB). Two cards of 80 GB or more. The catalog has 29 plans from 13 hosts with 160 GB of video memory or more; a quarter of them cost less than $2,725 a month.
- 405 billion in 4 bits (203 GB). Four cards of 80 GB or more. The catalog has six such plans at six hosts, from $3,870 a month.
If a model is split across several cards, the server software has to support it: in vLLM the number of cards is set by --tensor-parallel-size. Cards linked by NVLink (A100, H100) exchange data faster than over PCIe. More about the cards in the article which GPU for which job.
How fast it will answer
While generating an answer, every new word requires reading all the model's weights from memory, so speed is limited by memory bandwidth, not by computation. Hence a simple estimate of the ceiling for a single request: bandwidth divided by the size of the weights. Real speed is lower, but the order of magnitude is right.
- 8 billion in 4 bits (about 4 GB) on an L4 with its 300 GB/s: no more than 75 tokens per second; on an RTX 4090 with 1,008 GB/s up to 250.
- 70 billion in 4 bits (35 GB) on an L40S with 864 GB/s: up to 25 tokens per second; on an H100 with 3,350 GB/s up to 95.
For a chat with a single user even an L4 is enough. If a team uses the model, or it processes documents in batches, fast memory pays off.
When a CPU is enough
Ollama and llama.cpp run a model without a graphics card, in the server's ordinary memory. A 7–8-billion model in 4 bits takes about 4–5 GB and runs on a VPS with 16 GB of RAM — slowly, but enough for a bot, sorting mail or overnight processing. A VPS with 16 GB of RAM has a market median of $76 a month, five times less than the most affordable card. If there is a GPU but its memory is too small, Ollama splits the model between the card and the processor; you can check this with ollama ps, whose Processor column shows what percentage of the model is on the GPU. Selection: VPS for neural networks on CPU.
How much memory fine-tuning needs
Training needs many times more memory than inference. Full training in mixed precision with the Adam optimizer takes 6 bytes of weights, 8 bytes of optimizer state and 4 bytes of gradients per parameter, plus intermediate data. That is why usually not the whole model is fine-tuned but small added layers — the LoRA and QLoRA methods. Hugging Face's estimate for Llama 3.1:
- 8 billion parameters: full fine-tuning 60 GB, LoRA 16 GB, QLoRA 6 GB;
- 70 billion: full 500 GB, LoRA 160 GB, QLoRA 48 GB.
This matches the QLoRA authors' paper: a 65-billion-parameter model is fine-tuned on a single 48 GB card. So one L40S handles fine-tuning a 70-billion model, and an RTX 4090 is enough for an 8-billion one. Training is a job of hours and days, and it pays to do it on hourly rental; how to calculate the cost is in the article hourly GPU rental.
RAM and disk
The model's weights first sit on disk and are read through RAM at start-up. A 70-billion model in 16 bits is 140 GB of files, and you often need several versions to compare. Take NVMe with room to spare. One-card plans usually have plenty of RAM: the median is 64 GB with an L4 or T4, 128 GB with an RTX 4090 and 240 GB with an H100. If the load is constant and the model large, look at dedicated servers with GPU: the whole machine is yours and the disk is shared with no one.