How much VRAM do you need to run AI models locally?

The first question anyone asks before running a chatbot on their own PC is "will it fit?". For local AI, video memory (VRAM) matters more than almost any other spec: if the model fits in the GPU's memory it runs fast; if it doesn't, it crawls or refuses to load. Here is how to work out what you need in five steps, with real examples and no guesswork.

Count the parameters

Every model name tells you its size: Llama 3.1 8B has about 8 billion parameters, Qwen 32B about 32 billion. Each parameter is a number that has to sit in memory while the model runs.

Watch out for mixture-of-experts (MoE) models. They only use a fraction of their parameters for each token, which makes them fast, but all of the parameters still have to be loaded. OpenAI's gpt-oss-20b, for example, has about 21 billion parameters in total but only around 3.6 billion active per token. For memory planning, count the total.

Pick a precision (quantisation)

Models are trained in 16-bit precision, but for local use you usually download a quantised version that stores each parameter in fewer bits. The rule of thumb is simple: memory for weights ≈ parameters × bits ÷ 8.

4-bit is the usual sweet spot for home use: quality loss is small for most chat and coding tasks, and memory needs drop by about three-quarters compared with 16-bit. Going below 4 bits saves more space but quality falls faster, especially on smaller models.

Add memory for context (the KV cache)

While it generates text, a model keeps a KV cache, a running memory of the conversation. It grows with every token in the context window, and with long documents it can rival the model itself.

The size depends on the model's architecture: 2 × layers × KV heads × head size × bytes per value, per token. Two worked examples at 16-bit:

• Llama 3.1 8B (32 layers, 8 KV heads, head size 128): about 128 KB per token. That is roughly 1 GB for 8K tokens, 4 GB for 32K and 16 GB for the full 128K window.
• Llama 3.1 70B (80 layers, 8 KV heads, head size 128): about 320 KB per token, so roughly 2.5 GB at 8K and 10 GB at 32K.

This is why a GPU maker can say that an 8B model with a 128K context needs a 32 GB card even though the 4-bit model itself is under 5 GB. If you don't need huge documents, cap the context in your app (Ollama, LM Studio and llama.cpp all let you set it) and you save a lot of memory. Many tools can also store the KV cache at 8-bit to halve it.

Leave headroom

Add 10–20% on top for the runtime, temporary buffers and, on a desktop, whatever the screen and other apps already use. A model that fits "exactly" on paper often fails to load in practice.

If you go over, tools such as llama.cpp can offload some layers to system RAM and the CPU. The model will still run, but every token waits for the slower memory; ASUS, for example, says inference can drop by up to 90% when that happens. Offloading is fine for trying a model, not for daily use.

Match the model to your GPU

Putting it together, with 4-bit models and a moderate context of around 8K tokens:

Some practical notes:

• Two GPUs can share one model in llama.cpp, vLLM and similar tools, so 2 × 16 GB can run what 32 GB runs, a little slower than a single card.
• Unified memory (Apple M-series Macs, AMD Ryzen AI Max) lets the GPU use a large share of system RAM. A 64 GB or 128 GB machine can load 70B models that no single consumer card can hold, though it generates tokens more slowly than a top GPU.
• Software support matters as much as gigabytes. Nvidia's CUDA works with almost everything; AMD and Intel cards now work in llama.cpp, Ollama and others, but check your favourite app first.

If you are buying hardware for this, our laptop for local AI guide covers notebooks and NPUs, and the graphics card buying guide explains power, size and other specs. New to running models yourself? Start with what open-weight AI models are. For a recent example of a card built around memory, see the ASUS UGen Arc Pro B70 with 32 GB.