TokenAssemble
Check a machineStack BuilderWorkflowsCompareMethodology

Browse stack

NVIDIA GPU

NVIDIA GeForce RTX 4090

The NVIDIA GeForce RTX 4090 has 24GB of VRAM and 1010 GB/s of memory bandwidth. Its sweet spot tops out around Qwen3-30B-A3B — the largest model that loads cleanly at a quality quant with llama.cpp at 8K context.

VRAM
24 GB
Memory bandwidth
1010 GB/s
TDP
450 W
MSRP
$1,599
Backends
CUDA · Vulkan

Also good for gaming — 4K-class (estimated from FP32 compute, not a game benchmark)

Specs as of 2026-07-08 · source

Check price on Amazon

NVIDIA GeForce RTX 4090 for local AI — common questions

How much VRAM does the NVIDIA GeForce RTX 4090 have?
The NVIDIA GeForce RTX 4090 has 24GB of VRAM with 1010 GB/s of memory bandwidth. VRAM sets which models fit; bandwidth sets how fast they feel once they do.
Is the NVIDIA GeForce RTX 4090 good for running AI models locally?
Yes. 24GB is enough headroom for the large open models most people want to run locally. The largest model it loads cleanly is Qwen3-30B-A3B, at a quality quant with llama.cpp at 8K context.
What runtimes work with the NVIDIA GeForce RTX 4090?
It runs on CUDA · Vulkan, which covers Ollama, llama.cpp and LM Studio. Runtime choice affects speed and features, not whether a model fits.

Affiliate link — we may earn a commission. Verdicts are computed before any link is attached.

What an NVIDIA GeForce RTX 4090 can run

Every model we track, graded at its best clean-fit quant. Click any verdict for the full breakdown.

ModelBest-quant verdict · llama.cpp · 8K context
Qwen3-8BFits · A · Q8_0interactive
gemma-4-26B-A4B-itFits · B · UD-Q4_K_Minteractivebeta
Qwen2.5-7B-InstructFits · A · Q8_0interactive
Qwen2.5-1.5B-InstructFits · A · Q8_0interactive
Llama-3.2-1B-InstructFits · A · Q8_0interactive
Llama-3.1-8B-InstructFits · A · Q8_0interactive
DeepSeek-R1Won't fitbeta
Qwen3.5-9BFits · A · Q8_0interactivebeta
gpt-oss-20bFits · A · MXFP4interactivebeta
Qwen3.6-35B-A3BTight · C · MXFP4_MOEinteractivebeta
Qwen2.5-3B-InstructFits · A · Q8_0interactive
Qwen3-32BTight · C · Q4_K_Minteractive
Qwen3.6-27BFits · B · Q4_K_Minteractivebeta
Qwen2.5-0.5B-InstructFits · A · Q8_0interactive
Mistral-7B-Instruct-v0.3Fits · A · Q8_0interactive
gpt-oss-120bTight · C · MXFP4interactive · offloadbeta
Qwen3-14BFits · B · Q8_0interactive
Qwen3-30B-A3BFits · B · Q4_K_Minteractivebeta
Qwen2.5-32B-InstructTight · C · Q4_K_Minteractive
DeepSeek-V4-FlashWon't fitbeta
Qwen2.5-14B-InstructFits · B · Q8_0interactive
Llama-3.2-3B-InstructFits · A · Q8_0interactive
gemma-3-4b-itFits · A · Q8_0interactivebeta
gemma-3-12b-itFits · A · Q8_0interactivebeta
Llama-3.1-70B-InstructTight · C · Q4_K_Musable · offloadbeta
Phi-3.5-mini-instructFits · A · Q8_0interactive
Phi-4Fits · B · Q8_0interactive
DeepSeek-R1-Distill-Qwen-32BTight · C · Q4_K_Minteractive
gemma-3-27b-itTight · C · Q4_K_Minteractivebeta
Llama-3.3-70B-InstructTight · C · Q4_K_Musable · offloadbeta
Qwen2.5-72B-InstructTight · C · Q4_K_Mpainful · offloadbeta
gemma-2-2b-itFits · A · Q8_0interactivebeta
DeepSeek-R1-Distill-Qwen-14BFits · B · Q8_0interactive
DeepSeek-R1-Distill-Llama-8BFits · A · Q8_0interactive
DeepSeek-R1-Distill-Qwen-7BFits · A · Q8_0interactive
gemma-2-9b-itFits · A · Q8_0interactivebeta
Mistral-Nemo-Instruct-2407Fits · A · Q8_0interactive
QwQ-32BTight · C · Q4_K_Minteractive
Mistral-Small-24B-Instruct-2501Fits · B · Q5_K_Minteractive
gemma-2-27b-itFits · B · Q4_K_Minteractivebeta

Compare NVIDIA GeForce RTX 4090

Have a specific model in mind? Check it against the NVIDIA GeForce RTX 4090 at your own quant and context length.

Check with the NVIDIA GeForce RTX 4090 preselected →

Verdicts computed from published specs and measured GGUF sizes — how the math works.