Local LLM runtime
ExLlamaV2
ExLlamaV2 is a MIT-licensed local-LLM runtime for Linux · Windows, running on CUDA · ROCm. Ease of use 3/5 in our sourced ratings.
- License
- MIT
- Operating systems
- Linux · Windows
- Backends
- CUDA · ROCm
- Install
- pip / source
- Ease of use
- 3/5
- Latest version
- v0.3.2
- GitHub stars
- 4,581
- Engine
- exllamav2
- Graphical app (GUI)
- Server mode
- OpenAI-compatible API
- CPU offload
- Multi-GPU
- Speculative decoding
- LoRA support
- KV-cache quantization
As of 2026-07-08 · source · GitHub
Quant-format support
What the feasibility gate checks before any VRAM math — per format and backend.
| Format | Backend | Supported |
|---|---|---|
| exl2 | CUDA | yes |
| exl2 | Metal | no |
| gptq | CUDA | yes |
| mlx | Metal | no |
Will your machine run a model with ExLlamaV2?
Check with ExLlamaV2 preselected →Support facts come from each runtime's own documentation, stamped and dated — how the math works.