Local LLM runtime
vLLM
vLLM is a Apache-2.0-licensed local-LLM server-capable runtime for Linux, running on CUDA · ROCm, with an OpenAI-compatible API. Ease of use 2/5 in our sourced ratings.
- License
- Apache-2.0
- Operating systems
- Linux
- Backends
- CUDA · ROCm
- Install
- pip / source
- Ease of use
- 2/5
- Latest version
- v0.24.0
- GitHub stars
- 85,770
- Engine
- vllm
- Graphical app (GUI)
- Server mode
- OpenAI-compatible API
- CPU offload
- Multi-GPU
- Speculative decoding
- LoRA support
- KV-cache quantization
As of 2026-07-08 · source · GitHub
Quant-format support
What the feasibility gate checks before any VRAM math — per format and backend.
| Format | Backend | Supported |
|---|---|---|
| awq | CUDA | yes (compute capability ≥ 7.5) |
| awq | Metal | no |
| gptq | CUDA | yes (compute capability ≥ 7.5) |
| mlx | Metal | no |
Software that runs on vLLM
Frontends, assistants and harnesses that can use vLLM as their local backend.
Will your machine run a model with vLLM?
Check with vLLM preselected →Support facts come from each runtime's own documentation, stamped and dated — how the math works.