EXPLAINER

Local LLM Runtimes

Local LLM runtimes such as Ollama, llama.cpp and vLLM load open weights on hardware you control and often expose an HTTP API. They trade operational work and hardware limits for privacy and a predictable machine.

Short answer: Ollama, llama.cpp and vLLM are runtimes that serve open-weight models. Ollama and llama.cpp fit machines you sit next to; vLLM targets multi-user GPU servers. The privacy trade-offs of running locally are covered in What is Local AI? Running Models on Your Own Device.

What it is

A local LLM runtime is the program that loads model weights and runs inference on hardware you control. llama.cpp is a widely used C++ engine that runs quantised models efficiently, including on CPUs and Apple silicon, using formats such as GGUF. Ollama wraps that style of engine in a desktop-friendly installer and a local HTTP API: you pull a model and call localhost. vLLM is a server engine aimed at GPUs and many concurrent requests, with techniques that raise throughput on a shared machine.

These are not the models themselves. Weights still come from somewhere, often Hugging Face, under that model's licence. They are also not automatically private in every configuration: a runtime bound to a public network interface is a remote API you forgot you published.

Why and when it is used

Use Ollama or llama.cpp when a developer or a small team wants an assistant or a prototype on a workstation, or when prompts must not leave the building and the model fits in memory. Use vLLM (or a similar server stack) when you are hosting an open model for many users and GPU utilisation matters. Use a hosted model API when you do not want to operate GPUs and the data policy allows it. The broader decision is Local AI vs Cloud AI: Privacy, Cost, Capability.

Do not expect a 7-billion-parameter local model to match a frontier hosted model on hard reasoning. Do not pick a runtime because a blog said it was fastest without measuring your batch size and your latency budget.

How it works

The runtime maps weight tensors into memory, quantised or not, and generates tokens one after another (or in a batch, on servers). A quantised GGUF file is smaller and slightly less accurate than a full-precision checkpoint. Context length consumes memory too: a long prompt can fail to load even if the model "fits". Ollama's HTTP API imitates a simple chat API so application code looks like a model API call with a different base URL. vLLM serves an OpenAI-compatible endpoint in many deployments, which lets you keep one client and switch base URLs. Neither compatibility shim implements every vendor feature (tools, vision, batch discounts).

GPU memory is the hard constraint. If it does not fit, the process crashes or spills to RAM and becomes slow. Concurrent users multiply the memory needed for the attention cache, not just the weights.

Technologies and dependencies

A machine with enough RAM or GPU memory, a model file you are licensed to run, and the runtime's install method (binary, installer, or container). Python clients are common. NVIDIA GPUs dominate server examples; Apple silicon uses a different path that llama.cpp and Ollama already special-case. Tokenizer files must match the weights. Environment variables configure the listen address and any preload list. Your application should still have a timeout and a health check; a local process can wedge.

How to get started

  1. Check free memory. Pick a small instruction-tuned model whose quantised size is well under that number.
  2. Install Ollama or build llama.cpp from its current instructions. Pull or download one model.
  3. Ask it a question you know the answer to, in a terminal. Then call the local HTTP port from Python or curl. Confirm the port is bound to localhost (127.0.0.1), not 0.0.0.0.
  4. Time a response. If it is too slow, try a smaller quantisation or a smaller model before you rewrite your app.
  5. Point a thin wrapper in your app at that base URL behind a config flag so tests can use a fake.
  6. For a shared server, read vLLM's documentation for the GPU you actually have. Do not expose it to the internet without authentication.

Write down the exact model digest you pulled. "Latest" will change next week and your notes will not match the answers.

Cautions and trade-offs

Local does not mean correct, safe, or legal. The model can be wrong. The licence can forbid your use case. A helpful installer script can enable a tunnel you did not intend. Ollama and similar tools may check for updates or phone home depending on settings; read that if you are in a locked-down network.

Quantisation hurts some tasks more than others (math, names, formatting). Evaluate; do not assume. vLLM on a shared GPU needs operations skills: drivers, CUDA versions, and a restart story. llama.cpp flags are powerful and easy to copy from a gist that does not match your file. Finally, do not bake a huge model into a git repo. Use the runtime's pull or your artifact store, and keep the repo light.