What is it?
Most popular AI assistants run in a provider's data centre: your text travels over the internet, a large model processes it there, and the answer comes back. Local AI keeps the model on a machine you control. The model files are downloaded once; after that, inference (running the model) can happen offline.
Local models are usually open-weights models — models whose trained parameters are published so anyone can download and run them, subject to each model's licence terms (read the licence; "open weights" is not automatically "unrestricted use"). Local setups range from desktop apps and command-line runtimes (llama.cpp and Ollama are well-known examples) to browser-based runtimes using WebGPU/WebAssembly, and to your own servers.
Why does it matter?
- Privacy and data control: content need not be sent to a third party, which can matter for confidential, regulated or personal data.
- Offline use and independence from a vendor's availability or policy changes.
- Cost structure: no per-token fees, though you pay for hardware and electricity.
- Customisation: you can fine-tune or swap models (What is Fine-Tuning an AI Model?).
- Learning: it demystifies how models behave.
How does it work?
Running a model is mainly a memory and compute problem. The model's parameters must fit in memory (GPU memory, or system RAM at lower speed) while generating each token.
- Model size: more parameters generally means more capability and more memory. Local models are commonly in the "small to mid" range compared with the largest hosted models.
- Quantisation: storing parameters at lower precision (for example 4-bit instead of 16-bit) shrinks memory needs substantially, with some quality loss that depends on the model and method.
- Hardware: a recent GPU with enough memory, or Apple-silicon-style unified memory, makes interactive speeds practical; CPU-only is possible but slower.
- Runtime: software such as llama.cpp-based tools or Ollama loads the model and serves it through a local interface, often an API on
localhostthat your own apps can call. - Context and speed depend on the model, quantisation and hardware; check the model card and test on your machine instead of trusting generic numbers.
Example
A developer wants to summarise confidential meeting notes. They install a local runtime, download a small instruction-tuned open-weights model, and run:
ollama run <model-name>(Model names and commands change; follow the runtime's current documentation.) They paste the notes, and the processing happens on their laptop. They still check the summary for errors — local does not mean more accurate — and confirm the runtime is not configured to send telemetry or to expose its port to the network.
When should I use it?
Local AI fits when data sensitivity, offline operation or predictable cost outweighs the need for the very strongest models, or when you want to experiment and customise. Cloud models fit when you need top capability, large context, minimal setup, or scale. Many teams use both. The detailed trade-offs are in Local AI vs Cloud AI: Privacy, Cost, Capability.
Common mistakes
- Assuming local equals secure. Machine security, model downloads from untrusted sources, and exposed local ports are still risks (AI Privacy and Security: What to Know Before You Paste Data).
- Underestimating hardware needs.
- Expecting parity with the largest hosted models on hard tasks.
- Ignoring licences.
- Forgetting that local models hallucinate too (AI Hallucinations: Why Models Make Things Up).