Running LLMs locally: open models, hardware and when it makes sense

You do not have to send every prompt to a cloud API. Open-weight models (their trained weights are published for download) can run on your own laptop, workstation or server. This article explains the options and the honest trade-offs.

Note: "open-weight" is not always the same as "open source". Each model has its own licence that may limit commercial use or have size conditions, so read it before building a product on it.

Why run locally?

• Privacy: prompts and documents stay on your machine. Useful for sensitive data and strict compliance needs.
• Control: you choose the model and version, and it does not change under you.
• Cost at scale: no per-token fee. For heavy steady use, your own hardware can be cheaper, though you pay for hardware, power and your time.
• Offline use and low network latency.
• Learning and experiments with no usage bill.
• Freedom to fine-tune and customise.

Why not?

• The best proprietary models are usually stronger, especially at complex reasoning and long tasks. The gap varies by task, so test on your own use case.
• You need hardware and the skills to run, monitor, secure and update it.
• Throughput for many concurrent users needs serious GPUs.
• You carry the work of safety, evaluation and updates yourself.

What you need: memory is the key

A model's size is counted in parameters (such as 7B or 70B billion). The weights must fit in GPU memory (VRAM) or system RAM, and generation is fastest on a GPU.

Quantization stores weights in fewer bits (for example 4-bit instead of 16-bit) to shrink memory a lot, with some quality loss that is often acceptable. As a rough guide, a 4-bit model needs about half a gigabyte of memory per billion parameters plus extra for the context, but check the specific model's stated requirements.

Rough, general picture:

• Small models (a few billion parameters): run on a modern laptop, even without a big GPU. Good for simple tasks, summaries and experiments.
• Mid-size (around 7B to 30B): need a decent GPU or a machine with plenty of unified or system memory.
• Large (70B and above): need multiple or high-memory GPUs, or heavy quantization and patience.

Apple silicon machines with unified memory run mid-size models well. CPU-only works but is slow.

Popular tools

• Ollama: easy way to download and run models with one command and a local API.
• llama.cpp: efficient C/C++ runtime, widely used, supports quantized formats (GGUF) on CPU and many GPUs.
• LM Studio and similar apps: desktop interface for local models.
• vLLM, TGI and similar servers: high-throughput serving on GPUs for production workloads.

A typical first try:

ollama pull <model-name>
ollama run <model-name>

Pick a current model from a trusted library and check its size and licence.

Using it in an app

Many local servers expose an OpenAI-compatible HTTP API, so you can often switch an app between a hosted and a local model by changing the base URL and model name. This makes it easy to compare them on your evaluation set.

Security notes

• Download models only from trusted sources. Prefer safe weight formats, since some older formats can run code when loaded.
• Do not expose the model server to the internet without authentication, a firewall and rate limits.
• A local model is still vulnerable to prompt injection, and the same agent and tool rules apply.
• Keep the runtime and drivers updated.

How to decide

1. Write down your needs: quality, privacy, volume, latency, budget.
2. Test one or two open models on your own evaluation set and compare with a hosted model.
3. If quality is enough and privacy or cost matters, run locally or on your own server.
4. If you need the top quality or low operations work, use a hosted API. Many teams use both: local for private or simple tasks, hosted for the hard ones.

Takeaway: local open-weight models give privacy and control, memory (VRAM) and quantization decide what fits, tools like Ollama and llama.cpp make it easy, and you should compare against hosted models on your own tasks before committing.

---

AI & LLM series, Part 2 (quality, agents and cost): 5 of 5. Previous: Reducing LLM cost and latency: caching, routing and shorter prompts.