Running a large language model (LLM) on your own computer used to require a PhD and a server rack. In 2025, it takes about 10 minutes and a modern laptop. Here's everything you need to know.
What Does "Running an LLM Locally" Mean?
When you use ChatGPT or Claude, your messages travel to a remote server, get processed by a massive model, and the response comes back over the internet. Running locally means the model lives on your machine — your data never leaves, there's no subscription fee, and it works offline.
What You Need
The minimum requirements are modest: any computer with 8GB of RAM can run a 7B parameter model. For better quality, 16GB RAM handles 13B models, and 32GB+ opens the door to 30B models that rival GPT-3.5 quality.
Step 1: Install Ollama
Ollama is the easiest way to run LLMs locally. It handles model downloads, quantisation, and inference — all through a simple command-line interface.
Download Ollama from ollama.com (free, open-source). It works on macOS, Windows, and Linux. After installation, open your terminal and run:
Run this command: ollama run llama3.2
Ollama will download the 3B model (about 2GB) and start a chat session. That's it — you're running a local LLM.
Step 2: Choose Your Model
The right model depends on your hardware. As a starting point: Llama 3.2 3B for 8GB RAM, Llama 3.1 8B for 16GB RAM, Mistral 7B for coding tasks, and Llama 3.1 70B for 64GB+ RAM systems.
Step 3: Use a GUI (Optional)
If you prefer a graphical interface over the command line, LM Studio is the best option. It provides a ChatGPT-like interface for any Ollama-compatible model, with model download management and conversation history.
Step 4: Connect to Your Workflow
Ollama exposes a local API at localhost:11434 that's compatible with the OpenAI API format. This means you can use it with Cursor, Continue (VS Code extension), Open WebUI, and dozens of other tools that support custom API endpoints.
Privacy and Offline Use
The biggest advantage of local AI is privacy. Your prompts, documents, and conversations never leave your machine. This makes local LLMs ideal for processing sensitive business documents, personal health information, or confidential code.
FAQ
How fast are local models compared to ChatGPT?
On good hardware (Mac M4 Pro or RTX 4090), local models generate 10–20 tokens per second — fast enough for real-time conversation. On modest hardware, expect 2–5 tokens per second, which is still usable.
Which model should I start with?
Llama 3.2 3B is the best starting point — it's fast, capable, and runs on almost any hardware. Once you're comfortable, try Mistral 7B for better quality.
Can I use local models for image generation?
Yes — Stable Diffusion runs locally and generates images without any cloud dependency. ComfyUI and Automatic1111 are the most popular interfaces. You'll need a GPU with 6GB+ VRAM for good performance.
