Hardware & Setup · 8 min read · September 26, 2026

How to Run a Language Model Locally: A Beginner's First Pipeline

Install Ollama or LM Studio, load a small instruct model, ask one question you know, then quit and make sure you can delete the file.

Published by the GuideTopics Editorial Team.

One question, on a machine that is not a website

You can download a small language model, run it on your own computer, and ask it something without opening a chat account. The first answer will not beat the best hosted model. It will tell you the pipeline is real: a file on disk, a program that loads the file, a prompt, a reply, and a way to stop. This guide is that pipeline, laid out. If you are about to buy a computer for it, read the hardware guide before you spend. If you already have a laptop with some free disk and memory, start here and let a too-big download fail cheaply.

Nothing you install from a random thread is part of this guide. Use the project’s own site.

What you are installing

A runner is a program that downloads and executes a model. Two approachable ones are Ollama and LM Studio. Ollama is a small command-line tool with a local service. LM Studio is a desktop app with a chat window and a model browser. Either is enough. Installing both on day one is how people get two copies of a multi-gigabyte file and no idea which app is answering.

Underneath some local runners is llama.cpp, a project that runs models efficiently on ordinary hardware, often from GGUF files. You do not have to build llama.cpp to begin. You should know the name so that when a model page says "GGUF," you recognize it as a file format those tools can load, not as a brand of computer.

Pick a small instruct model for the first run. "Instruct" means the file was prepared to follow prompts, which is what you want in a chat. The model card on Hugging Face or in the runner's library will show the file size. Choose one that is obviously smaller than your free memory. The hardware guide shows the arithmetic. If you skip the arithmetic, pick the smallest instruct option the runner offers and learn the commands. You can move up after a success.

Install, then one command or one click

For Ollama, download the installer from ollama.com for your operating system and follow the installer. When it finishes, open a terminal and run a pull-and-chat command for a small model the library actually lists. The exact name changes as models are added and removed, so copy the name from the Ollama library page rather than from this article. The shape is stable: a pull that downloads the file, then a prompt. Wait for the download. The first run is mostly waiting. That is normal.

When you see a place to type, ask a question you know the answer to. "Capital of France" is a fine systems test. You are not evaluating world knowledge. You are checking that tokens come back. Type the exit command the docs name, often a slash command or a quit word, and confirm the process is gone. A runner you cannot stop is not a beginner success.

For LM Studio, download from the vendor's site, open the app, and use its discover or search panel to download a small instruct model. Load it in the chat tab. Ask the same boring question. Quit the app and confirm it is not still using the fan. The chat window feels familiar. The model file is still a download you should be able to find in the app's storage settings, so you can delete it later.

If the load fails with an out-of-memory error, the file was too big. Delete it and choose a smaller one. Do not start changing drivers until a small file has also failed. First failures are usually about size.

Where the files are, and how to delete them

Models are large. Find the folder the runner uses, which the docs will name, and look at it in your file manager. You should see gigabytes, not mystery. When you are done with a model, remove it with the runner's own remove command or button so the index and the file stay in agreement. Dragging a file to the trash without telling the runner can leave a broken entry. Either way, know how to get the disk space back before you download a second model "to compare."

Keep the installer. Updates are how you get fixes. They are also how a tool might change a default. Read the release note if the update mentions cloud features. A local runner that adds an optional hosted model is still local only when you do not select the hosted model.

A prompt that admits the limit

A small local model will ramble, miss a constraint, or invent a detail. Use the same brief you would use in a hosted chat, only shorter. The prompt guide is the long version. For this test, paste a paragraph you wrote and ask for a shorter version that adds no facts. Compare. If it added facts, you have learned the limit of this file, which is the lesson. A bigger file may do better and may not fit. That is the hardware conversation, not a reason to trust the small one with a client letter.

Do not paste secrets into a local app until you have checked two things. The app is not also calling a cloud model. The machine itself is locked, encrypted, and yours. Local inference avoids the model vendor seeing the prompt when the request never leaves the process. It does not hide the prompt from malware, from another user on the computer, or from a sync folder that uploads your documents. The glossary can help when a setting is named in jargon. The check is still practical: disconnect the network and see whether the reply still arrives. If it does, that request did not need the network. If the app refuses to run offline, believe it.

Point an editor at the local service only if you mean to

Ollama exposes a local HTTP service, by default on your machine, and some tools can be told to use an OpenAI-compatible base URL pointed at that service. LM Studio can do something similar when its local server is turned on. This is how an editor extension chats with your model instead of a hosted one.

Do this after the terminal or the desktop chat already works. Read the editor extension's docs for the base URL and for how to disable its cloud provider. The Cursor and Copilot comparison explains the cloud editors you might be leaving. A misconfigured extension will quietly keep using the cloud while you think you are local, which is the opposite of the privacy goal. The offline test is the test. Unplug, ask for a completion, see what happens.

Start with explanations and small edits. A small model given a whole repository will waste your afternoon. Paste one function.

A simple daily use, if the test felt good

Use the local model for the jobs that match its size. Rewriting a paragraph you provide. Explaining a code error you paste. Turning bullet notes into a first structure you will heavily edit. Keep hosted tools for the jobs where you have already decided the file may leave the building, and where the larger model earns the paste.

Write down the model name and the runner in a note next to the project. "Summaries of my own meeting bullets: local model name, Ollama, offline." Next month you will not remember which file was loaded. The note is the whole operations manual a beginner needs.

When an answer is wrong, correct the source, not the mythology. Small models do not become reliable because you phrased the prompt as a stern system message. They become slightly more useful when the source is in the prompt and the task is small. Take the win. Take the rest of the work back.

When to stop and when to go further

Stop for the day once you have downloaded, asked a known question, quit the program, and deleted or kept the file on purpose. That is a complete beginner run.

Go further when a real task is failing in a way hardware would fix, or when a policy says the cloud is forbidden for a class of files. Then return to the hardware guide with a file size in hand, and look through devices only after that size is written down. More apps, more model families, and more interfaces such as web UIs in front of the same runner are optional. They do not replace the pipeline you just proved: a file, a program, a prompt, a reply, and a way to stop.

If the small model felt like a toy, it is allowed to be a toy. The pipeline is what you were installing. The judgment about whether your work belongs on it comes after, on a document you care about, with the network unplugged, and with you still responsible for the sentence that leaves the building.

#Ollama #LM Studio #local LLM #llama.cpp

Recommended on Amazon

The Coming Wave

The Coming Wave
Mustafa Suleyman. A clear look at how frontier AI changes work, power, and everyday tools.

AI Superpowers

AI Superpowers
Kai-Fu Lee. How the AI landscape is shifting, and how to position your work inside it.

All-in On AI

All-in On AI
Thomas H. Davenport & Nitin Mittal. How companies put generative AI into real operations, not just demos.

2k to 10k

2k to 10k
Rachel Aaron. A short, practical book on writing faster while keeping the work specific.

As an Amazon Associate I earn from qualifying purchases.