Hardware & Setup · 8 min read · September 26, 2026

Hardware for Running AI Locally: Size the Memory to the File

Turn a model file's size into a RAM or VRAM budget, try the computer you have, and buy only for the wall you actually hit.

Published by the GuideTopics Editorial Team.

Start from the file, not from a brand

You want a model to answer you on a machine you control. The first spec that matters is the size of the model file, not the logo on the laptop lid. A chat demo on a vendor's cloud tells you nothing about whether your desk can hold the weights. This guide is how to turn a file size into a memory budget, and how to decide between a computer you already own, a used workstation, and a new machine. It is not a set of tested picks, and it does not quote speeds. Speed depends on the model, the quantization, the software, and the batch of work. Measure it yourself after the file fits.

The beginner guide to running a local model starts at install. Read this page first if you are about to spend money, and that page first if you already have a computer and want to know whether it is enough. The devices section of this site collects hardware in one place. Treat it as a catalog to compare specs you have already decided you need.

Weights, bits, and a back-of-envelope size

A model's parameter count is a headcount of numbers it stores. Those numbers are the weights. If each weight is stored in 16-bit floating point, that is two bytes per parameter. A 7-billion-parameter model at 16-bit is on the order of 14 gigabytes of weights, before the software's own overhead. Four-bit quantization stores each weight in half a byte, so the same 7-billion-parameter model lands nearer 3.5 to 4 gigabytes, plus overhead. The arithmetic is the same at other sizes: multiply parameter count by bytes per weight.

Quantization is a compromise you can read on the model card. The file gets smaller and the model gets a little less precise. For a first local chat, a small or mid-size instruct model in a 4-bit file is the usual starting point because it fits in machines people already have. You are not committing to that file forever. You are checking that the pipeline runs.

The file you download is the spec. Hosts such as Hugging Face list file sizes on the model page. Believe the file size you see there over a forum's memory. Download one small file before you buy anything.

RAM, VRAM, and unified memory

If you run on a separate graphics card, the weights want to sit in that card's VRAM. System RAM does not help much once the card is full, because shuffling weights across the bus is what makes a chat feel stuck. If the file is larger than VRAM, expect the run to fail or to fall off a cliff. Leave room for the operating system and the app. A 4 gigabyte file does not mean a 4 gigabyte card is comfortable.

If you run on the CPU, system RAM has to hold the file and everything else the machine is doing. It will work on modest hardware and it will be the slow path. That can still be the right path for a private document you are willing to wait on.

Apple Silicon Macs use unified memory, shared by the CPU and the GPU. There is no separate VRAM pool to fill first. A Mac with a lot of unified memory can hold a larger file than a PC whose discrete card is small, even when the PC has plenty of ordinary RAM. That is an architecture fact, not a benchmark. The amount of unified memory is the number to read on the spec sheet. A machine that cannot be upgraded later needs that number chosen honestly on day one.

Also leave disk space. Model files accumulate. A fast SSD makes loading less painful. It does not replace memory. Once the weights are loaded, the disk is not the bottleneck people think it is.

Image models are a different budget

The conversation so far is about language models. Image models, including the Stable Diffusion family, have their own files and they prefer a GPU. If pictures are the reason you are shopping, size the GPU from the image model's card, and read the comparison of image workflows so you know whether you need local weights at all. A hosted image tool may be cheaper than a card you will use twice a month.

Do not buy a large GPU because a language-model blog and an image-model blog used the same adjective, "local." Add the file sizes for the jobs you will actually run in the same afternoon. If you will not run them together, the larger of the two files sets the budget, not the sum.

What to do with the computer you have

Install a runner and try a small quantized instruct model, using the steps in the local guide. If it loads and the reply is usable for your notes, stop shopping. The desire for a larger model is real and it is not an emergency. Work for a week. Write down what failed: the file would not load, the wait was longer than the task was worth, or the quality of the small model was wrong for the documents you have. Those are three different purchases.

If the file would not load, you need memory, not a newer brand of the same memory. If the wait was the problem, a GPU with the file in VRAM is the usual fix, and you should confirm VRAM is larger than the file you intend to keep. If the quality was the problem, try one step up in model size only if the arithmetic says it fits. A larger model that does not fit will not be higher quality on your desk. It will be a download you cannot run.

Used, new, and the upgrade that is fake

Used workstation cards and used business PCs can be rational when the seller states the VRAM or the RAM in a way you can verify, and when the power supply and the case can actually take the card. A card that draws more power than the supply provides is not a bargain. Noise and heat are part of the purchase if the machine will sit next to you during a call.

New laptops are rational when you need the machine for ordinary work anyway and you choose memory at purchase because it is soldered. Buy the memory for the largest file you are willing to commit to, not for a flagship model you saw in a keynote. Flagship local models are a desktop conversation.

Upgrading storage when memory is the wall is the fake upgrade. Upgrading the CPU for a GPU-sized file is the same fake. Match the part to the wall you wrote down.

Privacy is a reason, and it has a shape

Local weights do not, by themselves, make a computer safe. They mean the prompt can be processed without sending it to that model's host, if the app is configured not to call a cloud API in parallel. Full-disk encryption, screen lock, and an account that is not shared still matter. A local model on a laptop that gets left in a car is not a privacy program.

If a client or a policy requires that certain files never touch a third-party model API, write that requirement next to the hardware budget. It justifies a machine that is merely adequate. It does not justify a second computer "for AI" that nobody patches. One maintained machine is the smaller risk.

The free stack guide is the alternative when privacy is not the requirement and the volume is low. Cloud free tiers cost no hardware and cost a different kind of care with what you paste. Choose with that trade visible.

A note you can take to a product page

Write four lines before you open a store.

If a listing does not tell you the memory number, it is not a listing you can use. Brands and years are secondary to that number. When you later measure tokens per second on your own files, keep the note. It is your benchmark. Someone else's chart will not be, because it was not your model file and it was not your machine.

The software side, the part where you type a prompt and get an answer without the cloud, is the other half. Do not buy ahead of it. Run the small file this week, on the hardware you have, and let the failure or the success edit the four lines.

#local AI #hardware #VRAM #Apple Silicon

Recommended on Amazon

The Coming Wave

The Coming Wave
Mustafa Suleyman. A clear look at how frontier AI changes work, power, and everyday tools.

AI Superpowers

AI Superpowers
Kai-Fu Lee. How the AI landscape is shifting, and how to position your work inside it.

All-in On AI

All-in On AI
Thomas H. Davenport & Nitin Mittal. How companies put generative AI into real operations, not just demos.

2k to 10k

2k to 10k
Rachel Aaron. A short, practical book on writing faster while keeping the work specific.

As an Amazon Associate I earn from qualifying purchases.