Can My PC Run It? A Practical Guide to Local LLM Hardware and Quantization
Want to run AI on your own machine but unsure whether your computer can handle it? Here is what actually matters — RAM vs VRAM, how to read parameter counts, and how to pick a quantization level that fits the hardware you already own.
Running a local AI model — fully offline, with none of your data leaving the machine — usually starts with one question: can my computer actually handle this?
It is easy to lose your nerve when you see model names ending in 3B, 8B or 70B, or file names like GGUF and Q4_K_M. In practice the requirements are far less intimidating than the jargon suggests, and modern compression techniques mean an ordinary laptop can run a genuinely useful model today.
This guide covers what hardware actually matters, and how to pick the model that gets the most out of the machine you already own.
The two numbers that matter: RAM and VRAM
Two pieces of hardware decide what you can run.
VRAM (video memory on your graphics card) is the one that counts most. If the whole model fits in VRAM, generation is fast — typically 30–80 tokens per second. A discrete GPU such as an NVIDIA RTX card is a large advantage here.
System RAM is the fallback. When VRAM runs out — or when there is no discrete GPU at all — a runtime like Ollama loads part or all of the model into system memory and runs it on the CPU instead. It still works, but expect a noticeable drop to roughly 2–10 tokens per second.
Apple Silicon (M1/M2/M3/M4) sits in a category of its own. Its unified memory architecture lets the GPU address system RAM directly as video memory, so a Mac with 16 GB of RAM can comfortably run models that would need a dedicated graphics card on a PC.
Quantization, explained without the maths
Loaded at full precision (16-bit, or FP16), an 8B model — 8 billion parameters — needs around 16 GB of memory. That is out of reach for most machines.
Quantization solves this. Think of it as saving a RAW photo as a JPEG: the file gets dramatically smaller, and the human eye struggles to tell the difference. The numbers inside the model are stored at reduced precision — 8-bit, 4-bit, sometimes lower.
- Q4 (4-bit) is the sweet spot. Model size drops by about 75% — that 8B model becomes roughly 4.8 GB — while answer quality typically falls by only 1–2%.
- Q8 (8-bit) keeps more precision and cuts size by about half. Worth it only if you have memory to spare and want maximum fidelity.
This is why Ollama pulls Q4_K_M by default when you download a model. It is the best trade-off for most hardware.
Which model for which machine
| Your machine | Model size that runs comfortably | Realistic expectation |
|---|---|---|
| 8 GB RAM, no discrete GPU | 2B–4B (quantized) | Fast, good for summarising and simple Q&A |
| 16 GB RAM, or 6–8 GB VRAM (RTX 3060/4060), or Mac 16 GB | 7B–9B (quantized) | The practical sweet spot for daily use |
| 32 GB RAM, or 12–24 GB VRAM, or Mac 32 GB+ | 14B–32B (quantized) | Noticeably stronger reasoning and coding |
A useful rule of thumb: a Q4-quantized model needs roughly 0.6 GB of memory per billion parameters, plus about 1–2 GB of headroom for context. An 8B model at Q4 therefore wants around 6 GB free.
Three models worth starting with
Llama 3.1 (8B) — the reliable all-rounder. Strong English reasoning, good at code, and by far the best supported by tooling and tutorials. If you only try one model, try this.
Gemma 2 (2B / 9B) — Google's carefully tuned family, and unusually strong at multilingual work for its size. On an 8 GB machine, gemma2:2b is startlingly capable for how little memory it uses. With 16 GB or more, gemma2:9b is worth the upgrade.
Phi-3 / 3.5 (3.8B) — Microsoft's small model, and the answer when memory is tight or the hardware is old. It uses about 2.2 GB yet matches older models twice its size on English Q&A and coding.
Getting started
You do not need an expensive workstation. Once you understand parameter counts and pick a sensible quantization level, the computer already on your desk can become a private, fully offline AI server.
Install Ollama, then run:
ollama run gemma2:2b
or, if you have 16 GB or more:
ollama run llama3.1
That is the whole setup. Owning your own inference — no API keys, no usage metering, no data leaving the machine — starts there.
Hardware worth considering
- RTX 3060 12 GB — 12 GB of VRAM at a used-market price. Still the best value point for running 7–14B models smoothly.
- 32 GB DDR4 (2×16 GB) — the sensible minimum if you plan to run 14B models on CPU, or keep several models loaded at once.
- 1 TB NVMe SSD — models are 4–9 GB each and load far faster from NVMe than from a hard disk. Enough room for a dozen of them.
These are affiliate links. If you buy through them the blog may earn a small commission at no extra cost to you — we only list hardware that actually fits this guide.
