GGUF Format Explained: What It Is and Which Quantization to Choose

If you've ever downloaded a local model, you've seen files with a .gguf extension and tags like Q4_K_M or Q8_0 in the name. It looks like a cipher, but there's simple logic behind it. Here's what the format is, what all those letters and numbers mean, and which variant to pick for your machine.

What GGUF is

GGUF is a file format for storing a local language model. One model, one file — you download it and run it on your own machine. The format was designed to make models comfortable to run on ordinary hardware, not just server-grade GPUs.

The main thing to understand: a .gguf file is already a ready-to-run, compressed version of a model. "Compressed" is the key word, and that's where it gets interesting.

Why quantization matters

A model in its original form weighs a lot and needs expensive hardware. Quantization is a way to shrink it by making the numbers inside it coarser. Roughly speaking, instead of very precise values, the model stores slightly rounded ones. The file becomes several times smaller and runs on an ordinary laptop, and quality drops less than you'd expect.

That's exactly why the same GGUF model exists in several variants: from heavily compressed and light to nearly untouched and heavy. You're picking a balance between size and quality for your own hardware.

What Q4_K_M, Q5, Q8, and the rest mean

Now to decode the labels. The number after Q is the compression depth: the smaller it is, the more compressed the model and the smaller the file.

• Q8 — nearly the original. The heaviest variant, top quality, but needs a lot of memory.
• Q5 — a middle ground closer to full quality. Noticeably lighter than Q8, with little difference in answers.
• Q4 (like Q4_K_M) — the most popular choice. A good balance: the model is already noticeably lighter, and quality is more than decent for most tasks.
• Below Q4 (Q3, Q2) — quite light, but quality starts dropping noticeably; worth it only if the model won't fit otherwise.

The letters K_M in Q4_K_M are a variant of the compression method; in practice it's enough to know that K_M is a modern, well-balanced variant, and it's the default choice.

What "instruct" in the name means

Another common tag. Instruct (or chat) in a model's name means it was fine-tuned to follow instructions and hold a conversation — to answer your requests instead of just continuing text. For chat and work, you want exactly this version; a model without instruct is a base model, and it behaves differently.

How to run a GGUF model

You can download a .gguf file from Hugging Face, the biggest model catalog out there. Then the file needs something to open it, and this is where the choice of tool decides how convenient the whole thing is.

You can spin up a model through developer tools on the command line. Or you can do it like a regular app: Doka can load its own .gguf file through a model manager, no terminal involved. Pick the file, activate it — the model runs locally.

If you don't want to bother picking a file, Doka has built-in models that install automatically for your hardware — and you can hook up a .gguf separately whenever you want a specific model.

How to tell what size your computer can handle

A simple rule: a model runs in memory, so look at the file size and your RAM (or VRAM, if you're running on a GPU). The model should fit with some room to spare. If the Q8 version doesn't fit, drop to Q5 or Q4_K_M — the file gets smaller. If you'd rather not eyeball it, there's a model picker calculator that reads your GPU or memory and tells you which size actually fits.

The main takeaway

GGUF is a format for ready-to-run local models, and tags like Q4_K_M just say how compressed the model is. A bigger number means higher quality and more weight, a smaller one means lighter and easier on your hardware. Start with Q4_K_M and a version with instruct in the name. You can run all of this without a command line — download Doka and load your file.