A Local Model Instead of a Cloud API: Ollama, LM Studio, and Qwen in an Agent

Most AI agents reach out to a cloud API by default: every request goes to a provider, and you pay by the token for it. But an agent's model can be swapped for a local one — and then it runs on your own hardware, for free and privately. Here's why that matters and how to connect a local model through Ollama, LM Studio, or directly.

Why swap the cloud for a local model

Three reasons people move off a cloud API to a local one.

The first is money. A cloud API counts tokens, and on active work, the bill adds up fast. A local model is free, no matter how much you run it. The second is privacy: code, files, and conversations don't go to someone else's servers. The third is independence: no foreign account, no card, no VPN needed, and you can work offline.

In exchange, you give up some capability — a top cloud model at its peak is stronger than a local one. But for a large share of tasks, a local model is more than enough, and the difference in cost and privacy is real.

Ollama, LM Studio, and your own file — what's the difference

There are a few ways to keep a local model, and they complement each other. All of them, by the way, sit on the same engine — there's a separate breakdown of the difference between an engine and a wrapper.

Ollama is a tool that runs a model and exposes it over a local API. Convenient if you already work with it and just want an agent to connect to it.

LM Studio is an app with a graphical interface and a model catalog; it can also expose a model over API. More welcoming to people who don't like the command line. How these two options differ in practice is covered in the article on Ollama or LM Studio.

Your own .gguf — you can skip the middle layer entirely and load a model file straight into the agent. There's a separate breakdown of what GGUF is and which version to pick.

How this works in Doka

Doka supports all three paths, and this is its standard mode, not a workaround.

The simplest is a built-in model: on first launch, Doka downloads one that fits your hardware, and it's already local. Want a specific model? Load your own .gguf through the model manager. And if you already have Ollama or LM Studio running, connect them in the "API" tab: point at the local server's address, and the agent starts talking to it instead of the cloud.

Which model to pick

Depends on the task and your hardware. For code, people reach for code-focused models like Qwen Coder; for general tasks, universal ones; for sensitive company data, an on-prem-friendly one. Size gets matched to memory: the more you have, the bigger and smarter a model you can run.

Not just a model — an agent around it

Worth remembering the point of all this. A local model on its own just answers in text. The value shows up once there's an agent around it giving it access to files and the terminal. Then you get not just a local chat, but a local executor that does the work while sending nothing outward.

The takeaway

You can put a local model into an agent instead of a cloud API — through Ollama, LM Studio, or your own .gguf — and stop paying for tokens without losing privacy. Capability is a bit more modest than the top cloud tier, but it's free, needs no accounts, and works offline. Download Doka for free — all three paths are supported.