What running an AI model locally means

What's the difference between a web chatbot and a local model?

A web chatbot runs on someone else's computers. You type into a page, your words travel over the internet, and an answer comes back. Wikipedia calls this hosted inference: the model runs in remote data centres and the output reaches you over the internet.¹

A local model flips that around. The model runs on a computer you own, like your laptop or an office PC. Wikipedia notes that some open-weight models can be downloaded and run on a personal computer.¹

That's the whole idea. Same kind of software, different place to run it.

What do I actually download?

You download the model's "weights". These are the numbers the model learned when it was built, and they're what lets it write sentences. LM Studio's documentation says that to run a model on your computer you first need to download the weights.²

Once downloaded, the files sit on your storage. When you ask a question, the weights and other data are held in memory so the model can work.¹ That's why memory matters so much. The hardware page covers it, and model sizes and memory explains how big the files get.

Which programs run a downloaded model?

You can't just double-click a weights file. You need a program that loads it and gives you a chat window or a command prompt. Two common free ones are Ollama and LM Studio.

Ollama's quickstart describes a single command that fetches a model and opens a chat.³ LM Studio is an app you install on a Mac, a Windows PC or a Linux machine.²

Each has its own page: how Ollama works and how LM Studio works.

Does a local model need the internet?

Only for some jobs. LM Studio's documentation says it doesn't generally need the internet. Downloading new models is one thing that does.⁴ So you need a connection to fetch the model. After that, you may not.

Does "local" mean private?

Partly. Your questions don't have to go to a web service, which is a real change. But local doesn't mean nothing can leave the machine, and it doesn't change what your staff are allowed to type in.

Secrets and customer details don't belong in prompts unless your company's own policy says they can go there. The same goes for a local model. The privacy page looks at what can still leave the computer.

Can any business use any downloadable model?

No. Each model comes with licence terms, and they differ. Read the licence page before you build a routine around one.