There is a moment when a cloud AI service stops being the right answer. You need to analyse a contract that cannot leave the office. The internet connection is down and the work cannot wait. The monthly bill for API calls has quietly tripled. Local AI, running a model on your own machine, is the answer to all three. Running AI locally is not a smaller version of the cloud. It is a different engineering problem, governed by memory, not by features. The hardest part is not installing software. It is understanding which model your hardware can actually run at a usable speed.

What local AI is, and what it is not

A local AI system has four parts. The model weights are the numbers that define the model's behaviour. An inference runtime loads those weights and generates text, the most common being llama.cpp and its wrapper Ollama. A user interface lets you talk to it, with Open WebUI the most widely used. Optional agent frameworks let it call tools and edit files. Four dimensions decide whether local beats hosted. Privacy means your data never leaves the machine, which matters for legal, medical, and source code work. Cost becomes fixed after the hardware purchase, with no per token billing. Latency is predictable because there is no network round trip. Availability survives internet outages. The trade off is capability and speed: a consumer GPU cannot match the largest hosted models, and it will not match their token rate either. A hybrid setup, local for sensitive work and cloud for heavy reasoning, is often the honest answer.

The one number that governs everything: memory

Local inference is, first and foremost, a memory problem. The model weights must fit in fast memory, usually graphics card memory, or generation becomes painfully slow. A simple rule of thumb: a model with seven billion parameters at four bit quantization needs roughly four gigabytes for the weights alone. Double the parameters, double the memory. Each bit of precision per weight costs roughly one byte per billion parameters, so an eight bit model of the same size needs twice the memory of a four bit one. Quantization trades a small amount of quality for a large reduction in memory, and four bit is, for most practical work, the point where the trade off stops being worth taking. On top of the weights, the key value cache grows with the context window, so a long conversation needs more memory than the model card suggests. If the weights do not fit in graphics memory, layers spill over to system memory and then to disk, and speed drops by orders of magnitude. Before buying hardware, estimate the largest model you want to run, multiply its parameter count by the bytes per weight of your chosen quantization, and leave headroom for the cache. A mid range card with eight to twelve gigabytes runs seven to eight billion parameter models comfortably. Twenty four gigabytes opens the range of seventy billion parameter models at four bit. Without a discrete graphics card, CPU inference works but is slow, and Apple Silicon uses unified memory under its Metal backend, where the same memory rules apply with a different numbering.

A reproducible stack, step by step

Each command below is on its own line so it can be selected and copied. On Linux, install Ollama, which wraps llama.cpp and handles model downloads.

curl -fsSL https://ollama.com/install.sh | sh

On Windows and macOS, download the installer from ollama.com instead. Verify that the runtime sees your graphics card.

ollama ps

Download and run a first model. The example uses a seven billion parameter model; replace the name with any model from the Ollama library.

ollama run llama3.1:8b

For a proper chat interface, Open WebUI connects to the local runtime and supports multiple models, conversation history, and document upload. With Docker installed, one command starts it.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Open localhost:3000 in a browser, create an admin account, and the local model appears in the model selector. For automation, Ollama exposes an OpenAI compatible API on port 11434, so any tool written for a cloud endpoint can point at localhost instead by changing one line of configuration. Pin the runtime version and record the exact model name and quantization in a text file. Without this, a setup that works today will break silently when either is updated, and reproducing a result becomes impossible.

Beyond chat: retrieval, coding agents, and the limits

Chat is the entry point, not the destination. Retrieval augmented generation over local files lets the model answer from your own documents: the documents are split into chunks, converted to embeddings, stored in a local vector database, and relevant chunks are placed in the model's context before it answers. Done locally, this keeps confidential documents entirely on the machine. Local coding assistants connect the same runtime to an editor or terminal, but they demand more context management, because a real codebase far exceeds any context window. Autonomous coding agents add tool use, and here the local advantage becomes a security requirement: an agent that can edit files and run shell commands must run inside a sandbox, with approval gates for any destructive action and least privilege on every tool. Local generation of images, audio, and video is possible but far more demanding. Diffusion image models fit on mid range cards. Video generation does not, on consumer hardware, in any reasonable time. Knowing where the boundary is prevents the most common disappointment.

When it is slow, find the real bottleneck

Two numbers describe performance. Time to first token measures how long you wait before any answer appears. Tokens per second measures how fast the answer streams. A long wait followed by fast text points to a slow prefill phase, usually caused by a large prompt or missing graphics acceleration. Fast first token followed by slow text points to the weights not fitting in memory. Out of memory errors mean the model, the cache, or both exceed available memory, and the fix is a smaller model, a lower quantization, or a shorter context window. Diagnose by separating hardware, runtime, model, and configuration, because each has a different fix, and changing the wrong one wastes days. Benchmark on your own tasks, not on published numbers, because your documents and your context length determine real performance more than any synthetic test.

Further notes on applied AI engineering are published on the blog, and setup questions can be sent through the contact page. The source reference is Hector L. Adler, Local AI on Your PC, 2026.