Ollama, in one sentence: it turns local LLM inference from a "PhD-level CUDA configuration exercise" into something as routine as installing Node.js.
At 179k stars on GitHub, it's a Go-based monolithic service that wraps ggml-org/llama.cpp underneath. One curl install.sh | sh and you have it running. Over the past two years, this product shape has essentially become the default for local LLMs. You may not use Ollama directly, but whatever you're using probably talks to Ollama's API.
This piece takes it apart.
The Problem It Solves
Running large-model inference used to be a high-friction affair:
- Heavy VRAM requirements (even a 4090 barely handles a 7B model)
- Quantization algorithm selection (GPTQ? AWQ? GGUF Q4_K_M?)
- Half a day spent installing CUDA, cuBLAS, and ROCm dependencies
- Model weights arriving as
.binor.safetensorsfiles, with Python glue code required to run them
Ollama packages all of this into a single binary. Download, quantization, loading, inference, and HTTP serving, all bundled. The goal is "one curl line and you're done." The analogy: Docker did this for containers, Ollama does it for local LLMs.
The value here isn't technical depth. It's the drop in barrier from "engineer-level" to "ordinary user."
How to Use It
Three steps: install, run, chat.
# 1. Install (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh
# 2. Pull a model and chat
ollama run gemma3
# 3. REST API call
curl http://localhost:11434/api/chat -d '{
"model": "gemma3",
"messages": [{"role": "user", "content": "Why is the sky blue?"}],
"stream": false
}'
# 4. Python SDK
from ollama import chat
resp = chat(model='gemma3', messages=[
{'role': 'user', 'content': 'Why is the sky blue?'}
])
print(resp.message.content)
The server listens on :11434 with an OpenAI-compatible REST API. The official SDK installs via pip install ollama, and you get a working chatbot in two lines. ollama-js covers the JavaScript side.
That API compatibility matters: Claude Code, Codex, and similar Agent tools can swap their base_url and route inference through a local Ollama instance. It cuts the dependency on cloud API lock-in by one layer.
How It Works
Ollama runs as a Go monolithic service, but internally it breaks into four layers:
┌─────────────────────────────────────────┐
│ ① CLI / Server entry (Gin + cobra) │
│ - ollama run / pull / serve │
│ - HTTP listener on :11434 │
└─────────────┬───────────────────────────┘
│
┌─────────────▼───────────────────────────┐
│ ② Model Registry + Modelfile parser │
│ - GGUF / Safetensors weight index │
│ - Chat template + quantization config│
│ - Turns models into addressable, │
│ versionable objects │
└─────────────┬───────────────────────────┘
│
┌─────────────▼───────────────────────────┐
│ ③ llama.cpp CGo binding layer │
│ - Dynamic library loading │
│ - Calls ggml-org/llama.cpp for │
│ inference │
│ - Metal / CUDA / ROCm / Vulkan │
└─────────────┬───────────────────────────┘
│
┌─────────────▼───────────────────────────┐
│ ④ Scheduler │
│ - Concurrent model load/unload │
│ - LRU VRAM management │
│ - Request queue + streaming chunks │
└─────────────────────────────────────────┘
Several points stand out:
Modelfile is Ollama's model-definition DSL, similar to a Dockerfile. Syntax like FROM qwen2.5-coder:7b, SYSTEM "You are a translator", and PARAMETER temperature 0.7 bundles model weights, prompt templates, and inference parameters into a single addressable object. Models can now be version-controlled with git.
llama.cpp binding runs through CGo. Ollama doesn't implement matrix multiplication itself. Inference performance, quantization, and KV cache all come from upstream ggml-org/llama.cpp. That also sets Ollama's performance ceiling.
Scheduler is the underrated layer. Ollama doesn't prevent you from pulling multiple models simultaneously, but VRAM is finite. The scheduler handles LRU eviction, timeout recycling, and concurrent request queuing. When two models collide on VRAM, the experience is "old model gets kicked, reload adds a few seconds of latency."
Platform backends are well-covered: Metal on macOS, CUDA + ROCm + Vulkan on Linux, CUDA on Windows. One binary covers the major platforms, sparing developers from backend selection.
Strengths and Weaknesses
Strengths
Consistent cross-platform experience. Install on Mac, Windows, Linux, or Docker and the usage is identical. This matters more for developer experience than it sounds. You don't need to relearn when switching machines.
Model marketplace aggregation. One ollama pull grabs mainstream open-weight models like Gemma, Qwen, DeepSeek, Kimi, and GLM. The official library covers dozens of models, and third-party models (community quantizations, LoRA merges) can be pushed to private registries.
OpenAI-compatible API. A smart strategic choice. The /v1/chat/completions endpoint matches OpenAI's, so any OpenAI-compatible tool (Claude Code, Codex, LangChain, LlamaIndex) just needs a URL swap. Ollama effectively captures the migration gravity of the OpenAI ecosystem.
Modelfile turns models into versionable assets. Previously, models were unmanaged .bin files. Now Modelfile plus tags give models the same reproducibility as Dockerfiles.
Broad ecosystem. Frontends like Open WebUI, LibreChat, and Lobe Chat connect directly to Ollama's API. A whole UI layer has grown on top of this runtime.
Weaknesses
Single-machine ceiling. Ollama doesn't do distributed inference across multiple nodes. Your model size is capped by one machine's VRAM. Running 70B+ at full precision requires either massive single-node VRAM or a different solution.
Fixed quantization strategy. Ollama exposes predefined quantization tiers (Q4_0, Q4_K_M, Q8_0, etc.). Fine-grained quantization control means going back to llama.cpp. Servers like vLLM offer more flexibility here.
Go monorepo tax. Want to customize inference logic? Patch llama.cpp, then rebuild Ollama's CGo bindings. Iteration cycles are longer than with a pure Python stack.
Closed-source drift. After version 0.4, parts of the server module went closed-source, sparking community controversy. GitHub issues show sharply divided opinions. This trend raises questions about long-term openness.
Side-by-side comparison with alternatives:
| Tool | Position | Strength | Weakness |
|---|---|---|---|
| Ollama | Local runtime, CLI-first | Fastest onboarding, widest ecosystem | Not the fastest inference, closed-source drift |
| LM Studio | GUI frontend | User-friendly for non-engineers | Not open source, limited features |
| vLLM | Server-side inference engine | Strong performance (PagedAttention), production-grade | Complex deployment, high local-debugging friction |
| llama.cpp | Low-level C++ engine | Maximum flexibility, raw performance ceiling | High usage barrier, no HTTP server |
Ollama picked the "good enough and easy" route, not the "extreme performance" route. That's a deliberate trade-off.
Hands-On Test
M2 Max, 64GB unified memory.
Baseline: qwen2.5-coder:7b
- Cold start to first token: ~1.8s
- Sustained generation: ~38 token/s
- Resident memory: ~5GB
Step up: 32B Q4_K_M quantized
- First token: ~3.2s
- Sustained generation: ~11 token/s
- Handles 8k context code-completion tasks
Concurrency test: Running two models (7B + 13B) simultaneously, VRAM pressure triggered LRU eviction by the scheduler. Reloading the evicted model took 4.5s, during which requests blocked. This experience has room for improvement. Ideally the scheduler would warm models ahead of time or refuse the concurrency upfront.
Overall: 7B-class models feel smooth for daily use; the 32B quantized version handles serious tasks but isn't fast. For production scenarios needing 70B+, a proper inference service is still the way to go.
Who Should Use It
Good fit:
- Developers debugging Agents locally or comparing model options
- Offline demos and privacy-sensitive scenarios (financial/medical data that can't leave the network)
- Small teams wanting to avoid OpenAI API pricing lock-in
- Agent developers bridging Claude Code or Codex to local inference
Not a fit:
- Serious 70B+ full-precision evaluation work (vLLM/SGLang serve better here)
- High-concurrency production inference (not a design goal)
- Training or fine-tuning (that's unsloth / LLaMA-Factory territory)
- Windows users needing peak GPU performance (Mac/Linux backends are more stable; Windows CUDA occasionally has compatibility issues)
Closing
Ollama isn't the strongest local-inference option by raw performance, nor the most flexible. But it nails the product experience of "running a large model locally." That matters more to most engineers than benchmark scores.
Docker won against complex LXC setups on the strength of "one command and it runs." Ollama is replaying that play in the local LLM space.
The closed-source trend bears watching. If you're planning to use Ollama as production infrastructure, consider pinning to the last fully open-source 0.3.x release rather than following 0.4+.
References: Ollama GitHub repository, llama.cpp project, Ollama official documentation.