How to Run AI Models Locally with Ollama: Complete Beginner's Guide

Ollama Local AI

TL;DR

Ollama lets you run powerful large language models (LLMs) directly on your own computer — no cloud, no API keys, no monthly bills. Just download the installer, pull a model, and start chatting. In this guide you will install Ollama on Windows, Mac, or Linux, compare popular models by size and capability, set up a beautiful Web UI, integrate local AI into VS Code, and use the API for your own projects. All data stays on your machine. Completely private. Completely free.

What Is Ollama?

Ollama is a tool that packages, serves, and manages large language models locally. It handles the entire GPU acceleration stack (CUDA, Metal, ROCm), model quantization (shrinking models to fit on consumer hardware), and weight management — things that would otherwise require deep ML engineering knowledge to set up.

In plain terms: Ollama is the Docker of AI models. You type ollama pull llama3.2, it downloads the model and all its dependencies, and then ollama run llama3.2 starts a ChatGPT-like conversation in your terminal. No configuration files. No Python virtual environments. No CUDA toolkit installation.

The project was created by the same engineers who built Docker and Kubernetes at Google and Docker, Inc. That infrastructure DNA shows: Ollama is reliable, well-documented, and cross-platform from day one.

Why Run AI Locally?

Privacy

When you use ChatGPT or Claude, every message you type is sent to a remote server, processed, and logged. OpenAI’s privacy policy explicitly states they may use your conversations for training (unless you opt out via a specific form). With Ollama, your data never leaves your device. This matters for: legal documents, proprietary code, personal journals, medical information, and anything covered by GDPR, HIPAA, or internal confidentiality policies.

Cost

Running Ollama costs $0 per query. No API credits to buy, no token metering, no surprise bills. Once you own the hardware, the marginal cost of running a model is just electricity — roughly $0.01-0.03 per hour of active GPU usage. Compare this to GPT-4o API pricing ($2.50-10 per million tokens) and the savings add up quickly for heavy users.

Offline Access

LLMs on a plane, in a rural area, or during an internet outage. Ollama works fully offline. All models are cached locally after the initial download. This is invaluable for field researchers, travelers, and anyone in areas with unreliable connectivity.

No Censorship or Content Filters

Cloud AI services apply content moderation filters that can reject legitimate requests. Local models do not have server-side filters. You control what the model can and cannot say through your system prompt. This is important for research on sensitive topics, creative writing with adult themes, and security testing.

System Requirements

Ollama runs on CPU-only machines, but performance improves dramatically with a GPU.

Component	Minimum	Recommended	Enthusiast
RAM	8 GB	16 GB	32 GB+
GPU VRAM	None (CPU only)	8 GB	24 GB+
GPU Type	Integrated graphics	NVIDIA RTX 3060+ / Apple M1+	NVIDIA RTX 4090 / Apple M2 Ultra
Storage	20 GB free	50 GB free	200 GB+ NVMe SSD
OS	Windows 10+, macOS 13+, Linux (kernel 5+)	Same	Same

GPU notes:

NVIDIA: CUDA-capable GPUs with 6 GB+ VRAM are ideal. RTX 3060 (12 GB variant) offers the best value.
Apple Silicon: M1/M2/M3/M4 chips use Metal acceleration. The unified memory architecture means an M2 Max with 32 GB RAM can run 70B-parameter models that would need a $2,000 NVIDIA GPU on PC.
AMD: ROCm support is present but less mature. Expect more troubleshooting on AMD GPUs.
CPU-only: Works for models up to 7B-8B parameters. Expect 2-5 tokens per second on a modern CPU with AVX2 support.

Installation Guide

Windows

Download the Windows installer from the Ollama website. It is a standard .exe installer (about 400 MB) that sets up Ollama as a Windows service. After installation, you will see the Ollama icon in the system tray.

Open PowerShell or Command Prompt and verify the installation:

ollama --version

If you have an NVIDIA GPU, make sure the NVIDIA drivers are up to date (version 535 or later for CUDA 12). Ollama auto-detects CUDA during installation.

Windows-specific note: Ollama stores models in %USERPROFILE%\.ollama by default. If your C drive is low on space, set a custom model directory using the OLLAMA_MODELS environment variable before launching Ollama. Create a system environment variable:

OLLAMA_MODELS=D:\ollama-models

macOS

Download the .dmg from the Ollama website, drag the app to Applications, and launch it. On first launch, macOS will prompt you to allow the app to run — approve it.

Ollama runs as a menu bar app on macOS. Click the llama icon to see status and quick actions.

For Apple Silicon Macs, Ollama automatically uses Metal (MPS) acceleration. Verify GPU support:

ollama run llama3.2 --verbose

Check the output for Metal or MPS in the backend information. If you see CPU fallback, ensure your macOS is version 13.3 or later.

Linux

Use the one-line install script:

curl -fsSL https://ollama.com/install.sh | sh

For NVIDIA GPUs on Linux, also install the NVIDIA Container Toolkit:

# Ubuntu/Debian
sudo apt install nvidia-container-toolkit
sudo systemctl restart ollama

To check if GPU acceleration is active:

ollama run llama3.2
# In the chat, type:
/bye
# Check logs:
journalctl -u ollama | grep -i cuda

You should see messages about CUDA device detection and memory allocation. If not, ensure nvidia-smi shows your GPU and the driver is version 535+.

Downloading Your First Model

Models live in the Ollama library. To see what is available, browse the library on the Ollama website or search via the CLI:

ollama list

Here is how to pull and run popular models:

Meta Llama 3.2

ollama pull llama3.2

At 3 billion parameters, this is the sweet spot for CPU-only machines. It fits in 2 GB of RAM, runs at conversational speeds even on integrated graphics, and delivers surprisingly coherent responses for general chat, summarization, and simple coding tasks.

ollama pull llama3.2:1b

The 1B variant runs on anything — even a Raspberry Pi 5. Quality is noticeably lower but it responds instantly.

DeepSeek-R1

ollama pull deepseek-r1:8b

DeepSeek-R1 is a reasoning model that thinks step-by-step before answering. The 8B parameter version is excellent at math, logic puzzles, and coding problems. It shows its reasoning process (chain of thought) in the chat, making it a great learning tool.

Available sizes: 1.5B, 7B, 8B, 14B, 32B, 70B, 671B. The 8B and 14B versions are the practical sweet spots for consumer hardware.

Qwen 2.5

ollama pull qwen2.5:7b

Alibaba’s Qwen 2.5 is currently the best multilingual model for non-English languages. It handles Chinese, Japanese, Korean, Arabic, and European languages natively. The 7B and 14B versions are competitive with Llama 3 in English while dramatically better in Chinese.

For coding tasks:

ollama pull qwen2.5-coder:7b

The Coder variant is fine-tuned for programming and rivals GPT-4o in many coding benchmarks while running entirely locally.

Mistral

ollama pull mistral:7b

Mistral 7B is a reliable general-purpose model from the French AI company Mistral AI. Excellent for creative writing, long-form content, and European language support. The mistral-nemo variant (12B) offers a step up in quality while still fitting in 8 GB VRAM.

Phi-4

ollama pull phi4:14b

Microsoft’s Phi-4 is a compact reasoning and coding model that punches well above its weight class. It excels at logic puzzles, competitive programming problems, and complex multi-step reasoning. The 14B model requires about 9 GB of VRAM.

Model Comparison Table

Model	Parameters	Size on Disk	RAM Needed (CPU)	VRAM Needed (GPU)	Best For	Speed (GPU)	Speed (CPU)
llama3.2:1b	1B	0.6 GB	2 GB	1 GB	Ultra-fast simple tasks	120+ tok/s	15-25 tok/s
llama3.2	3B	2.0 GB	4 GB	2 GB	General chat, everyday use	100+ tok/s	8-15 tok/s
qwen2.5:7b	7B	4.7 GB	8 GB	5 GB	Multilingual, Chinese content	70+ tok/s	4-8 tok/s
mistral:7b	7B	4.1 GB	8 GB	5 GB	Creative writing, EU languages	70+ tok/s	4-8 tok/s
deepseek-r1:8b	8B	4.9 GB	10 GB	6 GB	Math, coding, reasoning	65+ tok/s	3-7 tok/s
llama3.3:latest	8B	4.9 GB	10 GB	6 GB	High-quality general purpose	65+ tok/s	3-7 tok/s
phi4:14b	14B	8.5 GB	16 GB	9 GB	Advanced reasoning, coding	40+ tok/s	2-4 tok/s
deepseek-r1:14b	14B	8.9 GB	18 GB	10 GB	Complex reasoning tasks	35+ tok/s	2-3 tok/s
qwen2.5:32b	32B	20 GB	36 GB	20 GB	Production-quality output	15+ tok/s	N/A
deepseek-r1:70b	70B	43 GB	80 GB	40 GB	Near-API-quality reasoning	8+ tok/s	N/A

How to read this table: “tok/s” means tokens per second — how fast the model generates text. 10 tok/s feels conversational. 30+ tok/s feels instant. CPU speeds assume a modern processor (AMD Ryzen 7 or Intel i7, 13th-gen or later). GPU speeds assume an NVIDIA RTX 3060 12 GB or better.

Using the Ollama CLI

Basic Chat

ollama run llama3.2

This opens an interactive chat session. Type messages and press Enter. The model responds. Type /bye to exit, /help to see all commands.

Single Prompt (Non-Interactive)

ollama run llama3.2 "Explain quantum computing in simple terms"

Useful for scripts, piping, and one-off queries. The output goes to stdout.

echo "Summarize: $(cat meeting-notes.txt)" | ollama run llama3.2

List Downloaded Models

ollama list

Shows model name, ID, size, and last modified date.

Remove a Model

ollama rm llama3.2:1b

Frees up disk space. You can re-download any time.

Show Model Details

ollama show llama3.2

Displays the model’s Modelfile (configuration), parameter count, quantization level, and context window size.

GUI Frontends: Chat Without the Terminal

The terminal is great, but a web-based ChatGPT-style interface is more comfortable for daily use. Here are the best options.

Open WebUI (Recommended)

Open WebUI is the most polished and actively maintained GUI for Ollama. It looks and feels like ChatGPT. Features include:

Multi-user support with login
Conversation history with search
Markdown rendering, code highlighting, LaTeX math
Model switching mid-conversation
Document upload for RAG (chat with your files)
Web search integration
Mobile-friendly responsive design

Install with Docker:

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000, create an account, and start chatting. Open WebUI auto-detects your local Ollama instance.

Chatbox

Chatbox is a desktop app (not web-based) available for Windows, Mac, and Linux. It is lighter than Open WebUI and ideal if you do not want to run Docker. Download from the Chatbox website and point it at http://localhost:11434. Features include prompt libraries, message branching, and a compact UI that stays in your system tray.

AnythingLLM

AnythingLLM combines a chat interface with a built-in RAG pipeline. You can drag and drop PDFs, Word docs, and web pages into a workspace and ask questions about them. It uses Ollama as the backend LLM. Great for research and document-heavy workflows. Available as a desktop app and Docker container.

Comparison: GUI Frontends at a Glance

Frontend	Type	Docker Required	RAG Support	Multi-User	Mobile Friendly
Open WebUI	Web app	Yes	Yes (built-in)	Yes	Yes
Chatbox	Desktop app	No	No	No	Separate mobile app
AnythingLLM	Desktop + Docker	Optional	Yes (strong)	No (single user)	Limited
Ollama Web UI	Web app	Yes	No	No	Yes
LM Studio	Desktop app	No (self-contained)	No	No	No

Integrating with Your Apps

VS Code with Continue.dev

The Continue extension brings AI code completion and chat into VS Code and JetBrains IDEs. To connect it to Ollama:

Install the Continue extension from the VS Code marketplace
Open Continue’s config file (~/.continue/config.json)
Add your Ollama models:

{
  "models": [
    {
      "title": "Llama 3.2",
      "provider": "ollama",
      "model": "llama3.2"
    },
    {
      "title": "DeepSeek R1 8B",
      "provider": "ollama",
      "model": "deepseek-r1:8b"
    },
    {
      "title": "Qwen 2.5 Coder",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen Coder",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b"
  }
}

Use a small model like qwen2.5-coder:1.5b for tab autocomplete (needs to be instant) and a larger model for chat. Continue also supports embedding models from Ollama for codebase-wide search with @codebase.

Using the Ollama API

Ollama exposes a REST API at http://localhost:11434. Any programming language can call it.

Python example (no library needed, just requests):

import requests
import json

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Write a haiku about programming",
        "stream": False
    }
)
print(response.json()["response"])

Streaming response (real-time token generation):

import requests
import json

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "llama3.2",
        "prompt": "Explain recursion in one paragraph",
        "stream": True
    },
    stream=True
)

for line in response.iter_lines():
    if line:
        data = json.loads(line)
        print(data["response"], end="", flush=True)

Chat API (multi-turn conversations with roles):

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "llama3.2",
        "messages": [
            {"role": "system", "content": "You are a helpful coding tutor."},
            {"role": "user", "content": "What is a closure in JavaScript?"},
            {"role": "assistant", "content": "A closure is..."},
            {"role": "user", "content": "Can you give me a practical example?"}
        ],
        "stream": False
    }
)

Customizing Models with Modelfiles

A Modelfile is like a Dockerfile for AI models — it defines how a model behaves, what system prompt it uses, and what parameters it runs with.

Create a file called Modelfile:

FROM llama3.2

# Set the system prompt
SYSTEM """
You are a patient and knowledgeable Python programming tutor.
Always explain concepts with simple examples first, then show
the advanced usage. Never assume prior knowledge. Use analogies
that a beginner would understand.
"""

# Increase context window (default is 2048)
PARAMETER num_ctx 4096

# Adjust creativity (0 = deterministic, 1 = creative)
PARAMETER temperature 0.7

# Limit response length
PARAMETER num_predict 2048

# Prevent repetition
PARAMETER repeat_penalty 1.1

Build and run the custom model:

ollama create python-tutor -f Modelfile
ollama run python-tutor

Common parameters to adjust:

Parameter	Default	What It Does
`temperature`	0.7	Higher = more creative/random. Lower = more deterministic
`num_ctx`	2048	Context window size in tokens. Larger = more conversation history
`num_predict`	128	Max tokens in the response. -1 = unlimited
`top_p`	0.9	Nucleus sampling. 0.1 = only the most likely words
`repeat_penalty`	1.1	Higher = model punished more for repeating itself
`stop`	None	Strings that stop generation immediately

Tips for Better Performance

Use the right quantization level: Ollama automatically selects the best quantization for your hardware. You can force a specific quant with tags like llama3.2:q4_K_M. Q4_K_M is the best balance of quality and speed. Q8_0 is higher quality but uses 2x the memory. Q2_K is lower quality but runs on very limited hardware.
Close GPU-heavy applications: Browsers with hardware acceleration, video editors, and games compete for VRAM. Close them when running larger models.
Adjust num_ctx cautiously: The context window (default 2048 tokens) consumes memory linearly. Setting num_ctx 8192 on a 7B model roughly doubles memory usage. Increase only when you need longer conversations.
Batch your API requests: If you are processing many prompts via the API, use the api/embed or api/generate endpoints with proper connection reuse instead of spawning a new process per request.
Keep Ollama updated: Check for updates monthly (ollama --version). The team ships performance improvements and new quantization methods regularly.
Use the right model for the right task: Do not use DeepSeek-R1 70B for a simple classification task. Chain models: use llama3.2 for quick tasks and switch to a larger model only when you need deep reasoning.
GPU memory over-subscription: If a model needs 10 GB VRAM and your GPU has 8 GB, Ollama will offload some layers to system RAM. This works but is 5-10x slower per layer. Check with ollama ps to see how many layers are GPU-resident vs CPU-resident.

FAQ

Q: Is Ollama completely free?

Yes. Ollama itself is open-source (MIT license) and free forever. The models on the Ollama library are also free to download and use. Some models have usage restrictions (e.g., Llama models require accepting Meta’s license terms), so check the model’s page before using it in a commercial product.

Q: Can I use Ollama for commercial projects?

It depends on the model, not Ollama. Ollama (the software) is MIT-licensed and can be used commercially. Llama 3.2 from Meta is free for most commercial uses (under 700 million monthly active users). DeepSeek models are open-weight with permissive licenses. Always check the specific model’s license before deploying to production.

Q: Why is my model so slow on a good GPU?

Common causes: (a) the model is larger than your VRAM, causing layer offloading to system RAM (check ollama ps), (b) you are running other GPU-intensive apps simultaneously, (c) your NVIDIA drivers are outdated, (d) you chose a high quantization (Q8_0) when Q4_K_M would suffice and be 2-3x faster.

Q: Can I run two models at the same time?

Yes. Ollama loads models on demand and unloads idle models after 5 minutes (configurable with OLLAMA_KEEP_ALIVE). Running two models simultaneously that combined exceed your VRAM will cause thrashing — each model will swap layers in and out, making both extremely slow. Use models that fit within your VRAM budget together, or run them sequentially.

Q: How do I transfer my models to another computer?

Models are stored in ~/.ollama/models (Linux/Mac) or %USERPROFILE%\.ollama\models (Windows). Copy that directory to the same location on the new computer and restart Ollama. Models appear automatically in ollama list.

Q: Does Ollama support image generation or vision models?

Yes for vision, no for image generation. Ollama supports vision (multimodal) models like llava, bakllava, and llama3.2-vision. These models can look at images and answer questions about them. Image generation (like Stable Diffusion) is out of scope — use dedicated tools like ComfyUI or Automatic1111 for that.

Q: Can I create my own model or fine-tune an existing one?

You can import GGUF-format model files (the standard format for quantized LLMs) using ollama create. Fine-tuning (training) is not done within Ollama itself — you would use tools like Unsloth or Axolotl to fine-tune, export to GGUF, and then import into Ollama with a Modelfile referencing the GGUF file.

Q: Is Ollama safe? Can models access my files?

Ollama runs as a local service bound to 127.0.0.1 (localhost only) by default. It cannot access your files, network, or system beyond what the API allows. Models are static weights — they cannot execute code or access the filesystem unless you explicitly build a tool-calling pipeline that grants those capabilities (e.g., through Open WebUI’s function-calling features).

Q: How does Ollama compare to LM Studio?

Both are excellent local LLM runners. LM Studio has a built-in model browser, a polished GUI, and more model format support (GGUF directly). Ollama has better CLI tooling, a cleaner API, broader ecosystem integration (Open WebUI, Continue.dev, LangChain), and feels more like infrastructure. Choose LM Studio if you want an all-in-one desktop app. Choose Ollama if you want a server/API that other tools connect to.

Q: What models should a beginner start with?

Start with llama3.2 (3B parameters). It is small enough to run on any computer, fast enough to feel instant, and smart enough to be genuinely useful. Once you are comfortable, try deepseek-r1:8b for reasoning tasks or qwen2.5-coder:7b for programming help. Avoid jumping straight to 70B models — they require expensive hardware and the 7B-14B range already delivers impressive results.

This guide was updated in June 2026. Model availability, performance characteristics, and version numbers may change. Check the Ollama website and model library for the latest information. Hardware recommendations reflect mid-2026 consumer hardware landscape.