Running Local LLMs with Ollama: A Practical Guide

Ollama has become the easiest way to run open-weight language models on your own machine, with almost none of the setup pain that used to come with llama.cpp. This guide walks through installing it on Linux, running your first models, and wiring it into your own scripts.

Installing Ollama on Linux

curl -fsSL https://ollama.com/install.sh | sh

# Confirm the service is running
systemctl status ollama --no-pager

# Check it's listening
curl http://localhost:11434

Pulling and Running Models

# General-purpose chat model
ollama pull llama3.1:8b

# Strong at coding tasks
ollama pull qwen2.5-coder:7b

# Smaller, fast model for quick tasks
ollama pull phi3.5

ollama run llama3.1:8b

Calling Ollama from the REST API

Every model you pull is exposed over a local HTTP API, which makes it trivial to script against — no cloud account, no API key.

import requests
import json

def ask(prompt, model="llama3.1:8b"):
    resp = requests.post(
        "http://localhost:11434/api/generate",
        json={"model": model, "prompt": prompt, "stream": False},
    )
    return resp.json()["response"]

if __name__ == "__main__":
    answer = ask("Summarize the difference between TCP and UDP in two sentences.")
    print(answer)

Creating a Custom Modelfile

You can bake a system prompt, temperature, and other defaults into a named model with a Modelfile, similar to a Dockerfile:

FROM llama3.1:8b

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

SYSTEM """
You are a terse Linux sysadmin assistant. Answer with exact commands
and minimal explanation unless asked to elaborate.
"""

ollama create sysadmin-bot -f ./Modelfile
ollama run sysadmin-bot "How do I find which process is using port 8080?"

Managing Storage

Models add up fast — a handful of 7B-13B models can easily eat 50GB+. Keep an eye on what’s installed:

ollama list
ollama rm phi3.5
du -sh ~/.ollama/models

Wrapping Up

Ollama removes almost all the friction from local inference on Linux. Once it’s running as a systemd service, it behaves like any other local API you’d build against — which makes it a natural fit for privacy-sensitive scripting, offline tooling, and quick experimentation without burning API credits.