Self-Host Ollama: Run AI Models Locally in Your Homelab in 15 Minutes

Want to run LLMs like Llama 3, Mistral, or Code Llama on your own hardware — no API keys, no per-token costs, complete privacy? Ollama makes it dead simple. In this guide, you'll have a local AI model serving API up and running in 15 minutes.

Why Self-Host Ollama?

• Privacy: Your prompts never leave your server
• Cost: Zero per-token fees after hardware
• Offline: Works without internet once models are pulled
• Simple API: OpenAI-compatible endpoints
• GPU optional: Runs on CPU for smaller models

Prerequisites

• A VPS or homelab server with:
  • Minimum: 4 cores, 8GB RAM (for 7B models on CPU)
  • Recommended: 8+ cores, 16GB+ RAM, or NVIDIA GPU with 8GB+ VRAM
• Docker + Docker Compose installed
• ~10GB free disk space per model

Step 1: Create the Docker Compose File

Create a directory for Ollama and a docker-compose.yml:

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_HOST=0.0.0.0
    restart: unless-stopped
    # Uncomment for NVIDIA GPU support:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  ollama_data:

Step 2: Start Ollama

docker compose up -d

# Verify it's running
docker logs -f ollama

You should see Ollama is running in the logs.

Step 3: Pull Your First Model

# Pull Llama 3.1 8B (great all-rounder)
docker exec -it ollama ollama pull llama3.1:8b

# Or a smaller, faster model
docker exec -it ollama ollama pull phi3:mini

# For coding tasks
docker exec -it ollama ollama pull codellama:7b

Model sizes to expect:

• phi3:mini — ~2.3GB, runs great on CPU
• llama3.1:8b — ~4.7GB, good balance
• codellama:7b — ~3.8GB, code completion
• llama3.1:70b — ~40GB, needs serious GPU

Step 4: Test It

# Interactive chat
docker exec -it ollama ollama run llama3.1:8b

# Or via the API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Why is self-hosting AI important?",
  "stream": false
}'

Step 5: OpenAI-Compatible API

Ollama exposes an OpenAI-compatible endpoint, so existing tools work out of the box:

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain Docker in one sentence."}
  ]
}'

Point any OpenAI-compatible client to http://your-server:11434/v1 and it just works.

Step 6: Add a Web UI (Optional)

For a ChatGPT-like interface, add Open WebUI to your compose file:

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - openwebui_data:/app/backend/data
    restart: unless-stopped
    depends_on:
      - ollama

volumes:
  ollama_data:
  openwebui_data:

docker compose up -d
# Open http://localhost:3000

Verification Checklist

• curl http://localhost:11434 returns "Ollama is running"
• docker exec ollama ollama list shows your pulled models
• API call returns a coherent response
• Open WebUI loads and can chat (if installed)

Hardware Recommendations

Next Steps

• Pair with LiteLLM for a unified API gateway across multiple models
• Add Ollama to n8n for AI-powered automation workflows
• Set up monitoring with Prometheus + Grafana to track GPU/RAM usage
• Secure it behind Traefik with Authentik SSO

Related Guides

• Self-Host Portainer — manage Ollama containers visually
• Self-Host Authentik — protect your AI endpoints
• Self-Host Borgmatic — back up your model data

Published: September 12, 2026 | Sysbrix Self-Hosting Series