Self-Host Ollama: Run Local LLMs on Your Own Server in 15 Minutes

Every AI tool you use today phones home to OpenAI, Anthropic, or Google. Your data leaves your infrastructure, you pay per token, and you're locked into someone else's rate limits. Ollama changes that. It lets you run large language models — Llama, Mistral, Gemma, CodeLlama — directly on your own hardware, with zero API keys and full data privacy.

In this guide, you'll deploy Ollama with Docker, pull a model, and start chatting via CLI and API — all in about 15 minutes.

What Problem Does Ollama Solve?

• Data privacy: Sensitive prompts never leave your server.
• No API costs: Run inference locally after initial download.
• Offline capable: Works without internet once models are cached.
• Simple API: OpenAI-compatible REST endpoint for easy integration.

Prerequisites

• A VPS or bare-metal server with at least 4 CPU cores, 8 GB RAM, 50 GB SSD (16 GB RAM recommended for 7B+ models).
• Docker & Docker Compose installed.
• Optional: NVIDIA GPU with CUDA drivers for faster inference.

Step 1 — Deploy Ollama with Docker Compose

Create a project directory and docker-compose.yml:

mkdir ~/ollama && cd ~/ollama
cat > docker-compose.yml <<'EOF'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    environment:
      - OLLAMA_ORIGINS=*
    # Uncomment for NVIDIA GPU support:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]

volumes:
  ollama-data:
EOF

Start the container:

docker compose up -d

Step 2 — Pull Your First Model

Ollama hosts pre-quantized models. Pull Llama 3.1 (8B) — a solid general-purpose model:

docker exec -it ollama ollama pull llama3.1

Other great options:

• ollama pull mistral — fast, capable 7B model
• ollama pull codellama — code generation specialist
• ollama pull gemma2 — Google's lightweight model

Step 3 — Chat via CLI

docker exec -it ollama ollama run llama3.1

Type prompts directly. Exit with /bye.

Step 4 — Use the REST API

Ollama exposes an OpenAI-compatible API on port 11434:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Explain Docker networking in one paragraph."
}'

Streaming chat completion (OpenAI-compatible):

curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "Write a Python function to reverse a string."}]
}'

Step 5 — Verification

• Check container health: docker ps | grep ollama
• List downloaded models: docker exec ollama ollama list
• Test API response: curl -s http://localhost:11434/api/tags | jq

Next Steps

• Add a web UI: Pair Ollama with Open WebUI for a ChatGPT-like interface.
• Integrate with n8n: Use the HTTP node to call Ollama in automation workflows.
• Multi-model routing: Deploy LiteLLM in front of Ollama for unified API access.
• Read next: Self-Host Open WebUI

Questions? Drop a comment or reach out — happy self-hosting.