Self-Host LiteLLM: One API for All Your AI Models in 15 Minutes

Tired of juggling different APIs for OpenAI, Anthropic, and local models? LiteLLM gives you a single OpenAI-compatible endpoint for 100+ LLM providers — self-hosted, private, and under your control.

The Problem: API Chaos

If you're building AI-powered applications, you know the drill. OpenAI uses one SDK. Anthropic uses another. Local models via Ollama have yet another interface. Your code becomes a mess of provider-specific integrations, each with its own auth, error handling, and quirks.

LiteLLM solves this. It's a lightweight proxy server that normalizes 100+ LLM providers behind a single OpenAI-compatible API. Write your code once, swap models freely.

Why Self-Host LiteLLM?

• Privacy: Your API keys and request logs stay on your infrastructure
• Cost control: Set budgets, rate limits, and track spending per model/user
• Vendor independence: Switch between OpenAI, Anthropic, or local models with zero code changes
• Unified auth: One API key for your apps, LiteLLM handles provider keys internally

Prerequisites

• A VPS or homelab server with Docker and Docker Compose installed
• Minimum 1GB RAM (2GB+ recommended for production)
• API keys for providers you want to use (OpenAI, Anthropic, etc.) — optional if only using local models

Step 1: Create the Docker Compose File

Create a directory for LiteLLM and add the following docker-compose.yml:

version: "3.8"

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
    environment:
      - LITELLM_MASTER_KEY=sk-your-secure-master-key-here
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    restart: unless-stopped
    networks:
      - litellm-network

  # Optional: Redis for rate limiting and caching
  redis:
    image: redis:7-alpine
    container_name: litellm-redis
    restart: unless-stopped
    networks:
      - litellm-network

networks:
  litellm-network:
    driver: bridge

Step 2: Create the Config File

Create config.yaml in the same directory:

model_list:
  # OpenAI models
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  # Anthropic models
  - model_name: claude-sonnet-4
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

  # Local models via Ollama
  - model_name: llama3.1
    litellm_params:
      model: ollama/llama3.1
      api_base: http://host.docker.internal:11434

# General settings
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

# Rate limiting and budgets
litellm_settings:
  fallbacks: [{"gpt-4o": ["claude-sonnet-4"]}]
  num_retries: 3
  request_timeout: 600
  redis_host: redis
  redis_port: 6379

# Spend tracking
router_settings:
  routing_strategy: least-busy

Step 3: Start the Server

docker compose up -d

Check the logs to ensure it started correctly:

docker logs -f litellm

Step 4: Verify It's Working

Test with a simple request:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-secure-master-key-here" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

You should get a standard OpenAI-format response. The same endpoint works for any configured model — just change the model parameter.

Step 5: Add to Your Application

Point your existing OpenAI SDK to LiteLLM:

import openai

client = openai.OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-your-secure-master-key-here"
)

response = client.chat.completions.create(
    model="gpt-4o",  # or "claude-sonnet-4" or "llama3.1"
    messages=[{"role": "user", "content": "Write a haiku about Docker"}]
)
print(response.choices[0].message.content)

Advanced: Multiple Users and Budgets

LiteLLM supports virtual keys for different users or applications:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-your-secure-master-key-here" \
  -H "Content-Type: application/json" \
  -d '{"max_budget": 10.0, "models": ["gpt-4o", "llama3.1"]}'

This creates a virtual key with a $10 budget limited to specific models.

Production Checklist

• Put LiteLLM behind a reverse proxy (Traefik or Nginx) with HTTPS
• Enable Redis caching for faster responses and lower costs
• Set up Prometheus metrics at /metrics for monitoring
• Configure fallbacks so requests route to backup models if one fails
• Review the admin UI at /ui for spend tracking and key management

Next Steps

Now that you have a unified AI API, consider:

• Self-hosting Ollama for local model inference
• Adding Grafana + Prometheus to monitor LiteLLM metrics
• Deploying n8n to build AI workflows with your new unified API

LiteLLM turns AI model management from a headache into a solved problem. One endpoint, any model, full control.