Skip to Content

LiteLLM Setup Proxy: One Gateway for Every LLM in Your Stack

Deploy LiteLLM with Docker, route requests to OpenAI, Anthropic, and local models through a single OpenAI-compatible API, and stop managing API keys in every app.

LiteLLM Setup Proxy: One Gateway for Every LLM in Your Stack

Every new LLM provider ships its own SDK, its own auth format, and its own rate limits. Your code ends up littered with if provider == "anthropic" branches. LiteLLM fixes this. It is an open-source AI gateway that exposes a single OpenAI-compatible API and routes requests to 100+ providers behind the scenes. This guide gives you a production-ready LiteLLM setup proxy—from a Docker container to a unified LLM gateway with virtual keys, rate limits, and spend tracking.

If you want a deeper architectural overview, see LiteLLM Setup Proxy: The Complete Developer Guide to a Unified LLM Gateway.

What You Will Build

By the end of this guide you will have:

  • LiteLLM Proxy running in Docker with a config-driven model list
  • Multiple providers (OpenAI, Anthropic, Azure, Ollama) routed through one endpoint
  • Virtual API keys with per-key rate limits and spend tracking
  • PostgreSQL persistence for key management and usage logs
  • HTTPS access behind a reverse proxy

Prerequisites

Before you start, make sure you have:

  • A Linux server (Ubuntu 22.04+ or Debian 12+) with Docker and Docker Compose installed
  • At least 2 CPU cores and 2 GB RAM
  • API keys for at least one LLM provider (OpenAI, Anthropic, Azure, etc.)
  • A domain or subdomain pointed at your server (for HTTPS)
  • Basic familiarity with YAML, environment variables, and REST APIs

Check your Docker version:

docker --version
docker compose version

If Docker is missing:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
newgrp docker

Step 1: Configure LiteLLM with Multiple Providers

LiteLLM uses a config.yaml file to define which models it serves and where they live. You can mix OpenAI, Anthropic, Azure, Bedrock, Ollama, and custom OpenAI-compatible endpoints in one file.

Create a project directory:

mkdir ~/litellm-proxy && cd ~/litellm-proxy

Create litellm_config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: "os.environ/OPENAI_API_KEY"

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-sonnet-20240229
      api_key: "os.environ/ANTHROPIC_API_KEY"

  - model_name: gpt-4-azure
    litellm_params:
      model: azure/my-gpt4-deployment
      api_base: "os.environ/AZURE_API_BASE"
      api_key: "os.environ/AZURE_API_KEY"
      api_version: "2025-01-01-preview"

  - model_name: llama-local
    litellm_params:
      model: ollama/llama3.1
      api_base: "http://ollama:11434"

general_settings:
  master_key: sk-1234
  database_url: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"

Key concepts:

  • model_name is what your clients call. It is an alias.
  • model under litellm_params is the provider and actual model identifier.
  • os.environ/VAR_NAME pulls values from environment variables at runtime.
  • master_key is your admin key. It must start with sk-.

Step 2: Deploy with Docker Compose

For production, run LiteLLM with PostgreSQL for persistence. This stores virtual keys, usage logs, and rate limit state.

Create docker-compose.yml:

version: "3.8"

services:
  db:
    image: postgres:16
    container_name: litellm-db
    restart: unless-stopped
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: llmproxy
      POSTGRES_PASSWORD: dbpassword9090
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - litellm-net

  litellm:
    image: ghcr.io/berriai/litellm-database:latest
    container_name: litellm-proxy
    restart: unless-stopped
    ports:
      - "4000:4000"
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      AZURE_API_KEY: ${AZURE_API_KEY}
      AZURE_API_BASE: ${AZURE_API_BASE}
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    depends_on:
      - db
    networks:
      - litellm-net
    command: >
      --config /app/config.yaml --detailed_debug

volumes:
  postgres-data:

networks:
  litellm-net:
    driver: bridge

Create a .env file for your provider keys:

OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
AZURE_API_KEY=your-azure-key
AZURE_API_BASE=https://your-resource.openai.azure.com

Start the stack:

docker compose up -d

Check the logs:

docker logs -f litellm-proxy

When you see RUNNING on http://0.0.0.0:4000, the proxy is ready.

Step 3: Test the Unified API

LiteLLM Proxy is 100% OpenAI-compatible. Any client that speaks OpenAI can use it.

Test with curl:

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello from LiteLLM"}]
  }'

Switch to Claude by changing one field:

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "claude-sonnet",
    "messages": [{"role": "user", "content": "Hello from LiteLLM"}]
  }'

Your application code does not change. Only the model parameter changes. LiteLLM handles the rest—auth translation, request formatting, response parsing.

Step 4: Create Virtual Keys with Rate Limits

Handing out your master key is a bad idea. LiteLLM lets you create virtual keys with per-key budgets, rate limits, and model access controls.

Generate a virtual key with a 10 requests-per-minute limit:

curl -X POST 'http://localhost:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "rpm_limit": 10,
    "max_budget": 5.00,
    "models": ["gpt-4o", "claude-sonnet"]
  }'

Expected response:

{
  "key": "sk-abc123..."
}

Give this key to a specific app or team. They can only use the models you allow, only spend the budget you set, and only hit the rate limit you define. If they exceed it, LiteLLM returns a 429 automatically.

For a deeper dive into key management and team-based access, read LiteLLM Setup Proxy: The Complete Developer Guide to a Unified LLM Gateway.

Step 5: Secure LiteLLM with HTTPS

Running on port 4000 over HTTP is fine for testing. In production, add a reverse proxy with TLS.

Add Caddy to Your Compose File

  caddy:
    image: caddy:2.8
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy-data:/data
      - caddy-config:/config
    networks:
      - litellm-net

volumes:
  caddy-data:
  caddy-config:

Create Caddyfile:

llm.yourdomain.com {
    reverse_proxy litellm-proxy:4000
}

Restart the stack:

docker compose up -d

Caddy fetches a Let's Encrypt certificate automatically. Your LLM gateway is now at https://llm.yourdomain.com.

Step 6: Integrate with Your Application

Because LiteLLM speaks OpenAI, integration is trivial. Point your existing OpenAI client at the proxy.

Python Example

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.yourdomain.com",
    api_key="sk-abc123..."  # your virtual key
)

response = client.chat.completions.create(
    model="claude-sonnet",
    messages=[{"role": "user", "content": "Explain LiteLLM in one sentence"}]
)

print(response.choices[0].message.content)

Node.js Example

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://llm.yourdomain.com',
  apiKey: 'sk-abc123...'
});

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Explain LiteLLM in one sentence' }]
});

console.log(response.choices[0].message.content);

Your code stays clean. One client, one auth pattern, any model behind it.

If you want to see how LiteLLM fits into a larger AI architecture, check out LiteLLM Setup Proxy: One Gateway to Rule Every LLM in Your Stack.

Tips and Troubleshooting

Model returns "not found" or 404

  • Verify the model_name in your request matches an entry in model_list
  • Check that the provider key environment variable is set and passed into the container
  • Confirm the provider model string is correct (e.g., anthropic/claude-3-sonnet-20240229)

Database connection errors on startup

  • Ensure PostgreSQL is healthy before LiteLLM starts. Add a depends_on in Compose
  • Verify the database_url in general_settings matches your Postgres credentials
  • Grant the database user full permissions: GRANT ALL PRIVILEGES ON DATABASE litellm TO llmproxy;

Rate limits not enforced

  • Rate limits require PostgreSQL. Without a database, LiteLLM cannot track usage across restarts
  • Verify the virtual key was created successfully and is being used in the Authorization header

SSL verification errors with self-signed certs

  • Add ssl_verify: false under litellm_settings in your config for internal endpoints
  • Do not disable this for production providers

High latency on first request

  • LiteLLM may cold-start the provider connection. Subsequent requests are faster
  • Enable caching with Redis for repeated prompts

Spend tracking shows zero

  • Ensure the database is connected and the LiteLLM_SpendLogs table exists
  • Check that the virtual key has max_budget set and the request uses that key, not the master key

Next Steps

You now have a working LiteLLM setup proxy with multi-provider routing, virtual keys, rate limits, and HTTPS. That is a solid foundation. From here, consider:

  • Adding fallback models so requests route to a backup provider if the primary is down
  • Enabling Redis caching for repeated prompts to reduce latency and cost
  • Setting up team-based budgets and model access for multi-tenant environments
  • Integrating Prometheus and Grafana for LLM usage dashboards
  • Adding custom callbacks for logging requests to your own analytics pipeline

LLM infrastructure should not be a maze of provider-specific SDKs. With LiteLLM, you get one API, one auth pattern, and full control over how your applications consume AI.

Need Help at Scale?

If you are running production workloads and need multi-region failover, enterprise SSO, custom model routing, or migration from a direct provider integration, we can help. Contact the Sysbrix team and we will architect an LLM gateway that fits your stack.

Deploy with Coolify: Turn Any VPS Into Your Own Heroku
Install Coolify on a fresh VPS, connect your domain, and deploy your first app with SSL in under 30 minutes.