LiteLLM Setup Proxy: One Gateway to Rule Every LLM in Your Stack

Your app calls OpenAI. Your team wants Claude. Your finance team wants Azure for compliance. Your infra team is tired of managing five different SDKs, credential stores, and rate limit handlers. Enter LiteLLM: a single OpenAI-compatible API gateway that routes to 100+ LLM providers, handles fallbacks, tracks spend, and manages virtual keys. This LiteLLM setup proxy guide gets you from zero to production gateway in 20 minutes.

Want more depth? See our related guides: LiteLLM Setup Proxy: A Complete Developer Guide to Building a Unified LLM Gateway and LiteLLM Setup Proxy: One Gateway to Rule Every LLM in Your Stack.

What You'll Build

By the end of this guide, you'll have:

• LiteLLM Proxy running in Docker with persistent configuration
• Multiple LLM providers configured (OpenAI, Anthropic, Azure, local models)
• Virtual API keys for team-based access control
• Load balancing and fallback routing between providers
• Spend tracking and rate limiting enabled
• A single OpenAI-compatible endpoint your apps can call

Prerequisites

Before we start, make sure you have:

• Docker 24.0+ and Docker Compose v2 installed
• A Linux server (Ubuntu 22.04/24.04 LTS recommended) or local machine with port 4000 available
• At least 2GB RAM and 1 CPU core (LiteLLM is lightweight; the LLMs are the heavy lifters)
• API keys for at least one provider (OpenAI, Anthropic, Azure, etc.)
• curl and jq for testing

LiteLLM itself doesn't run models—it proxies requests. The resource requirements are minimal unless you enable the built-in UI or connect to a large PostgreSQL database for analytics.

Step 1: Quick Start with Docker

Let's get the proxy running with a basic configuration. This validates everything works before we add complexity.

1.1 Create a Configuration File

Create litellm_config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gpt-4o-azure
    litellm_params:
      model: azure/gpt-4o
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2025-01-01-preview"

general_settings:
  master_key: sk-1234  # Admin key for the proxy UI
  proxy_batch_write_at: 60  # Flush logs every 60s

Key concepts:

• model_name: The alias your apps will use. Pick whatever makes sense.
• litellm_params.model: The provider/model identifier. Format is provider/model-name.
• os.environ/VAR: Reads from environment variables. Never hardcode API keys.
• master_key: The admin key for accessing the proxy UI and managing virtual keys.

1.2 Run with Docker

docker run -d \
  --name litellm-proxy \
  -p 4000:4000 \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=sk-your-openai-key \
  -e ANTHROPIC_API_KEY=sk-ant-your-anthropic-key \
  -e AZURE_API_KEY=your-azure-key \
  -e AZURE_API_BASE=https://your-resource.openai.azure.com \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

Verify it's running:

curl -s http://localhost:4000/health | jq .

You should see {"status":"ok"}. The proxy is live.

1.3 Test Your First Request

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

Notice the OpenAI-compatible endpoint. LiteLLM translates this to whatever provider you're using. Change "model": "claude-3-5-sonnet" and the same request hits Anthropic's API.

Step 2: Production Docker Compose with PostgreSQL

The quick start is fine for testing. Production needs persistence for virtual keys, spend tracking, and request logs.

2.1 Extended Configuration

Update litellm_config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 100  # Requests per minute limit

  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 50

  - model_name: gpt-4o-fallback
    litellm_params:
      model: azure/gpt-4o
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2025-01-01-preview"

router_settings:
  routing_strategy: simple-shuffle  # Load balance across deployments
  num_retries: 3
  timeout: 30s
  fallbacks: [{"gpt-4o": ["gpt-4o-fallback"]}]

general_settings:
  master_key: sk-1234
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 60
  store_model_in_db: true

litellm_settings:
  success_callback: ["langfuse"]  # Optional: send traces to Langfuse
  failure_callback: ["langfuse"]

2.2 Docker Compose Stack

Create docker-compose.yml:

version: "3.8"

services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: secure_db_password_123
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
      AZURE_API_KEY: ${AZURE_API_KEY}
      AZURE_API_BASE: ${AZURE_API_BASE}
      DATABASE_URL: postgresql://litellm:secure_db_password_123@db:5432/litellm
    depends_on:
      db:
        condition: service_healthy
    command: ["--config", "/app/config.yaml"]
    restart: unless-stopped

volumes:
  postgres_data:

Create .env:

OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
AZURE_API_KEY=your-azure-key
AZURE_API_BASE=https://your-resource.openai.azure.com

Deploy:

docker compose up -d

Check logs:

docker compose logs -f litellm

Step 3: Manage Virtual Keys and Teams

Hardcoding your master key in every app is a security nightmare. LiteLLM's virtual keys let you issue scoped credentials with spend limits and model restrictions.

3.1 Access the Admin UI

Open http://localhost:4000/ui and log in with your master_key (sk-1234 in our example).

3.2 Create a Virtual Key via API

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "prod-app-key",
    "max_budget": 100.00,
    "models": ["gpt-4o", "claude-3-5-sonnet"],
    "tpm_limit": 10000,
    "rpm_limit": 100
  }'

Response includes a new virtual key (e.g., sk-abc123...). This key:

• Can only call gpt-4o and claude-3-5-sonnet
• Has a $100 monthly budget limit
• Is rate-limited to 10K tokens/min and 100 requests/min
• Cannot access the admin UI or generate other keys

3.3 Use Virtual Keys in Your App

import openai

client = openai.OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-your-virtual-key"  # Not the master key
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Your app uses the standard OpenAI SDK. LiteLLM handles the translation, routing, and provider-specific quirks.

Step 4: Configure Load Balancing and Fallbacks

When GPT-4o hits rate limits or Azure goes down, your app shouldn't break. LiteLLM can retry, fall back, and load balance across deployments.

4.1 Multiple Deployments of the Same Model

Update litellm_config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 100
      weight: 50  # 50% of traffic

  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2025-01-01-preview"
      rpm: 200
      weight: 50  # 50% of traffic

  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 50

router_settings:
  routing_strategy: latency-based  # Route to fastest deployment
  num_retries: 3
  timeout: 30s
  retry_policy:
    AuthenticationErrorRetries: 0
    RateLimitErrorRetries: 5
    TimeoutErrorRetries: 3
  fallbacks: [{"gpt-4o": ["claude-3-5-sonnet"]}]

What's happening:

• Load balancing: Two gpt-4o deployments split traffic 50/50 by weight, or route to the lowest-latency option
• Retries: Rate limit errors retry 5 times with exponential backoff
• Fallbacks: If both GPT-4o deployments fail, automatically route to Claude 3.5 Sonnet

4.2 Test Fallback Behavior

# Temporarily break OpenAI by using a bad key
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-your-virtual-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Test fallback"}]
  }'

# Watch logs to see the retry + fallback in action
docker compose logs -f litellm | grep -i "fallback\|retry"

Step 5: Monitor Spend and Usage

LiteLLM tracks every request, token count, and dollar spend. Query it via API or view the admin UI.

5.1 Check Spend by Virtual Key

curl -X GET http://localhost:4000/spend/keys \
  -H "Authorization: Bearer sk-1234" | jq .

5.2 Export Usage Logs

curl -X GET "http://localhost:4000/spend/logs?start_date=2026-07-01&end_date=2026-07-06" \
  -H "Authorization: Bearer sk-1234" | jq '.[] | {key: .api_key, model: .model, spend: .spend, tokens: .total_tokens}'

5.3 Set Budget Alerts

Add to litellm_config.yaml:

alert_settings:
  budget_alert_webhook: https://hooks.slack.com/services/your/webhook/url
  budget_threshold: 0.8  # Alert at 80% of budget

When a virtual key hits 80% of its monthly budget, LiteLLM POSTs to your webhook. Build automations from there.

Step 6: Tips, Troubleshooting, and Production Hardening

6.1 Proxy Won't Start

Check the most common issues:

docker compose logs litellm | tail -n 50

• Config file not found: Ensure the volume mount path matches. The container expects /app/config.yaml
• Database connection failed: PostgreSQL must be healthy before LiteLLM starts. The depends_on with healthcheck handles this
• Invalid model name: Check the provider prefix. It's openai/gpt-4o, not just gpt-4o
• API key errors: Verify environment variables are passed correctly. Use docker compose exec litellm env | grep API_KEY

6.2 Requests Return 401 or 403

• Virtual key expired or invalid: Regenerate in the admin UI or via API
• Model not in allowed list: The virtual key's models array restricts what it can call
• Budget exceeded: Check spend logs. Increase max_budget or reset the key

6.3 Fallbacks Not Working

• Fallback model not configured: Both primary and fallback must be in model_list
• Same provider failing: If OpenAI is down globally, falling back to another OpenAI deployment won't help. Use cross-provider fallbacks (OpenAI → Anthropic → Azure)
• Timeout too short: Increase router_settings.timeout if providers are slow to error

6.4 Performance Tuning

• Enable Redis caching: Cache identical requests to reduce API calls and cost
• Connection pooling: LiteLLM uses HTTPX with keep-alive. For high throughput, increase router_settings.max_parallel_requests
• Batch logging: The proxy_batch_write_at setting buffers logs to PostgreSQL. Increase for lower DB load, decrease for fresher data

6.5 Security Hardening

• Change the master key: Use a strong, random key. openssl rand -hex 32
• Enable HTTPS: Put LiteLLM behind Traefik or Nginx with Let's Encrypt
• Restrict admin UI: IP whitelist or VPN-only access to /ui
• Rotate provider keys regularly: Update environment variables and restart the container
• Audit virtual keys: Review active keys monthly. Revoke unused ones

6.6 Backup and Restore

Your virtual keys, spend data, and request logs live in PostgreSQL:

docker compose exec db pg_dump -U litellm litellm > litellm-backup-$(date +%F).sql

Restore:

docker compose exec -T db psql -U litellm litellm < litellm-backup-2026-07-06.sql

Wrapping Up

You now have a unified LLM gateway that speaks OpenAI to your apps and translates to 100+ providers behind the scenes. With virtual keys, spend tracking, load balancing, and automatic fallbacks, you've turned a mess of API credentials and SDKs into a single, manageable endpoint.

LiteLLM's power is in its simplicity: one config file, one API format, one place to manage access and costs. For teams running multiple LLM providers—or planning to migrate between them—it's the sanest architecture choice available.

That said, production deployments at scale—multi-region routing, custom model hosting, fine-grained RBAC, and deep observability integration—need architecture beyond a single Docker host.

Need help designing a production LLM gateway, integrating custom providers, or building cost controls and compliance into your AI infrastructure? Talk to our team—we've deployed LiteLLM-backed gateways for fintech, healthcare, and SaaS platforms managing millions of tokens daily. We'll get your LLM routing right.