LiteLLM Setup Proxy: The Complete Developer Guide to a Unified LLM Gateway

If your team is juggling API keys for OpenAI, Anthropic, Azure, and Bedrock, you are bleeding time and money. LiteLLM Proxy fixes this. It gives you one OpenAI-compatible endpoint that routes to every LLM provider you use. One key. One URL. One place to enforce rate limits, track spend, and swap models without touching application code.

This guide walks you through a production-ready LiteLLM setup proxy deployment. By the end, you will have a unified gateway running on your infrastructure with multi-provider routing, virtual key management, and observability hooks. If you are new to LiteLLM, you may also find our companion guide LiteLLM Setup Proxy: One Gateway to Rule Every LLM in Your Stack useful for understanding the broader architecture.

What You Will Build

Here is what a working LiteLLM Proxy setup looks like in practice:

• Developers call http://your-proxy:4000/chat/completions with any OpenAI SDK or HTTP client
• The proxy routes requests to OpenAI GPT-4o, Anthropic Claude, Azure OpenAI, or AWS Bedrock based on the model alias
• Each team gets a virtual key with its own rate limits and budget
• All requests are logged and cost-tracked in a PostgreSQL database

No provider-specific SDKs. No key sprawl. No code changes when you switch from GPT-4o to Claude 4.

Prerequisites

Before you start, make sure you have the following:

• Docker installed and running (or Python 3.9+ if you prefer the CLI)
• API keys for at least one LLM provider (OpenAI, Anthropic, Azure, or AWS Bedrock)
• A PostgreSQL database for virtual keys and spend tracking (local, cloud, or Docker)
• Basic familiarity with YAML and curl

If you are running this locally for testing, Docker Compose with a Postgres container is the fastest path. For production, use a managed Postgres like Supabase, Neon, or AWS RDS.

Step 1: Install LiteLLM Proxy

LiteLLM offers three ways to run the proxy: Docker, Docker Compose (with Postgres), or the Python CLI. Pick the one that fits your environment.

Option A: Docker (Fastest for Testing)

docker run -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --port 4000

Option B: Docker Compose (Proxy + Database)

This is the recommended setup for anything beyond a quick test. It spins up the proxy alongside a Postgres container for key management and spend tracking.

Create a docker-compose.yml:

version: "3.9"
services:
  litellm:
    image: ghcr.io/berriai/litellm-database:latest
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - DATABASE_URL=postgresql://litellm:litellm@db:5432/litellm
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    command: ["--config", "/app/config.yaml", "--detailed_debug"]

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm
      POSTGRES_DB: litellm
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  pgdata:

Option C: Python CLI

If you already have Python set up and prefer to avoid Docker:

# Install LiteLLM with proxy dependencies
pip install 'litellm[proxy]'

# Run the interactive setup wizard
litellm --setup

The wizard will ask you to pick providers, paste API keys, and set a port. It writes a litellm_config.yaml automatically. For a deeper walkthrough of the CLI path, see LiteLLM Setup Proxy: A Complete Developer Guide to Building a Unified LLM Gateway.

Step 2: Configure Your Model List

The heart of LiteLLM Proxy is the model_list in litellm_config.yaml. This is where you register every model your applications can call.

Create litellm_config.yaml in your project root:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: azure-gpt4o
    litellm_params:
      model: azure/my-azure-deployment
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2025-01-01-preview"

  - model_name: bedrock-claude
    litellm_params:
      model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
      aws_region_name: us-east-1
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY

general_settings:
  master_key: sk-1234
  database_url: "postgresql://litellm:litellm@db:5432/litellm"

litellm_settings:
  drop_params: true
  request_timeout: 60

Key rules to remember:

• model_name is what your clients send in the model field
• litellm_params.model uses the format provider/model-id
• Use os.environ/VAR_NAME to load secrets from environment variables
• master_key must start with sk- and is your admin key for creating virtual keys

Step 3: Start the Proxy and Test It

With your config saved, start the proxy:

docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -p 4000:4000 \
  ghcr.io/berriai/litellm-database:latest \
  --config /app/config.yaml --detailed_debug

You should see RUNNING on http://0.0.0.0:4000 in the logs. Now test it with a curl:

curl -X POST 'http://0.0.0.0:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello from LiteLLM Proxy!"}]
  }'

The response is standard OpenAI format, regardless of which provider handles the request. That is the point. Switch the model to claude-sonnet or azure-gpt4o and the API contract stays identical.

Step 4: Set Up Virtual Keys for Teams

Hardcoding the master key in every app is a bad idea. LiteLLM lets you generate virtual keys with per-key budgets, rate limits, and model access controls.

Create a Virtual Key with Rate Limits

curl -X POST 'http://0.0.0.0:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "rpm_limit": 100,
    "tpm_limit": 10000,
    "max_budget": 50.00,
    "models": ["gpt-4o", "claude-sonnet"]
  }'

The response returns a new key like sk-abc123.... Hand this to a team. They can only call the models you allowed, only up to the budget and rate limits you set. When they hit the limit, LiteLLM returns a 429 automatically.

Test the Virtual Key

curl -X POST 'http://0.0.0.0:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-abc123...' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Test virtual key"}]
  }'

Step 5: Add Load Balancing and Fallbacks

If you run multiple deployments of the same model, LiteLLM can load-balance across them and fall back if one fails.

Update your config with multiple deployments and routing settings:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 200
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-deployment
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      rpm: 300

litellm_settings:
  num_retries: 3
  request_timeout: 30
  fallbacks: [{"gpt-4o": ["claude-sonnet"]}]

router_settings:
  routing_strategy: simple-shuffle

With this config, requests to gpt-4o are split between OpenAI and Azure based on their RPM weights. If both fail after retries, LiteLLM falls back to claude-sonnet. Your application never sees the complexity.

Step 6: Enable Observability and Logging

You cannot manage what you cannot see. LiteLLM Proxy integrates with popular observability tools out of the box.

Send Logs to Langfuse

litellm_settings:
  success_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: pk-...
  LANGFUSE_SECRET_KEY: sk-...
  LANGFUSE_HOST: https://cloud.langfuse.com

Enable Prometheus Metrics

LiteLLM exposes a /metrics endpoint compatible with Prometheus. Add this to your docker-compose.yml or scrape it directly:

# Prometheus scrape config
scrape_configs:
  - job_name: 'litellm'
    static_configs:
      - targets: ['litellm:4000']
    metrics_path: /metrics

Key metrics include request latency, token usage per model, error rates, and spend per virtual key. For more on production monitoring patterns with LiteLLM, check LiteLLM Setup Proxy: The Complete Developer Guide to a Unified LLM Gateway.

Tips and Troubleshooting

Model Not Found Errors

If you get Model not in model_list, check that the model_name in your request exactly matches an entry in litellm_config.yaml. LiteLLM does not do fuzzy matching.

Database Connection Failures

If the proxy exits with All connection attempts failed, verify your database_url and ensure the Postgres user has CREATE DATABASE privileges. On managed databases, you may need to pre-create the litellm database.

SSL Certificate Errors

Behind a corporate proxy or self-signed cert? Add this to your config:

litellm_settings:
  ssl_verify: false

Rate Limits Not Enforced

RPM and TPM limits only work when a database is configured. If you are running the proxy without database_url, virtual keys and rate limiting are disabled. Use the ghcr.io/berriai/litellm-database image or set database_url explicitly.

Keep Secrets Out of Config Files

Never commit API keys to version control. Always use os.environ/VAR_NAME syntax in litellm_config.yaml and inject secrets via Docker env vars or a secret manager.

Next Steps

You now have a working LiteLLM setup proxy that unifies every LLM provider behind a single OpenAI-compatible API. Your teams can swap models, enforce budgets, and monitor usage without rewriting a line of application code.

If you are scaling this to production across multiple environments or need help with SSO, RBAC, or custom routing policies, get in touch with our team. We help enterprises deploy and operate unified LLM gateways at scale.