LiteLLM Setup Proxy: The Complete Developer Guide to a Unified LLM Gateway

Build a production-ready unified LLM gateway that routes requests to 100+ providers with a single OpenAI-compatible API.

What This Guide Covers

If your team is juggling API keys for OpenAI, Azure, Anthropic, and a self-hosted model, you're not alone. Most engineering teams end up with fragmented LLM integrations—each provider with its own SDK, auth pattern, and response format. That's where a LiteLLM setup proxy changes the game.

By the end of this guide, you'll have a working unified LLM gateway that can route chat, completion, and embedding requests to any provider from a single endpoint. No more rewriting client code every time you switch models. For a broader overview of why this matters, see our post on LiteLLM Setup Proxy: One Gateway to Rule Every LLM in Your Stack.

Prerequisites

Before you start, make sure you have:

Python 3.9+ or Docker installed
API keys for at least one provider (OpenAI, Azure, Anthropic, or AWS Bedrock)
A working terminal and curl for testing
Optional: uv for faster Python package management

If you already have a working LiteLLM setup and want a deeper dive into architecture, check out LiteLLM Setup Proxy: A Complete Developer Guide to Building a Unified LLM Gateway.

Step 1: Install LiteLLM Proxy

LiteLLM gives you two ways to run the proxy: via the CLI or Docker. Pick the one that fits your stack.

Option A: CLI Install (Recommended for Local Dev)

The fastest way to get started is with the install script or uv:

# Install via the official script
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh

# Or install with uv (faster, isolated)
uv tool install 'litellm[proxy]'

Verify the install:

litellm --version

Option B: Docker (Recommended for Production)

For production or team environments, Docker keeps dependencies clean:

docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --config /app/config.yaml --detailed_debug

Both methods expose the same OpenAI-compatible API on http://0.0.0.0:4000.

Step 2: Create Your Config

The config file is where the magic happens. It defines your model aliases, provider credentials, and routing behavior. LiteLLM uses a YAML file—usually named litellm_config.yaml—to map user-facing model names to actual provider deployments.

Basic Config with OpenAI and Azure

Here's a minimal config that exposes one alias for each provider:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: "os.environ/OPENAI_API_KEY"
  - model_name: gpt-4o-azure
    litellm_params:
      model: azure/my-azure-deployment
      api_base: "os.environ/AZURE_API_BASE"
      api_key: "os.environ/AZURE_API_KEY"
      api_version: "2025-01-01-preview"
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: "os.environ/ANTHROPIC_API_KEY"
  - model_name: gemini-pro
    litellm_params:
      model: vertex_ai/gemini-1.5-pro
      vertex_project: "os.environ/GCP_PROJECT"
      vertex_location: us-central1

general_settings:
  master_key: sk-1234  # admin key for proxy access

Key rules:

model_name is what your clients send in the model field.
litellm_params.model is the internal provider string LiteLLM uses.
Use os.environ/VAR_NAME to keep secrets out of the config file.

Adding a Self-Hosted Model (vLLM or Ollama)

You can route to local or self-hosted models the same way:

  - model_name: local-llama
    litellm_params:
      model: openai/llama-3.1-8b
      api_base: http://localhost:8000/v1
      api_key: none

The openai/ prefix tells LiteLLM the endpoint is OpenAI-compatible, which covers vLLM, TGI, and Ollama.

Step 3: Start the Proxy

With your config saved, start the server:

CLI

litellm --config litellm_config.yaml

Docker

docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e AZURE_API_KEY=$AZURE_API_KEY \
  -e AZURE_API_BASE=$AZURE_API_BASE \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --config /app/config.yaml

You should see a log line confirming the loaded models:

LiteLLM: Proxy initialized with Config, Set models:

Step 4: Test Your Gateway

LiteLLM exposes a 100% OpenAI-compatible /chat/completions endpoint. Test it with curl:

curl -X POST 'http://0.0.0.0:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "What is a unified LLM gateway?"}
    ]
  }'

Switch the model field to claude-sonnet or gemini-pro and the request format stays identical. That's the whole point.

Test with the OpenAI SDK

Your existing code barely needs to change:

from openai import OpenAI

client = OpenAI(
    base_url="http://0.0.0.0:4000",
    api_key="sk-1234"
)

response = client.chat.completions.create(
    model="claude-sonnet",
    messages=[{"role": "user", "content": "Explain load balancing"}]
)
print(response.choices[0].message.content)

LangChain, LlamaIndex, and the Anthropic SDK work the same way—just point them at the proxy URL.

Step 5: Load Balancing and Fallbacks

Production gateways need redundancy. LiteLLM supports multiple deployments under the same model alias with automatic load balancing.

Define Multiple Deployments

Add two entries with the same model_name and set rpm or tpm limits:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-eu
      api_base: https://my-endpoint-eu.openai.azure.com/
      api_key: "os.environ/AZURE_API_KEY_EU"
      rpm: 6
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-ca
      api_base: https://my-endpoint-ca.openai.azure.com/
      api_key: "os.environ/AZURE_API_KEY_CA"
      rpm: 6

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30

litellm_settings:
  fallbacks: [{"gpt-4o": ["claude-sonnet"]}]
  context_window_fallbacks: [{"gpt-4o": ["gpt-4o-64k"]}]

Available routing strategies:

simple-shuffle – weighted random based on rpm/tpm
least-busy – routes to the deployment with the fewest active requests
latency-based-routing – picks the fastest deployment based on recent response times
usage-based-routing – distributes by token usage

For a full walkthrough of gateway architecture, read LiteLLM Setup Proxy: One Gateway to Rule Every LLM in Your Stack.

Step 6: Authentication and Virtual Keys

In production, you don't hand out the master key to every client. LiteLLM lets you generate scoped virtual keys with spend tracking and rate limits.

Enable a Database

Virtual keys require a Postgres database. Use Docker Compose for a one-command setup, or point to an existing Postgres instance:

general_settings:
  master_key: sk-1234
  database_url: "postgresql://user:pass@localhost:5432/litellm"

Generate a Virtual Key

Call the proxy's key management endpoint:

curl -X POST 'http://0.0.0.0:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "duration": "7d",
    "models": ["gpt-4o", "claude-sonnet"],
    "max_budget": 10.00,
    "tpm_limit": 10000
  }'

The response gives you a new key scoped to those models with a $10 budget and 10k TPM limit. Pass it to your clients instead of the master key.

Tips, Troubleshooting, and Gotchas

1. Model Not Found Errors

If you get 404 model not found, check that the model_name in your request matches the alias in config.yaml exactly. LiteLLM does not do fuzzy matching.

2. Provider-Specific Params

Some providers need extra headers or parameters. Pass them in litellm_params:

  - model_name: gpt-4o-team1
    litellm_params:
      model: openai/gpt-4o
      api_key: "os.environ/OPENAI_API_KEY"
      extra_headers: {"AI-Resource-Group": "team1"}

3. Debugging Requests

Run the proxy with --detailed_debug to see full request/response traces. You can also set the env var:

export LITELLM_LOG=DEBUG
litellm --config litellm_config.yaml --detailed_debug

4. Embedding Models

LiteLLM handles embeddings the same way. Add them to your model list:

  - model_name: text-embedding-3-small
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: "os.environ/OPENAI_API_KEY"

Call /embeddings with the same OpenAI-compatible format.

5. Fallbacks Not Triggering

Make sure fallbacks is defined under litellm_settings, not router_settings. Also verify the fallback model is defined in model_list.

6. Wildcard Models

If you want to expose every OpenAI model without listing them individually:

  - model_name: "*"
    litellm_params:
      model: "*"
      api_key: "os.environ/OPENAI_API_KEY"

This is useful for dev environments, but avoid it in production where you want explicit control.

Next Steps

You now have a working unified LLM gateway. From here, you can:

Add Redis for distributed state across multiple proxy instances
Connect Langfuse or Slack for observability and alerting
Set up budget alerts and team-based access controls
Deploy behind a reverse proxy with TLS termination

For a deeper architectural overview, see our companion guides:

Need Help Scaling This?

Setting up a proxy is straightforward. Running it at scale—with proper auth, load balancing, cost controls, and multi-region failover—is a different challenge. If your team needs a production-grade unified LLM gateway without the trial and error, reach out to us. We design, deploy, and operate LLM infrastructure for engineering teams that can't afford downtime.

in Guides

# AI Infrastructure Developer Guide LLM Gateway LiteLLM OpenAI Proxy

Vaultwarden Bitwarden Self-Host: Your Passwords, Your Server, Zero Trust Issues

Deploy Vaultwarden with Docker Compose, secure it with HTTPS, and take full control of your password vault—no subscription, no cloud lock-in.