LiteLLM as Your LLM Gateway: Budgets, Fallbacks and a Safe Setup

1. The Problem: Provider Keys Everywhere, Visibility Nowhere

Calling providers directly feels simple until the third integration. Each service holds its own copy of the keys, each SDK has its own error format and retry logic, and finance gets one invoice per provider with no breakdown by feature or team. When one provider has a bad hour, every service handles it differently, usually by failing.

A gateway fixes this at the network layer. Clients speak the OpenAI API to one base URL; the proxy maps model names to providers, enforces who may spend what, retries and falls back when a provider fails, and writes every request and its cost to a database. LiteLLM is the open-source option most teams reach for, and the rest of this guide builds it step by step.

2. Minimum Concepts Before Touching Code

Six ideas cover 90% of what you need. Get these straight and the config below reads like plain English.

3. Step 1 — Run the Proxy: Pinned Image + config.yaml

Start with the config. Aliases are the contract with your clients, so name them by role (fast, smart), not by vendor. Provider keys are read from the environment with os.environ/..., never written in the file. Virtual keys, budgets and spend logs need a database, so the proxy also needs DATABASE_URL.

litellm_config.yaml — aliases, master key, log hygiene

# litellm_config.yaml
model_list:
  - model_name: fast            # alias clients request, named by role
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: fast-backup     # same role, different provider
    litellm_params:
      model: openai/gpt-5-mini  # swap for the IDs your providers list today
      api_key: os.environ/OPENAI_API_KEY
  - model_name: smart
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  disable_error_logs: true
  maximum_spend_logs_retention_period: "30d"
  maximum_spend_logs_retention_interval: "1d"

Now run it. Pin an exact image tag instead of latest (1.105.1 is the latest stable on the releases page as I write; 1.106 release candidates exist, so check when you read this) and bind the port to localhost behind your own TLS reverse proxy. Generate the master key and the salt key once and keep them in a secret manager.

Run it — master key, salt key, pinned image

export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
export LITELLM_SALT_KEY="sk-$(openssl rand -hex 32)"  # keep it; never change after adding models
export DATABASE_URL="postgresql://litellm:<password>@<db-host>:5432/litellm"

docker run -d --name litellm \
  -p 127.0.0.1:4000:4000 \
  -v "$(pwd)/litellm_config.yaml:/app/config.yaml:ro" \
  -e LITELLM_MASTER_KEY -e LITELLM_SALT_KEY -e DATABASE_URL \
  -e LITELLM_MODE=PRODUCTION \
  -e ANTHROPIC_API_KEY -e OPENAI_API_KEY \
  ghcr.io/berriai/litellm:v1.105.1 \
  --config /app/config.yaml

4. Step 2 — Teams, Virtual Keys, Budgets and Rate Limits

With the proxy up, use the master key to create structure and then never again from an app. A team carries the shared budget; keys under it carry tighter, per-workload limits. Every field below is documented on the virtual keys and budgets pages.

curl — a team and a scoped key

# 1) team: shared budget + defaults (the response includes a team_id)
curl -s http://localhost:4000/team/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_alias": "support-bot",
    "max_budget": 50,
    "budget_duration": "30d",
    "rpm_limit": 60,
    "models": ["fast", "fast-backup"]
  }'

# 2) key for ONE workload inside that team (the response includes the sk-... key)
curl -s http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_id": "<team_id from step 1>",
    "models": ["fast", "fast-backup"],
    "max_budget": 10,
    "budget_duration": "7d",
    "rpm_limit": 30,
    "tpm_limit": 100000,
    "duration": "30d",
    "metadata": {"owner": "support-team"}
  }'

Clients then use the plain OpenAI SDK. Point base_url at the gateway, pass the virtual key as api_key and tag requests with metadata.tags so the spend logs can be sliced by feature.

Python — OpenAI SDK against the gateway, with spend tags

from openai import OpenAI

client = OpenAI(
    api_key="sk-<virtual-key>",          # the scoped key, never the master key
    base_url="http://localhost:4000",    # the gateway
)

resp = client.chat.completions.create(
    model="fast",                        # alias from model_list
    messages=[{"role": "user", "content": "Summarise this ticket in one line."}],
    extra_body={"metadata": {"tags": ["feature:ticket-summary"]}},
)
print(resp.choices[0].message.content)

5. Step 3 — Fallbacks Across Providers and Spend Logs

Reliability lives in router_settings. num_retries retries the same model group; if it still fails, fallbacks sends the request to the next group, here a different provider. After allowed_fails failures a deployment goes on cooldown for cooldown_time seconds, so you stop hammering a dead endpoint. The numbers are placeholders: size them against your own traffic.

router_settings — retries, cooldown, cross-provider fallback

router_settings:
  routing_strategy: simple-shuffle   # default, recommended for production
  num_retries: 2                     # per model group, before falling back
  allowed_fails: 3                   # failures per minute before cooldown
  cooldown_time: 30                  # seconds out of rotation
  fallbacks: [{"fast": ["fast-backup"]}]

Two related keys: context_window_fallbacks (it needs enable_pre_call_checks: true) re-routes a prompt that does not fit the model’s context, and content_policy_fallbacks handles provider-side content refusals. You can also send fallbacks per request in the body, or block them for one key with metadata.disable_fallbacks. Then read the money back:

curl — read the spend back

# per-request rows, paginated (page_size max 1000)
curl -s "http://localhost:4000/spend/logs/v2?start_date=2026-10-07%2000:00:00&end_date=2026-10-14%2023:59:59&page=1&page_size=100" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

# spend grouped by team
curl -s "http://localhost:4000/global/spend/report?start_date=2026-10-01&end_date=2026-10-14&group_by=team" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

# one key: spend so far and its limits
curl -s "http://localhost:4000/key/info?key=sk-<virtual-key>" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

6. Run It Safely: What the March 2026 Incident Teaches

On March 24, 2026, litellm 1.82.7 and 1.82.8 were published to PyPI carrying a credential stealer. LiteLLM’s own security update, NetSPI, Snyk and the GitHub issue thread agree on the core: two malicious releases, a payload that gathered environment variables, SSH keys, cloud and Kubernetes credentials and posted them to models.litellm.cloud (not an official domain), and a litellm_init.pth file in 1.82.8 that runs on every Python start, without importing litellm.

LiteLLM reports that the official proxy Docker image and 1.82.6 or earlier were not affected, that the compromise started with a Trivy dependency in its CI pipeline, and that a clean 1.83.0 shipped on March 30 through a rebuilt pipeline. Sources differ on how long the packages were live (about 40 minutes per LiteLLM, roughly three hours per Snyk), so do not lean on the window: if either version touched a machine, assume exposure.

Verify before you run

# A) Container: verify the signature (key pinned to an immutable commit)
cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.105.1

# B) pip: exact pin + hashes for the whole dependency tree
#    requirements.in contains: litellm[proxy]==1.105.1
pip-compile --generate-hashes requirements.in -o requirements.txt
pip install --require-hashes -r requirements.txt

# C) Audit: was a bad build ever installed here?
pip show litellm | head -n 2
find / -name litellm_init.pth 2>/dev/null

7. Common Errors (All of Them Fixable in Minutes)

Almost every gateway incident I have read about falls into one of these buckets. The left column is what sane deployments do; the right column is what pages you at night.

Conclusion

You now have the full loop: a pinned LiteLLM proxy with role-based aliases, a team and scoped virtual keys with budgets and rate limits, cross-provider fallbacks with cooldowns, spend you can query by key, team and tag, and a checklist for running it after the March incident.

Do one thing this week: put a single internal service behind the gateway with its own key, a small budget and a fallback, then read its spend logs after a few days. SSO, Redis and Kubernetes can follow. I have not load-tested this setup, so size workers, pool limits and Redis with the production guide before real traffic.

Sources

• LiteLLM docs — Docker quick start (master key, salt key, database requirement)
• LiteLLM docs — Virtual keys (/key/generate, /key/info, rotation)
• LiteLLM docs — Budgets and rate limits (teams, users, keys)
• LiteLLM docs — Reliability (retries, fallbacks, cooldowns)
• LiteLLM docs — Spend tracking (/spend/logs, tags, reports)
• LiteLLM docs — Production best practices
• LiteLLM docs — Security best practices (Admin UI, image pinning)
• LiteLLM — Security update, March 2026 (affected versions, timeline, remediation)
• NetSPI — LiteLLM supply chain compromise
• Snyk — Poisoned security scanner backdooring litellm
• GitHub — Issue #24512: malicious litellm_init.pth in litellm 1.82.8
• GitHub — LiteLLM v1.105.1 release (cosign verification)
• pip — Secure installs (hash-checking mode)