Railway and LiteLLM Breaches: A Hands-On Secrets Rotation Runbook

Timeline, October 2026 (UTC; vendor reports)

Railway: list every secret you keep there

Railway CLI 5.x (5.71.0 as I write; railway variables still works as an alias of railway variable). Print names, never values, for each service and environment. The CLI does not return sealed variables, so check those in the dashboard. Export GitHub and registry audit logs first: Railway’s own GitHub guide says to do it before you revoke anything.

# Names only, per service and environment: values never hit your terminal
railway login
railway link        # pick project, environment and service
railway variable list --service api --environment production --kv | cut -d= -f1

# Sealed variables are not returned by the CLI: list them from the dashboard
# Before revoking anything, export:
#   GitHub org: Settings > Logs > Audit log (JSON or CSV)
#   Registries (Docker Hub, GHCR): access-token and push history

LiteLLM: find 1.107.0 without starting Python

Run it on laptops, servers, self-hosted CI runners and images. Blocks 1–3 only read files; block 4 starts Python, so keep it for hosts that already came back clean. File names, paths, the environment marker and the wheel hashes come from the GitHub issue, LiteLLM’s incident post, Aikido and Datadog.

# 1. Lockfiles and requirements
grep -rn 'litellm==1.107.0' --include='*.txt' --include='*.in' --include='*.toml' .
grep -rn -A2 'name = "litellm"' --include='uv.lock' --include='poetry.lock' . | grep '1\.107\.0'
# SHA-256 prefixes of the seven malicious wheels
grep -rnE '7598727f209d|0aaf7ba346d4|462d0c9f67ca|668169d35b87|1bbce8caa94f|c2e32d0ce254|12d4ccc5c6f8' .

# 2. Installed environments, filesystem only
find / -type d -name 'litellm-1.107.0.dist-info' 2>/dev/null
find / -path '*-packages*' \( -name cook.pth -o -name cook.py -o -name sitecustomize.py \) 2>/dev/null
ls -la /tmp/.pc_once_* /tmp/remote_code_loader.lock 2>/dev/null
sudo grep -l PRIVCOLLECT_ACTIVE /proc/*/environ 2>/dev/null   # Linux: payload running now
# PowerShell: Get-ChildItem C:\ -Recurse -Filter cook.pth -ErrorAction SilentlyContinue

# 3. Images: the malicious index digest, then look inside without running
docker images --digests | grep -iE 'litellm|1bbcb1443bc9'
cid=$(docker create <IMAGE_ID>)   # local image ID, not the tag: tags were repointed
docker export "$cid" | tar -tf - | grep -E 'cook\.pth|litellm-1\.107\.0\.dist-info'
docker rm "$cid"
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}' | grep -i litellm

# 4. Only on hosts that passed 1-3 (these start Python)
pip show litellm | head -n 2
uv pip list | grep -i litellm

Railway: dual-key overlap from the CLI

This is Railway’s documented zero-downtime pattern. Give the service a healthcheck that actually uses the credential, so a bad value fails the new deploy while the old one keeps serving. Piping the key through stdin keeps it out of your shell history.

# 1. Create the new key at the provider. The old one still works.
# 2. Stage it without a restart (railway link chose the environment)
read -rs NEW_KEY
printf '%s' "$NEW_KEY" | railway variable set OPENAI_API_KEY --stdin \
  --service api --skip-deploys
# 3. Repeat for every variable that changes, then ship them in one deploy
railway redeploy --service api
# 4. Check that the new deployment is healthy and calls succeed
railway logs --service api --since 10m
# 5. Only now revoke the old key at the provider

# Several services? Keep the key once as a shared variable and reference it:
#   OPENAI_API_KEY=${{ shared.OPENAI_API_KEY }}

Cloud, GitHub and SSH from the command line

Same order as the list above. Placeholders in capitals are yours to fill in.

# AWS: new key, deactivate the old one, delete it after a quiet day
aws iam create-access-key --user-name APP_USER
aws iam update-access-key --user-name APP_USER --access-key-id OLD_KEY_ID --status Inactive
aws iam delete-access-key --user-name APP_USER --access-key-id OLD_KEY_ID
aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=OLD_KEY_ID \
  --start-time 2026-10-14T00:00:00Z   # per region: repeat with --region

# GCP service-account keys
gcloud iam service-accounts keys create new-key.json --iam-account=SA_EMAIL
gcloud iam service-accounts keys delete OLD_KEY_ID --iam-account=SA_EMAIL

# GitHub CLI: logout is local only
gh auth status                       # accounts, hosts and token scopes here
gh auth logout --hostname github.com --user YOUR_USER
#   then github.com/settings/applications > GitHub CLI > Revoke
gh auth login                        # fresh token, from a clean machine

# SSH: new key, register it, remove the old one everywhere
ssh-keygen -t ed25519 -C "laptop-2026-10" -f ~/.ssh/id_ed25519_2026_10
gh ssh-key add ~/.ssh/id_ed25519_2026_10.pub --title "laptop 2026-10"
gh ssh-key list
gh ssh-key delete OLD_SSH_KEY_ID

Pin with hashes, and know what they don’t do

Hashes guarantee that the file you install is the file you locked. They don’t stop you from locking a malicious new release: had you re-locked during those two hours, the hash would have been the stealer’s. That is what the cooldown is for.

# requirements.in: litellm[proxy]==1.106.4
uv pip compile requirements.in --generate-hashes -o requirements.txt
pip install --require-hashes -r requirements.txt

# uv projects: uv.lock already stores hashes; fail CI if it drifts
uv sync --locked

Dependency cooldowns: a two-hour attack meets a seven-day wait

litellm 1.107.0 was on PyPI for under two hours; the March releases were up for between 40 minutes and three hours, depending on the source. A minimum release age of one day would have skipped both. Check your tool version first: uv 0.9.17+ for relative durations, pip 26.1+ for day durations (26.0 takes timestamps), pnpm 10.16+, and npm with min-release-age (11.10+, per release coverage). Keep an escape hatch for urgent security fixes.

# uv >= 0.9.17: pyproject.toml
[tool.uv]
exclude-newer = "7 days"

# pip >= 26.1: ISO 8601, days only
pip install --uploaded-prior-to P7D -r requirements.txt

# pnpm >= 10.16 (pnpm 11 defaults to 1440): pnpm-workspace.yaml, minutes
minimumReleaseAge: 10080

# npm: .npmrc, days
min-release-age=7

# Renovate: renovate.json
{ "minimumReleaseAge": "7 days" }

# Dependabot: .github/dependabot.yml, inside each updates entry
    cooldown:
      default-days: 7

Docker: digest, not tag

Four tags, latest and main-stable included, pointed at the malicious image for up to two and a half hours. A digest can’t be repointed. Verify signatures with LiteLLM’s new key (commit e82a3c6, in use since 24 October); the old 0112e53 key fails on new releases. One caveat from Datadog: the malicious artifacts carried build provenance naming the lookalike fork as source, so if you check provenance, check which repository it names.

# Resolve the tag once, verify it, then pin the digest
docker buildx imagetools inspect ghcr.io/berriai/litellm:v1.106.4   # prints Digest: sha256:...
cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/e82a3c646908ba0a6ee0beddc4310d7a6cac4065/cosign.pub \
  ghcr.io/berriai/litellm:v1.106.4

# Dockerfile or compose: tag for humans, digest for the machine
FROM ghcr.io/berriai/litellm:v1.106.4@sha256:<DIGEST_YOU_VERIFIED>