running-the-proxy

Running the Proxy

liter-llm api is a drop-in OpenAI-compatible gateway: 22 REST endpoints that route to 143 providers, with multi-tenant virtual keys, rate limits, budgets, and cost tracking.

Start it

liter-llm api --config liter-llm-proxy.toml

The proxy auto-discovers liter-llm-proxy.toml in the current directory.

Configuration

[server]
host = "0.0.0.0"
port = 4000

[auth]
master_key = "${LITER_LLM_MASTER_KEY}"

[[virtual_keys]]
key = "sk-team-frontend"
models = ["openai/*", "anthropic/*"]
rpm = 60
tpm = 100000
budget = 50.0

[[providers]]
name = "openai"
api_key = "${OPENAI_API_KEY}"

[[providers]]
name = "anthropic"
api_key = "${ANTHROPIC_API_KEY}"

${ENV_VAR} interpolation keeps secrets out of the file.

Call it like OpenAI

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-team-frontend" \
  -d '{"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

Notes

• Virtual keys scope which models a caller may use and carry their own RPM/TPM and budget limits; the master_key administers them.
• The OpenAPI 3.1 spec is served at /openapi.json.
• Endpoints cover chat, embeddings, images, audio, moderations, files, batches, responses, and model listing.
• Docker: docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-key ghcr.io/xberg-io/liter-llm.