LUMEN

Lightweight Unified Model ENdpoint

A universal, self-hostable LLM gateway written in Rust. One OpenAI-compatible endpoint in front of many providers - for chat, embeddings, reranking and typed decisions alike. It is designed to be light, fast and sovereign: a single static binary, zero telemetry, and prompts that are never logged by default.

LUMEN is an alternative to LiteLLM (Python, heavier) and OpenRouter (SaaS, not self-hostable). The gateway's own overhead is microseconds, off-network: the per-request CPU work it adds is ~3.2 µs median (resilience executor + OpenAI-surface (de)serialization), measured with cargo bench on Apple Silicon. Streaming forwards upstream bytes verbatim with no per-chunk re-serialization (see ADR 004). A full, reproducible head-to-head against LiteLLM under load ships in bench/; the honest numbers and the method are in docs/perf-baseline.md.

Full documentation: https://qdequele.github.io/lumen/ - guides per capability, operations (analytics, budgets, resilience), examples, and reference.

Contents

• Capabilities & API
• 5-minute quickstart
• Providers × capabilities
• Features
• Configuration
• Benchmarks
• Security

Capabilities & API

Method & path	What it does
POST /v1/chat/completions	Chat completions, OpenAI format, streaming SSE.
POST /v1/embeddings	Embeddings, OpenAI format.
POST /v1/rerank	Reranking, Cohere format (query, documents, top_n).
POST /v1/decisions	Typed decisions (calibrated probabilities) from TypeSafe, Perplexity, OpenAI, Cloudflare and Ollama models, in the OpenAI format (input, questions array) or the TypeSafe format (state, questions object), answered in the format received. See Decisions. Replaces POST /v1/systemone, removed in 0.7.0 (Migrating).
GET /v1/models	Lists configured models with a capabilities array and "virtual": true/false (virtual models also carry an optional description; listed = false hides one), plus release_date / created when configured, to sort by release.
GET /v1/models/{id}	Retrieves one model (same object as the list entry); unknown id is a 404 (LM-2001).
GET /health	Liveness. No I/O, never touches the DB or providers.
GET /health/providers	Background provider-probe results (opt-in, see below).
GET /metrics	Prometheus exposition.
POST/GET/PUT/PATCH/DELETE /admin/*	Keys, budgets, budget webhooks, usage reporting & export, provider-key rotation, whole-document and granular (per-provider, per-section) config read/apply. Only mounted when auth is enabled. See Config source modes.

A foundation model id is owned entirely by you and may serve one or more of the four capabilities (chat, embed, rerank, decisions); a virtual model serves exactly one. The router resolves each request by (capability, model).

Vision (image input): POST /v1/chat/completions also accepts OpenAI's content-parts message shape (text + image_url parts) for any model whose config opts in with modalities = ["text", "image"] (default ["text"]). OpenAI-family kinds and vllm forward image parts verbatim; anthropic and google translate them. See docs/providers.md.

5-minute quickstart

Zero to a successful chat + embed + rerank request. For ready-made scenario configs (RAG, multi-tenant, fallback chains, …), see examples/.

1. Write a minimal config.toml

The ids below (gpt-4o, text-embedding-3-small, rerank-english) are the same ones used throughout config.example.toml. This minimal file needs an OpenAI key (chat + embeddings) and a Cohere key (rerank).

# config.toml - minimal quickstart config
[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]

2. Run it

Docker (the released image; sets LUMEN_SERVER__HOST=0.0.0.0 for you):

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  -e COHERE_API_KEY=... \
  ghcr.io/qdequele/lumen:latest

From source (needs a recent stable Rust toolchain):

export OPENAI_API_KEY=sk-...
export COHERE_API_KEY=...
cargo run -p server -- --config config.toml

The bundled config.example.toml also runs as-is (cargo run -p server -- --config config.example.toml); it additionally wires Anthropic/Ollama/Jina/Voyage/TEI and demonstrates fallbacks. Providers whose API-key env var is unset are only rejected when a request actually routes to them, so a partial set of keys is fine.

By default auth is off, so these requests need no Authorization header.

3. Chat completion

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello in one word."}]
  }'

Stream it by adding "stream": true - the response becomes text/event-stream with data: {…} frames and a terminal data: [DONE].

4. Embeddings

curl -s http://localhost:8080/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{
    "model": "text-embedding-3-small",
    "input": ["the quick brown fox", "a lazy dog"]
  }'

5. Rerank

curl -s http://localhost:8080/v1/rerank \
  -H 'content-type: application/json' \
  -d '{
    "model": "rerank-english",
    "query": "What is the capital of France?",
    "documents": ["Paris is the capital of France.", "Berlin is in Germany."],
    "top_n": 2
  }'

Results come back sorted by descending relevance_score. documents must be non-empty (an empty list is rejected with LM-2010).

Providers × capabilities

Twenty-seven provider kinds: sixteen native integrations plus eleven OpenAI-compatible hosts. The kind string is what you put in a [[providers]] block. Self-hosted kinds are keyless and require a base_url; hosted kinds read their API key from the env var named by api_key_env (bedrock uses AWS env credentials; for vertex_ai the api_key_env var holds a GCP service-account JSON).

kind	Chat	Embed	Rerank	Decisions	Auth	Notes
openai	✅	✅		✅	api_key_env	decisions: gpt-6-luna
mistral	✅	✅			api_key_env	OpenAI-style API, native module
anthropic	✅				api_key_env	bidirectional translation
google	✅	✅			api_key_env	Gemini Developer API
vertex_ai	✅	✅			GCP service account	Gemini on GCP; base_url = region
bedrock	✅	✅			AWS SigV4 (env creds)	Converse API
azure	✅	✅			api_key_env, base_url	deployment-routed URLs
cohere	✅	✅	✅		api_key_env	one model can serve chat, embed and rerank
jina		✅	✅		api_key_env
voyage		✅	✅		api_key_env
mixedbread			✅		api_key_env	mxbai-rerank-*
pinecone			✅		api_key_env	Api-Key header; reports units
nvidia			✅		keyless, base_url	NIM /v1/ranking; logit scores
tei		✅	✅		keyless, base_url	self-hosted (Text Embeddings Inference)
ollama	✅	✅		✅	keyless, base_url	self-hosted; chat via its OpenAI-compatible /v1; local decision models (Ollama 0.35+)
typesafe				✅	api_key_env	Jev and any TypeSafe-format vendor (decisions_path); input-only pricing
perplexity	✅			✅	api_key_env	pplx-decider-*; one block can mix chat and decision models
cloudflare	✅	✅	✅	✅	api_key_env, base_url	Workers AI; decisions: Clef

OpenAI-compatible hosts (chat + embed, reusing the OpenAI path with a built-in base URL): groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra, huggingface (HF Inference router), cloudflare (Workers AI - base_url carries your account id), and vllm (any self-hosted OpenAI-compatible server: vLLM, llama.cpp, LM Studio, …). Anything else that speaks the OpenAI format works via kind = "openai" + a base_url. Note that groq, deepseek, perplexity and xai serve chat only (no upstream embeddings API): declaring embed on them is rejected at config load, unless a custom base_url fronts the host with an embedding-capable proxy. Two of the OpenAI-compatible hosts additionally serve rerank natively: together (LlamaRank) and cloudflare (BAAI bge-reranker-*, through Workers AI's native /ai/run/{model} endpoint rather than the OpenAI path). perplexity and cloudflare also appear in the table above for their decision models. Reranking through a decision model is a virtual model remap (any decisions kind, see Reranking). Full capability table in docs/providers.md.

Per-provider setup (env var, base_url, defaults, batch limits) is in docs/providers.md.

Features

Each area is summarized here; the linked docs and ADRs carry the detail.

Auth, keys & hard budgets

Off by default - with [auth].enabled = false the gateway is an open proxy with no database at all. When enabled (requires LUMEN_MASTER_KEY, 64 hex chars), it adds virtual keys, hard budgets and RPM/TPM quotas, all enforced in memory before any upstream call, so a rejected request never spends. See Keys, quotas & budgets.

Budget webhooks (ADR 011)

Absent by default - with no webhook configured the gateway calls nothing but its providers, and exports no webhook metric. Enable it and a billing backend gets pushed budget.threshold, budget.exhausted and key-lifecycle events, so it can auto-recharge through POST /admin/keys/{id}/grant before a customer hits a 402 instead of polling for it. Detection is a compare on the budget settle that already happens per request; delivery is a bounded queue and a background sender with HMAC-signed, idempotent, at-least-once semantics. Payloads carry accounting facts only - never a key, never prompt or response content.

Configure it declaratively in [webhooks], or entirely through the admin API (GET/PUT/DELETE /admin/webhooks, plus PUT/DELETE /admin/webhooks/signing-key, which seal and forget the HMAC secret at rest) - every field is editable at runtime, and API-written settings win over the file. See Outbound webhooks for budget events.

Resilience

Survives flaky upstreams without becoming flaky itself: retries with exponential backoff + jitter (retryable failures only, never a client 4xx), fallback through virtual models, a per-(provider, model) circuit breaker, and per-phase timeouts. Optional background health checks publish GET /health/providers. See Resilience tuning.

Virtual models (ADR 014)

Public ids that carry routing logic on top of your foundation models: fallback, weighted split, conditional routing (switch on budget group, request metadata and request shape), presets (a stored system prompt and sampling defaults), and Jev as a reranker, all declared in [[virtual_models]] and hot-reloadable. Responses report the path taken in x-lumen-route. See docs/virtual-models.md.

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]

An older config that uses per-model fallbacks is migrated with lumen config migrate (--dry-run previews it).

Observability & token accounting (ADR 003)

Every request of every capability produces a token count - upstream usage when reported, otherwise a local byte-heuristic estimate flagged "estimated": true. Never a silent zero. Surfaced three ways: in the response body, on /metrics, and (when auth is on) in the usage_log table. See Token accounting & cost and Metrics & dashboards.

Config hot reload

SIGHUP, a file-watch, an admin config write, or an admin provider-key rotation triggers a reload: the new config is validated, then the provider registry, price table, resilience policy, the runtime-safe [auth] knobs, the webhook delivery policy, the image-fetch policy and the tokenizer are atomically swapped; in-flight requests are unaffected. An invalid config is rejected - the old config keeps serving. Boot-layer settings (bind address and the rest of [server], log format, telemetry labels, the auth switch and database path, usage-log channel sizing) need a restart. See Deployment.

--check-config

lumen --check-config [--config <PATH>] validates a config file the same way the server does at boot (parsing, semantic validation and provider registry construction) and exits: 0 if valid, non-zero otherwise. It binds no listener, opens no database, and contacts no provider, so it is safe to run in a CI or deploy pipeline ahead of a real boot. See Installation.

Security headers

Every response carries a standard set of security headers. TLS is intentionally left to a terminating reverse proxy. See Deployment.

Configuration

Everything is one TOML file (plus LUMEN_* env overrides, using __ for nesting, e.g. LUMEN_SERVER__PORT=9090). The exhaustively commented reference is config.example.toml; the full walkthrough lives in the book's Configuration basics.

API keys are never written in the config - a provider references the name of the env var that holds its key.

Meilisearch Lab

A gateway can report billing usage to the Meilisearch Lab control plane as one of its instances. Three environment variables, the same ones Scrapix and glutony read, switch it on:

LAB_URL=https://lab.meilisearch.com      # the Lab's base URL
LAB_INSTANCE_ID=<uuid>                   # minted by the Lab for this gateway
LAB_INSTANCE_SECRET=<64 hex>             # shown once by the Lab, never logged

Everything else (what an event carries, per-account scoping of /admin/*, delivery guarantees, metrics and the runbook) is in Lab integration (source: docs/operations/lab-integration.md). Without these variables (or a [usage_events] block) the gateway makes no call to the Lab.

Benchmarks

From docs/perf-baseline.md (Apple Silicon, release profile, rustc 1.97.0):

Measure	Value
Executor around an instant provider (executor_overhead_chat)	~1.21 µs median
Parse a chat request (json_request_deserialize)	~1.34 µs median
Serialize a chat response (json_response_serialize)	~0.60 µs median
Total added CPU per non-streaming chat request	~3.2 µs median
Idle RSS (release binary, one provider)	~8.8 MB

Reproduce the in-process numbers:

cargo bench -p server --bench gateway_overhead

The full loaded head-to-head against LiteLLM (added latency p50/p99, RAM, req/s) is a one-command Docker + k6 harness - see bench/README.md. A recorded baseline is committed under bench/results/: in that run LUMEN added ~2.5 ms at p50 over the direct-to-mock baseline, while LiteLLM's p50 reached ~324 ms at roughly 140x the RAM, and LUMEN sustained about 25x LiteLLM's throughput. The absolute numbers come from a noisy shared host - treat the relative comparison as the meaningful part, and re-run bench/run.sh on your own hardware before making capacity decisions. Method and caveats: docs/perf-baseline.md.

Security

LUMEN runs inside your own trust boundary. Provider keys are referenced by env-var name (or encrypted at rest under LUMEN_MASTER_KEY), never logged, never returned in errors. Prompts and responses are never logged by default. Vulnerability reporting and the full security model are in SECURITY.md.

Reference

• Error codes: docs/errors.md
• Provider setup: docs/providers.md
• Performance: docs/perf-baseline.md
• Architecture decisions: docs/adr/
• Changelog: CHANGELOG.md
• Examples: examples/
• Documentation site: https://qdequele.github.io/lumen/

License

Apache-2.0.