embedding-batch-gateway

FastAPI-based OpenAI-compatible /v1/embeddings gateway. It accepts concurrent embedding requests, coalesces individual input items into upstream batch calls, retries transient upstream failures, and returns OpenAI-compatible responses to clients such as Dify.

Run locally

uv sync
cp config.example.yaml config.yaml
uv run uvicorn embedding_batch_gateway.app:create_app --factory --host 0.0.0.0 --port 18005 --workers 1

Use http://localhost:18005/v1 as the OpenAI-compatible API base URL and set the client API key to one of gateway.api_keys.

Configuration

Configuration is loaded from config.yaml when present. Environment variables override the file using nested __ names, for example:

EMBEDDING_GATEWAY__UPSTREAMS__0__API_KEY=sk-...
EMBEDDING_GATEWAY__GATEWAY__API_KEYS='["dev-gateway-key"]'

See config.example.yaml and .env.example.

Batch settings support global defaults plus per-upstream-model overrides. The override key is the actual upstream embedding model name after model_map resolution:

batch:
  max_items: 64
  max_tokens: 8192
  per_model:
    BAAI/bge-m3:
      max_items: 64
      max_tokens: 8192
      max_wait_ms: 50
    bge-large-zh-v1.5:
      max_items: 64
      max_tokens: 512
      max_wait_ms: 50

Docker

docker build -t embedding-batch-gateway:local .
docker compose -f docker-compose.example.yml up

The default command uses uvicorn --workers 1 because batching is in-process.

GHCR

GitHub Actions builds and publishes the image to GitHub Container Registry:

docker pull ghcr.io/<owner>/<repo>:latest

Pull requests build without publishing. Pushes to main/master, version tags like v0.1.0, and manual workflow runs publish image tags to GHCR.