LLM Workbench

LLM Workbench is a browser shell for local AI service consoles and workflow playgrounds.

The app has two kinds of surfaces:

• stable consoles for local services such as llm-pool, tts-pool, image-pool, and translation-services
• playgrounds for designing and testing workflows before they become a service or a permanent console

The backend is a FastAPI app. It serves the frontend from static/, proxies service APIs on the same origin, and owns temporary workbench session state.

Index

• Workbench Areas
• Repository Role
• Related Services
• Code Map
• Backend Contract
• Runtime Model
• Configuration
• Development
• Verification
• License

Workbench Areas

Permanent Service Consoles

These surfaces are expected to stay. Their UI can change.

LLM Pool

• Models lists, loads, unloads, and inspects llm-pool models.
• Text generation runs one-off prompts against loaded models.
• Chat runs multi-turn chat when the selected model supports it. Models without multi-turn support use a flattened prompt fallback.

TTS Pool

• Models lists, loads, unloads, and inspects tts-pool models.

Image Pool

• Models lists, loads, unloads, and inspects image-pool models.
• Image generation runs image generation and image edit requests.
• Train manages image datasets, captions images, configures LoRA training, starts training runs, and monitors progress.

Translation Services

• Image translation submits image translation requests and inspects artifacts.
• Image translation regression re-runs image translation fixtures and re-baselines approved changes (capture happens on the Image translation view).
• PDF translation submits PDF translation requests, previews the translated document, and captures a completed run as a regression fixture.
• PDF translation regression re-runs document fixtures (replay + per-page diffs) and shows the benchmark-on-replay score against the accepted baseline.
• PDF testing shows the PDF benchmark comparison matrix and imports external translations for scoring.
• Prompt Library manages prompts used by translation services.

The same pattern can host later service consoles. One likely example is TTS Services for shared voice libraries and stable generated voices.

Playground And Incubation

These surfaces are useful, but may move or be extracted.

• Realtime Translation / Replay & Translate replays .pc transcript streams and inspects translation behavior.
• Realtime TTS / Replay & Speak replays committed transcript segments through tts-pool.

Developer And Design Helpers

• Icons / Colors is an internal helper for comparing icon and color options.

Repository Role

This repo owns:

• the FastAPI workbench app
• the same-origin frontend shell
• service proxy routes for local AI services
• workflow UI modules under static/src/workflows/
• replay session orchestration for current playgrounds
• prompt-library storage used by the realtime translation playground
• workbench-side image training dataset management

This repo does not own:

• LLM inference or model lifecycle internals; llm-pool owns those
• image model inference and training execution; image-pool owns those
• TTS model inference; tts-pool owns that
• image translation execution; translation-services owns that
• reusable realtime translation engine behavior; realtime-translation-engine owns that

Related Services

• llm-pool provides text model inference and runtime model administration.
• image-pool provides image generation, image editing, image model administration, and LoRA training execution.
• tts-pool provides TTS model inference and runtime model administration.
• translation-services provides image translation, prompt storage, artifacts, and regression fixtures.
• realtime-translation-engine provides reusable translation state and dispatch behavior.
• omniscripta can produce .pc replay files from realtime transcript streams.

Code Map

Backend

• app/main.py creates the FastAPI app, mounts /api, exposes websocket routes, and serves the frontend.
• app/router.py wires backend route groups.
• app/llm_pool/ proxies llm-pool model and admin APIs.
• app/tts_pool/ proxies tts-pool model and admin APIs.
• app/image_pool/ proxies image-pool model, LoRA, generation, editing, and training APIs. It also manages workbench-side training datasets.
• app/translation_services/ proxies the translation-services API.
• app/prompt_testing/ implements text generation and chat requests through llm-pool.
• app/realtime_translation/ contains the replay playground and prompt-library endpoints.
• app/realtime_tts/ contains the TTS replay playground.
• promptlib/ contains prompt-library storage helpers.

Frontend

• static/index.html is the browser entrypoint.
• static/app.js wires the shell: sidebar, routing, theme, and view lifecycle.
• app/plugins.py is the plugin registry: which sidebar categories and views exist, and which retired route names still resolve. It is what /plugins.js is generated from. The routers belong to the core: app/router.py mounts all of them, always, so switching a category off never removes an endpoint.
• static/src/plugins/registry.js turns the generated list into the sidebar, the route table and the lazy view loader.
• static/foundation/spa-foundation/ contains the shared shell, routing, modal, and sidebar helpers.
• static/src/shared/api/ contains the request plumbing every client shares, plus the four calls that more than one plugin uses.
• static/src/plugins/<category-id>/api.js is the API client of one plugin, with the websocket class of its own views where it has one. A view imports the client of its own plugin and, where needed, the shared core client — never another plugin's.
• static/src/workflows/ contains the workflow views.
• static/css/ contains shared application styling.

Support Files

• config/settings.json contains committed defaults.
• config/local.json is ignored and can override local settings.
• data/realtime_translation/sample/ contains sample .pc replay files.
• docs/ contains design notes and working observations.
• deploy/systemd/ contains example local service wiring.
• tests/ contains backend tests.

Backend Contract

The frontend calls the workbench backend on the same origin. The workbench then calls the configured local service.

Main endpoint families:

Endpoint family	Role
/api/models*	llm-pool models, admin state, load/unload, and GPU memory.
/api/text-generation/run	Single-turn text generation through llm-pool.
/api/chat/run	Chat through llm-pool.
/api/prompts/test-translation	Prompt rendering and test translation for the replay playground.
/api/tts-pool/models*	tts-pool models, admin state, load/unload, and GPU memory.
/api/image-pool/models*	image-pool models, admin state, load/unload, and GPU memory.
/api/image-pool/images/*	Image generation and image edit requests through image-pool.
/api/image-pool/loras	LoRA discovery for image generation.
/api/image-pool/training/*	Dataset management, captioning, training start/stop, and training status.
/api/translation/*	translation-services requests, prompts, artifacts, and regression fixtures.
/api/pdf-translation/*	translation-services PDF translation requests and document artifacts.
/api/pdf-benchmark/*	translation-services PDF benchmark results, testset listing, and scoring runs.
/api/pdf-regression/*	translation-services PDF document-regression fixtures: capture, replay, accept, artifacts.
/api/replay*	Realtime translation replay sessions.
/api/realtime-tts/replay*	Realtime TTS replay sessions and audio artifacts.
/ws/replay/{session_id}	Translation replay websocket updates.
/ws/replay-speak/{session_id}	TTS replay websocket updates.

Runtime Model

The workbench is a single FastAPI process with a static frontend.

The backend owns workbench state while a session is active. Replay state lives in memory. Image training datasets and generated training artifacts live under data/image_pool/, which is ignored by git.

The backend does not load AI models directly. Model inference and model lifecycle are delegated to local pool or service processes.

The sidebar comes from app/plugins.py: the registry names each plugin's views, and FastAPI serves the enabled ones as a generated /plugins.js that static/index.html loads before app.js. A view is loaded on first activation instead of at startup.

A plugin is a menu entry and nothing more. app/router.py mounts every router and app/main.py registers both websockets, whatever the settings say, so switching a category off changes the sidebar and nothing else. An Image Pool-only workbench therefore still reaches the LLM Pool service for the model list that five views outside LLM Pool use.

Configuration

Committed defaults live in:

config/settings.json

Machine-local overrides can be placed in:

config/local.json

config/local.json is ignored by git.

Which sidebar categories the workbench shows:

{
  "plugins": {
    "enabled": ["image-pool"]
  }
}

Leave plugins.enabled out and every category is in the menu, so a category added to the registry appears by itself. Name the list and only those ids are in the menu, in registry order. The ids are realtime-translation, realtime-tts, llm-pool, tts-pool, image-pool, video-pool, translation-services, and developer (the Icons item). An unknown id, an empty list, a plugins section that is not an object and a settings file whose root is not an object are all refused instead of quietly producing a different menu. "enabled": null in local.json turns every category on: that is the only way a local file widens the menu instead of narrowing it. Only whole categories switch, and a page reload is enough: the file is read per request. LLM_WORKBENCH_SETTINGS_FILE points the workbench at another file altogether, with local.json looked up beside it; that is how a deployment keeps its settings outside the repo, and how tests/browser/check_plugin_registry.py runs against the shipped defaults. A bookmark to a view of a category that is off falls back to the first view of the menu.

Configured service connections:

Settings key	Default
llm_pool.base_url	http://127.0.0.1:8011
tts_pool.base_url	http://127.0.0.1:8020
image_pool.base_url	http://127.0.0.1:8013
translation_services.base_url	http://127.0.0.1:8030

Environment overrides:

Variable	Overrides
LLM_RESPONSES_API_BASE_URL	llm_pool.base_url
TTS_POOL_API_BASE_URL	tts_pool.base_url
IMAGE_POOL_API_BASE_URL	image_pool.base_url
TRANSLATION_SERVICES_API_BASE_URL	translation_services.base_url
LLM_WORKBENCH_SETTINGS_FILE	which settings file is read, instead of config/settings.json

Replay defaults also live under replay in config/settings.json.

Development

Create an environment and install the workbench:

python3 -m venv .venv
./.venv/bin/python -m pip install -e .

Install the realtime translation engine when working on replay translation:

./.venv/bin/python -m pip install -e ../realtime-translation-engine

Install test tooling if the environment does not already have it:

./.venv/bin/python -m pip install pytest

Run the backend:

./.venv/bin/python -m uvicorn app.main:app --host 127.0.0.1 --port 8000

Open:

http://127.0.0.1:8000/

The CLI entrypoint is also available:

./.venv/bin/python -m app

Static files are served with Cache-Control: no-cache, so an ordinary reload picks up frontend edits without clearing anything. There is no build step and no asset fingerprinting, so a browser is never told by URL that a module changed; revalidation is what keeps that workable.

Verification

Run the Python tests:

./.venv/bin/python -m pytest tests

The tests are mostly unittest-style tests and can also be run with:

./.venv/bin/python -m unittest discover -s tests

There is no JavaScript build step. The frontend plugin contract does have a test suite, which needs the Node node --test runner; Node is not a declared project dependency.

node --test 'tests/js/**/*.test.mjs'

It resolves every view module and factory named by app/plugins.py, and checks that every icon is in the sprite. It needs the venv: it reads the plugin list from Python so the two never drift. Individual ES modules can be syntax-checked with node --input-type=module --check.

The layout-metrics inspector behind the PDF translation Render panel has its own Node test, which needs neither a server nor a browser:

node tests/omnidoc_layout_metrics_ui.mjs

The plugin loader's browser behaviour has its own script. It starts the workbench on a free port itself, drives it with Playwright, and stops it again, so it needs the venv (uvicorn) and a Playwright Chromium build.

./.venv/bin/python tests/browser/check_plugin_registry.py

It walks every route and alias and covers the paths that only exist in the browser: the loading placeholder, the error panel for a view that cannot be fetched, the retry after such a failure, and a menu with a single category in it. It runs the workbench against the shipped config/settings.json, so a machine that switched categories off in config/local.json does not hide routes from it.

License

Licensed under the Apache License, Version 2.0. See LICENSE.