Inference Lab - LLM & VLM Benchmarking Studio

A high-performance, full-stack benchmarking and evaluation playground for Large Language Models (LLMs) and Vision-Language Models (VLMs). Designed for OpenAI-compatible inference backends such as vLLM, SGLang, Ollama, and OpenAI API.

---

🖥️ Cross-Platform Native Desktop Support

Inference Lab is designed to run seamlessly across platforms:

• Windows: Uses native Edge WebView2 runtime out of the box for an ultra-lightweight, zero-overhead desktop window.
• Linux (Ubuntu / Debian): Integrates with system WebKitGTK (gir1.2-webkit2-4.1) for native window rendering without bundling heavy Chromium runtimes.
• Web Browser Fallback: Gracefully opens in your default browser if native window libraries are unavailable.

---

🚀 Key Features

⚡ Real-Time Streaming Performance Metrics

• Time to First Token (TTFT): Precise millisecond-level calculation of prompt processing and time to first generated token.
• Inter-Token Latency (ITL): Detailed latency distributions including Average, P50 (Median), and P95 (Spikes).
• Throughput & Decode Rate: Live generation rate in tokens/second.
• End-to-End Latency: Total request completion time and finish reasons, shown in plain language ("Finished normally", "Cut off at Max Tokens").
• Token counts from server usage: streaming calls request stream_options={"include_usage": true} and read the final usage block, so throughput and token totals reflect the backend's own counts rather than SSE chunk counts. When a backend reports no usage, figures are labelled as estimates (token_source / reasoning_source flags).
• Prompt accounting: prompt_tokens, cached_tokens (prefix-cache hits) and a derived prefill rate are captured alongside the output metrics.
• Plain-language dashboard: the playground groups stats into a summary strip plus Speed and Tokens cards, each with its unit in the value and a one-line caption; derived or partly-estimated figures say so.
• Optional warmup: 0–20 unmeasured requests can be sent before the timed run to absorb cold-start and graph-capture cost. Progress is reported beside the Execute button and the button switches to "Measuring..." when the timed request begins.

🧠 Reasoning & Thinking Model Support (DeepSeek-R1 / Qwen 2.5 & 3 / vLLM)

• vLLM enable_thinking Integration: Automatic parameter injection (chat_template_kwargs={"enable_thinking": True, "reasoning_effort": "low"} and the same keys at the top level of extra_body), built by one shared helper used by every run page.
• Reasoning Effort Selector: Playground, Batch and Tool Calling each expose low / medium / high / xhigh, sent per request and defaulting to low. No endpoint-wide default.
• Thinking Budget Control: Caps reasoning tokens per request (thinking_budget_tokens), exposed in the Playground, Batch and Tool Calling run options. The budget is sent as a top-level request field, which is where vLLM reads it — the same key nested in chat_template_kwargs is silently ignored. Verified against a live vLLM 0.33 Qwen3.5 endpoint: a top-level budget of 128 yields 127 reasoning tokens, while the nested form ran past 1300. Leaving the field blank sends no budget at all, so the backend's own default applies.
• Reasoning shares the response budget: reasoning and the answer draw on the same max_tokens, so a low Max Tokens with thinking on can consume the whole budget before any answer is emitted. Defaults are kept generous (2048 in Batch and Tool Calling), and a sample that hits the cap is labelled "Max Tokens ran out during reasoning; no answer was produced" instead of appearing as a blank row.
• Exact reasoning counts: completion_tokens_details.reasoning_tokens is read from the server when available, so the reasoning/answer split is measured rather than guessed from the stream.
• Streaming Reasoning Extraction: Supports both native delta.reasoning_content and real-time streaming tag parsing for thinking tags inside content streams.
• Interactive Reasoning Drawer: Unfolds live reasoning trace; token counts for the run are shown on the dashboard rather than duplicated in the trace header.

📄 Multimodal Vision (VLM) & Multi-Page PDF Processing

• Document & Image Ingestion: Direct upload support for PNG, JPEG, WEBP, and multi-page PDF documents.
• High-Fidelity Rendering via PyMuPDF: PDF pages are rendered to base64 images on demand. Pages are chosen with From / To range inputs plus Apply, All, and Reset, and a DPI selector (100–300, default 200).
• Page-Range Send: Only the pages inside the chosen range are rendered and sent, in ascending page order. Page ordering is a delivery guarantee and is not surfaced as a UI list.
• Preset Page Memory: Page numbers chosen in a preset's sample attachment are stored with the preset (sample_pages, sample_pdf_name) and restored in the picker grid when the preset is selected again.
• Editable Stored Ranges: Selecting a preset shows its stored pages in the same picker. The From/To controls stay live, so the range can be narrowed again (or restored with All) before sending; the request payload always follows the picker's current choice.
• Shared Picker Component: static/pdf_picker.js powers the main media panel and the preset modal on both the Playground and the Structured Decisions pages.
• URL & Data URI Linking: Benchmark images via remote public URLs or inline Base64 data strings. A single plain image is labelled "Attached image" with its source (file name or image URL); the PDF badge is shown only for actual PDF pages.

🎯 Structured Decisions Studio (/v1/systemone)

• Typed Probabilistic Evaluation: Test discrete decision queries evaluated directly via first-token read log probabilities (label_mass) without auto-regressive decoding overhead.
• Supported Decision Types: Full UI workflows for choice (categorical options), noul (yes/no affirmation probabilities), and score (ordered scale distributions and expected values).
• Prefix-Cache Aware: Designed to benchmark prefix-caching behavior where the state is cached and subsequent question probes evaluate with zero prompt re-computation.

📊 Concurrency & Batch Evaluation (Hugging Face Datasets)

• Zero-Dependency Dataset Loading: Stream and evaluate rows directly from Hugging Face Datasets API (e.g., openai/gsm8k, openai/openai_humaneval, allenai/ai2_arc).
• Configurable Concurrency: Run asynchronous parallel workers (1 to 32 parallel requests) and track aggregate TTFT, cluster throughput, and per-sample latency.
• Balanced setup layout: the page is three comparable cards — 1. Target & Model, 2. Dataset, 3. Run Options — rather than one card split into uneven columns. Model-call parameters stay in the Target & Model card while per-run behaviour (reasoning effort, thinking budget, warmup) sits with the Start button in Run Options. The same three-card layout is used by the Tool Calling page.
• Per-sample reasoning captured: each row keeps its reasoning trace (thinking_trace), marked with a small think badge, so a thinking run's output column is never silently empty.
• Truncation made visible: samples that hit Max Tokens report finish_reason and a note, the row shows "Max Tokens ran out during reasoning; no answer was produced" in place of a blank cell, and the summary counts affected samples (empty_outputs).
• Aggregate summary: p50/p95 TTFT, total prompt and cached tokens, warmup count and empty-output count are carried in the summary and its tooltips.

🧰 Tool Calling Benchmark (/tools)

• Function-call evaluation: streams scenarios from public Hugging Face function-calling datasets (Team-ACE/ToolACE, glaiveai/glaive-function-calling-v2, Mustafaege/qwen3.5-functioncalling-v1) and checks whether the model calls the expected tool with the expected arguments.
• Invocation matrix: every scenario renders as a coloured block — correct, incorrect, error or pending — and clicking one opens an inspector with the prompt, expected and actual tool call, reasoning trace, function arguments and raw model payload.
• Shared three-card setup: 1. Target & Model, 2. Dataset, 3. Run Options, matching the batch page. Reasoning effort, thinking budget, thinking toggle and warmup all live in Run Options, and the Start button sits there with an inline status line to its left.
• Reasoning-aware: scenarios are run with enable_thinking and the same top-level thinking budget as the other pages, so reasoning cost can be capped rather than left to the backend default.
• Warmup feedback: the backend's warmup notice is rendered next to the run button while warmup runs and clears when the first scenario completes.

🛠️ Persistent JSON-Backed Registries

• Inference Endpoints: Saved in data/endpoints.json with per-model capabilities (supports_thinking, is_vlm, context_window, api_key). Endpoints use UUID string ids.
• Edit In Place: Saved endpoints and presets can be reopened with Edit from their manage modal. The edit keeps the stored id, so saving overwrites the record instead of duplicating it, and Cancel returns the modal to create mode.
• Endpoint Fields on Save: the Context Window, Supports Thinking and Vision Model (VLM) fields in the endpoint popup apply to every model listed on that endpoint and are written on save.
• Context Window Is Metadata: context_window is stored and shown again in the edit popup, but it is not sent with requests and does not limit prompt size or image count. Only the backend server rejects an oversized request. Note that Max Tokens is the response budget (max_tokens), not a context limit.
• Custom Benchmark Presets: Save, edit and delete custom prompt templates in data/templates.json with optional sample image/document attachments, including the chosen PDF page numbers.
• Custom Decision Presets: Maintain custom state/question definitions in data/decision_templates.json directly from the UI, with the same edit and page-range support.
• Protected Built-Ins: Default system presets live in code and cannot be deleted. Editing one saves an override under the same id, and only its wording is editable — its sample media stays read-only. A new record whose generated id collides with an existing one is refused with a prompt to use Edit instead.

---

📁 Project Structure

LLM-Benchmarks/
├── app/
│   ├── main.py                     # FastAPI setup, route mounts & page handlers
│   ├── routers/
│   │   ├── benchmark.py            # Streaming benchmark & PDF conversion endpoints
│   │   ├── batch.py                # Concurrent dataset batch runner
│   │   ├── decisions.py            # /v1/systemone runner & decision presets API
│   │   ├── endpoints.py            # Endpoint CRUD registry API
│   │   ├── tools.py            	# Tool call benchmark runner
│   │   └── templates.py            # Benchmark prompt template store & router
│   ├── schemas/
│   │   ├── benchmark.py            # Benchmark request, metrics & payload schemas
│   │   ├── batch.py                # Batch request, sample result & aggregate schemas
│   │   ├── decision.py             # Structured decision schemas (choice, noul, score)
│   │   ├── endpoints.py            # Endpoint & model metadata schemas
│   │   ├── tools.py            	# Tool call benchmark schemas
│   │   └── template.py             # Prompt template schemas (incl. sample_pages)
│   ├── services/
│   │   ├── benchmark_service.py    # Streaming client, PyMuPDF engine & metrics
│   │   ├── batch_service.py        # Asynchronous batch executor with semaphore
│   │   ├── decision_template_store.py # JSON file store for decision presets
│   │   ├── endpoint_store.py       # JSON file store for endpoints
│   │   ├── template_store.py       # JSON file store for user prompt templates
│   │   ├── metrics.py              # ITL, percentiles & throughput computation
│   │   ├── reasoning.py            # Shared extra_body builder (thinking, effort, budget)
│   │   ├── usage.py                # Server usage parsing, stream_options & warmup helper
│   │   └── tool_benchmark_service.py # Tool calling benchmark
├── data/
│   ├── endpoints.json              # Configured inference endpoints
│   ├── templates.json              # User custom benchmark templates
│   └── decision_templates.json     # User custom decision templates
├── Images/                         # Default test assets (OCR, sample docs, cats)
├── scripts/
│   └── build_deb.sh                # Linux packaging script
├── static/
│   ├── benchmark.html              # Main playground & PDF benchmark studio
│   ├── benchmark.js                # SSE stream parser, presets & endpoint modals
│   ├── batch.html              	# Batch benchmark UI
│   ├── batch.js                	# Batch evaluation streamer
│   ├── tools.html              	# Tool calling benchmark UI
│   ├── tools.js                	# Tool evaluation functions
│   ├── decisions.html          	# Structured Decisions studio UI
│   ├── decisions.js            	# /v1/systemone runner & distribution visualizer
│   └── pdf_picker.js               # Shared PDF page-range picker (range, DPI, grid)
├── run_app.py                      # Standalone native desktop launcher
├── requirements.txt 				# Python dependencies
└── README.md

---

🛠️ Installation & Setup

1. Prerequisites

• Python 3.12+
• uv (recommended) or pip

Native Desktop Window Requirements (by OS):

• Windows: Uses Microsoft Edge WebView2 (pre-installed on Windows 10 & 11).
• Linux (Ubuntu / Debian): Requires system WebKitGTK and PyGObject libraries for native window rendering:

  sudo apt update
  sudo apt install -y python3-gi gir1.2-gtk-3.0 gir1.2-webkit2-4.1 libglib2.0-dev libgirepository-2.0-dev pkg-config

2. Install Dependencies

Using uv:

uv sync

Or using pip:

python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

# On Linux, install PyGObject for native window support:
pip install PyGObject

---

🏃 Running the Application

Development Mode (with Hot Reload):

uv run uvicorn app.main:app --host 0.0.0.0 --port 3333 --reload

Standalone Desktop Launcher:

Runs the local server on a dynamic free port and opens a native desktop application window:

uv run run_app.py

---

📦 Packaging & Standalone Release Builds

Inference Lab can be packaged into standalone native desktop releases for both Windows (.exe) and Linux (.deb).

🪟 Windows (.exe)

Build a single-file executable using PyInstaller:

Single-Line Command:

pyinstaller --noconfirm --onefile --windowed \
  --name "LLM_Benchmarking_Studio_v1.5.0" \
  --distpath "releases" \
  --icon "assets/app_icon.ico" \
  --add-data "static;static" \
  --add-data "Images;Images" \
  --add-data "data;data" \
  --add-data "assets;assets" \
  --collect-all "uvicorn" \
  --collect-all "fastapi" \
  --collect-all "starlette" \
  --collect-all "pymupdf" \
  --collect-all "webview" \
  run_app.py

The output .exe will be generated in releases/.

---

🐧 Linux / Ubuntu (.deb Package)

1. Build the Debian package:

Ensure scripts/build_deb.sh is executable and run the build script:

chmod +x scripts/build_deb.sh
./scripts/build_deb.sh

This freezes the application, bundles all static assets, and builds a Debian .deb package with desktop integration and icon registration.

The output package will be created at: releases/llm-benchmarking-studio_1.5.0_amd64.deb

2. Install the .deb package on Ubuntu / Debian:

Install using apt so all required system runtime dependencies (WebKitGTK) are automatically resolved:

sudo apt install ./releases/llm-benchmarking-studio_1.5.0_amd64.deb

(Alternative using dpkg):43. Launching the App:

• Launch directly from your application launcher / desktop menu (LLM Benchmarking Studio)
• Or from any terminal:

  llm-benchmarking-studio

To Uninstall:

sudo apt remove llm-benchmarking-studio

---

📡 API Reference

Endpoint	Method	Description
/	GET	Main benchmarking playground UI
/batch	GET	Batch evaluation dashboard UI
/decisions	GET	Structured Decisions (/v1/systemone) studio UI
/tools	GET	Tool calling benchmark UI
/health	GET	Health probe
/benchmark/run	POST	Executes streaming single benchmark via Server-Sent Events (SSE). Sends image_urls in stored page order
/benchmark/process-pdf	POST	Renders PDF pages to base64 images via PyMuPDF. Form fields: file, pages (1-based CSV, e.g. 3,1,7), dpi, count_only
/batch/run	POST	Executes concurrent dataset benchmark with live SSE progress
/tools-benchmark/run	POST	Executes the tool calling benchmark set
/decision/run	POST	Dispatches structured question reads to /v1/systemone
/decision/process-pdf	POST	Same page rendering as /benchmark/process-pdf, for the decisions page
/decision/templates	GET, POST	List or register structured decision presets
/decision/templates/{id}	DELETE	Delete user custom decision preset
/endpoints/	GET, POST, DELETE	List, save/update, or delete configured endpoints (POST upserts by id)
/endpoints/{id}	GET, DELETE	Retrieve or delete specific endpoint by ID
/endpoints/{id}/models	POST	Append or update a single model inside an existing endpoint
/templates/	GET, POST	List or register/update benchmark prompt templates (upserts by id)
/templates/{id}	GET, DELETE	Retrieve or delete a user custom benchmark template

---

📄 License

This project is open-source and available under the MIT License.