CONVERSATION_CACHE=1 reuses unchanged turns from one active conversation. It is
available as Reuse unchanged conversation turns (0/1) in advanced configuration;
FLASHCHAT_CONVERSATION_CACHE=0 overrides it for a test run. The setting participates
in server invalidation, so a running server restarts before changed settings are reused.
The server compares exact rendered tokens. It retains live state when possible, otherwise restores a checkpoint before the latest assistant response and processes that response plus new input. This handles client normalization of reasoning and tool arguments without treating a matching session ID as proof of matching input. Changed/shortened history falls back safely; cancellation discards conversation reuse. Both Chat Completions and Responses use this path. Clients still send the complete conversation; this does not add a server-side conversation-ID storage API.
The cache is held only in memory and is independent of persistent system/tool
caching. It keeps one recurrent-state checkpoint and token ledger; attention history
uses the existing buffers. Interleaving unrelated conversations replaces the active
entry. Logs state how many history tokens were reused and how many prompt tokens
remain to process. When history cannot be reused, they separately report tokens
supplied by the system/tool cache, for example: conversation cache: no reusable history; system/tool cache supplied 10197 prompt tokens. Saved current-state and
checkpoint token counts are also logged. See
experiment 26 for measurements
and limits.
New configurations use Qwen3.6-35B-A3B q4 with a q8 context cache and a
65,536-token context window. Existing configurations retain their cache choice:
an old empty or missing KV_QUANT means fp32; the configuration menu now saves
an explicit off when fp32 is selected. Updating Flashchat does not opt an
existing configuration into q8.
New configurations cap the expert RAM cache at 8 GiB (also constrained by the configured free-memory fraction) and display thinking tokens. Displaying thinking does not change the model's reasoning setting.
Launcher and configuration-menu fallback values come from lib/config.sh;
model and sampling-profile defaults still come from the model registry. Production
and benchmark launches share the same resolved-settings export helper. Saved
choices remain intact when missing keys are added during migration.
Advanced configuration also exposes the existing IO_THREADS (8 disk readers),
GPU_ROPE (1), FUSED_ATTN (1, fp32 attention only), and PREFILL_RELEASE (1)
controls. These retain their previous engine defaults. Environment overrides use
the same names prefixed with FLASHCHAT_ and participate in server invalidation.
SERVER_BIND is the IPv4 address the server listens on, default 127.0.0.1
(local access only). Set it explicitly to 0.0.0.0 to listen on all IPv4
interfaces. This also applies to upgraded configurations: remote clients require
an explicit non-loopback bind address. SERVER_HOST remains the destination
Flashchat clients connect to; it is not the listening address. Both settings are
available in the configuration menu. Do not expose the API to an untrusted network.
MAX_TOKENS supplies the response limit for API requests that omit one, as well
as for Flashchat clients; explicit API limits take precedence. The server caps
response limits at 32,768 tokens. The default remains 8,192.
The benchmark harness creates a fresh shipping configuration and clears inherited
FLASHCHAT_* runtime overrides before launching its server. It exports the
resolved settings through the shared config helper, including q8, expert split
I/O, and vocabulary-head locking. Its fixed prompts, temperature zero, and short
response limits remain intentional measurement overrides.
Flashchat runs one generation at a time. A second generation request receives
HTTP 503 with error type server_busy; there is no conversation queue or
concurrent model execution. Clients can retry after the active request finishes.
OpenCode title requests receive the opening words of the first usable user request
directly from the HTTP thread, before inference admission. The title skips OpenCode's
wrapper message, collapses whitespace, and fits within 50 visible characters,
including an ellipsis when truncated. Truncation prefers a word boundary and never
splits an emoji or combined character. Empty input falls back to New conversation.
Both streaming and JSON title responses bypass the busy check and never acquire
the inference slot or touch conversation caches; a concurrent first chat therefore
does not contend with its
background title request. Matching uses the leading system/developer instruction,
not title-like text in user messages.
The HTTP event thread accepts bounded requests, handles /health, /v1,
/v1/models and preflight requests, and relays inference output. The original
inference thread owns all model state, caches, and accelerator operations.
Streaming and non-streaming responses use the same bounded relay. A client
that stops accepting output for ten seconds is disconnected. Request uploads
also have a ten-second deadline, with a 1 MiB request limit and at most sixteen
pending HTTP connections. HTTP payload logging remains available.
Closing a streaming request cancels its work once the transport detects the disconnect. The worker checks between prompt-processing layers and generated tokens (between speculative batches when MTP is active), even when tool-call or reasoning output is buffered. Streaming heartbeats continue during otherwise silent generation so a disconnected reader can be detected. An in-flight GPU, Neural Engine, or disk operation finishes before cleanup; cancellation does not interrupt accelerator operations or allow overlapping inference.
Cancelled prefill does not save a partial system-prompt snapshot. Incomplete live context is cleared, request allocations are released, and the server becomes available after cleanup. Logs distinguish prefill and generation cancellation. A client that only half-closes its request-writing side may still read the full response; that alone is not treated as cancellation. For a non-streaming response, a TCP reset is detected while computing, but a graceful half-close cannot be distinguished from abandonment until output is attempted.
SIGTERM/SIGINT stop the HTTP transport and use the same cooperative cancellation checks, waiting for in-flight operations and request cleanup before model teardown.
tests/test_server_http.py covers transport cancellation without model weights.
tests/test_api_cancel.py --port PORT exercises prefill/generation disconnects and
checks that a subsequent request matches a clean response. Run it only against a
dedicated functional-test server, not a user's active server.
The inference worker publishes a small status snapshot under a short mutex.
No inference or network I/O occurs while that mutex is held. /health retains
its previous fields and adds:
| Field | Meaning |
|---|---|
phase |
idle, preparing, prefill, or generating |
context_used |
Positions committed across the model; retained after completion |
max_context |
Actual configured context window |
cached_tokens |
Positions reused from the active conversation or restored from the system prompt cache |
prompt_tokens |
Tokens requiring prompt processing, excluding restored context |
prefill_done |
Completed prompt positions; advances when a whole chunk completes |
generated_tokens |
Generated output tokens, including reasoning/tool output |
chunk, chunks, layer, layers |
Current batched prefill work; zero when not applicable |
During a chunk, some layers have processed additional positions, but those
positions do not count as occupied context until every layer has completed.
The layer indicator shows progress during this interval. Per-token prefill
updates completed positions. Updates also work for non-streaming requests.
ready: true means the loaded server is operational, not that it has capacity
for another generation.
The management menu fetches one snapshot per refresh for quantization,
context usage, and processing progress. An unavailable server produces an
explicit unavailable reading instead of zero usage. Its capacity fallback
uses the configured window (64K by default), clamped by known model limits.
The MiB figure describes allocated context-cache capacity, not memory used by
just the occupied positions. The menu remains a refresh-on-action interface.
The status block always has two rows: context usage and one processing summary.
Prompt processing shows its percentage, token counts, and current layer when
available; generation shows the generated token count. Idle, stopped, preparing,
and unavailable states use plain descriptions in that same row. Chunk and reused
context details remain available through /health, not as placeholder menu rows.
Validation:
make server-http-smoke # Real transport with a lightweight fake worker + menu rendering
python3 tests/test_server_live.py --url http://127.0.0.1:9999 # Idle, updated model server
make tool-template-smoke
make bench-api # Idle machine, canonical performance validation
make bench-reportSee live acceptance and baseline comparison and experiment 45 for the measured results and test-artifact provenance.
Sampling uses the selected model profile, with explicit API parameters taking precedence. Temperature and repetition-related penalties remain the same inside tool calls, including file contents and other free-form arguments. Tool syntax and argument validation do not silently change sampling settings.