fix(server): clamp /v1/models context_length to the allocated KV pool

[oscar-investmatic]

Fixes #448.

The problem

The engine clamps its own max_seq_len to the allocated KV pool, at startup and again after every rebuild:

# python/freetoken/engine/engine.py:445  (and :859, _refresh_seq_state)
self.max_seq_len = min(config.max_seq_len, num_tokens)   # num_tokens = num_pages * page_size

and the scheduler admits against that clamped value (scheduler/scheduler.py:513). That is why #448 shows a correct rejection:

prompt is too long: 171034 tokens > 171008 maximum (prompt + generation)

171008 is the clamped max_seq_len — the real pool. The server knows its true limit.

But /v1/models is served from the frontend process, which holds an unclamped ServerArgs copy, so it published the model's positional ceiling instead. The engine and this route disagreed about one quantity, and the route's answer is the one agents consume.

Why it bites

launch.py:215 reads max_model_len into _context_window() (:283), which feeds every agent integration:

consumer	field	line
opencode	limit.context	launch.py:449
codex	context_window / max_context_window	launch.py:301
Claude Code	CLAUDE_CODE_MAX_CONTEXT_TOKENS	launch.py:422

So agents size their compaction threshold past what the server can hold and run into context_length_exceeded instead of compacting in time. That matches the report in #448 that it is "not just that agent and model".

Worth noting FALLBACK_CONTEXT_WINDOW = 128_000 (launch.py:24), used when the server reports no context length, is commented "Guessing low only costs earlier compaction." The blind fallback was safer than the number read from the API.

Measured on 0.1.3 (cac247a86), RTX 4070 12GB, Linux:

model	/v1/models before	real pool	/v1/models after
Qwen3.6-35B-A3B-NVFP4	262144	178176	178176
gemma-4-26B-A4B (bf16)	262144	44957	44957

The change

• kv_pool_geometry(state) — the pool resolution lifted out of cache_geometry (last rebuild → running snapshot → load-time ("meta", …) ack). Extracted rather than duplicated so /v1/models and /v1/cache/status cannot report different capacities for the same server, and so this tracks rebuilds rather than freezing the load-time value.
• _model_context_length clamps with it, falling back to the model ceiling while num_pages is still 0 (loading, or an engine build that sends no meta ack). Import is function-local because api_server imports openai_api.

On the previous behavior

The old docstring made this call deliberately — "The model ceiling, not min(ceiling, KV budget): a rebuild moves the latter, and agents read this once at startup." The rebuild concern is real, but the engine already recomputes the clamp on rebuild (engine.py:859), so only an agent's startup copy goes stale — and a stale reachable number seems strictly better than one that was never reachable. Happy to drop this if you disagree; the complete fix probably also re-reads /v1/models (or rewrites the agent config) after a rebuild, which I have not attempted here.

Tests

tests/server/test_openai_api.py:445 asserted max_model_len == 262144 and passed either way — FakeState carries no pool geometry, so it fell through to the ceiling. It was green by accident. It now asserts that fallback explicitly, plus:

• clamp to the pool (262144 ceiling, 178176 pool → 178176)
• pool larger than the ceiling does not extend context
• a rebuild's geometry supersedes the load-time allocation

Verified the new cases fail on main and pass with the change:

with the fix reverted:
  assert 262144 == 178176   FAILED test_models_route_clamps_the_context_length_to_the_kv_pool
  assert 262144 == 40000    FAILED test_models_route_prefers_the_last_rebuild_over_the_load_time_pool
with the fix:
  4 passed

tests/server/ + tests/scheduler/test_scheduler_kv_usage.py (the other cache_geometry consumer): 566 passed.

Not covered: I verified against the test suite and stubbed states, not a restarted server end to end, and I have not run the full suite (no GPU-free path for the engine tests on my box).