Skip to content

fix(python): preserve explicit token limits for Responses API - #222

Open
Kingsley_Li (kingsleyli920) wants to merge 2 commits into
braintrustdata:mainfrom
kingsleyli920:codex/responses-token-limit
Open

Kingsley_Li (kingsleyli920) wants to merge 2 commits into
braintrustdata:mainfrom
kingsleyli920:codex/responses-token-limit

Conversation

@kingsleyli920

@kingsleyli920 Kingsley_Li (kingsleyli920) commented Sep 13, 2026

Copy link
Copy Markdown

LLMClassifier(max_tokens=256) silently loses its explicit output limit when routed to the Responses API, either automatically for GPT-5 or with use_responses_api=True. prepare_responses_params currently discards that field. This maps it to the Responses API's max_output_tokens, so the caller's limit reaches the provider. Unspecified limits remain omitted.

The fix is shared by the synchronous and asynchronous clients. The LiteLLM adapter also translates max_output_tokens back to max_tokens before dispatching to litellm.completion or litellm.acompletion, removing the Responses-only field. Both native OpenAI and LiteLLM paths therefore preserve the explicit limit.

HTTP-boundary regression tests cover native clients, both routing modes, and explicitly supplied versus omitted limits. LiteLLM regressions exercise LLMClassifier through the real Autoevals wrapper and adapter, inspecting the arguments at the completion boundary for synchronous/asynchronous, automatic/explicit Responses, and regular Chat Completions routes. Existing parameter tests now also assert the translated field instead of only checking that max_tokens is absent.

Validation:

  • Before the fix: 5 regression failures due to missing max_output_tokens; the 5 default-limit checks passed.
  • The LiteLLM regression failed in all four explicitly limited Responses combinations before its reverse mapping was added; default-limit and regular Chat Completions controls passed.
  • After the fix: all 63 selected offline cases across test_llm.py, test_oai.py, and test_litellm.py passed with network sockets disabled. Four existing live model tests were excluded by exact test name.
  • A separate single-request smoke with a synthetic word-classification prompt on gpt-5.4-mini succeeded through the regular Autoevals client: outgoing Responses request contained max_output_tokens: 768, HTTP 200, parsed score 1.0.
  • Black, Ruff, codespell, and diff checks passed for the changed files.

This honors an explicitly requested limit and does not introduce a default token cap.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-13T01:50:29.272641Z 756fcee PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 756fceee36

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread py/autoevals/oai.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant