Skip to content

fix(runtime): bound OpenAI Chat image tool results - #5278

Open
Dante-dan wants to merge 2 commits into
apache:mainfrom
Dante-dan:fix/openai-chat-image-tool-results
Open

Dante-dan wants to merge 2 commits into
apache:mainfrom
Dante-dan:fix/openai-chat-image-tool-results

Conversation

@Dante-dan

Copy link
Copy Markdown
Contributor

Summary

  • prevent OpenAI Chat wires from lowering image Tool Results into textual base64
  • preserve the matching assistant tool call and tool result with a bounded text fallback, while leaving user-message and native image paths unchanged
  • apply the same projection rule to current and durable replay, with request-body regression coverage

Fixes #5277

Verification

  • npm run lint — 3,569 files checked
  • npm run format:check — 2,108 files checked
  • npm run build — all workspaces passed
  • npm run typecheck — all workspaces passed after the dependency build
  • npx knip --workspace apps/desktop
  • npx knip --workspace packages/ui
  • node --test packages/runtime/dist/__tests__/ai-sdk-backend.test.js — 240 passed
  • git diff --check

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Codex traced the protocol boundary, implemented the wire-aware Tool Result projection, and authored the request-body regression.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

Generated-by: Codex
Signed-off-by: Dante <duanjl.china@gmail.com>
@github-actions github-actions Bot added the effort/M Under 500 readable lines label Sep 14, 2026
Generated-by: Codex
Signed-off-by: Dante <duanjl.china@gmail.com>

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact commit 32ac7ddf7338c2d1ffdfc63db9a51541d9a42899. I found no P0-P3 issues.

The change keeps image Tool Results off the OpenAI Chat wire before artifact hydration, emits a bounded string fallback, and preserves the assistant tool-call/result identity. The durable projection path applies the same rule to live continuation and restart replay, while user-message images and image-capable non-Chat protocols remain unchanged.

I verified the production OpenAI-compatible request body: the tool result is a string, the matching tool call ID remains present, the image bytes are absent, and the artifact reader is not invoked. The real Host/store path covers the live turn, a later turn, and reopen replay. Checks passed: build:test, full typecheck/lint/format, ASF headers, git diff --check, Runtime 3472 passed / 13 skipped, focused Runtime/Runtime Host image-path tests, hosted test, and a clean merge with current main (f109ccde). Runtime Host full tests were 1917 passed / 19 skipped / 1 failed; the sole failure is the unchanged managed-Bash sandbox integration because this runner rejects both unshare and bwrap user namespaces. I did not call a real external OpenAI-compatible provider or exercise native Windows/macOS.

Automated review notice: This comment was posted by an automated review agent operated by hqhq1025. It is not an independent human review and does not replace one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/M Under 500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(runtime): OpenAI-compatible Chat stringifies Read image Tool Results into textual base64

2 participants