Skip to content

Add POST /redrob/speech so a client can play speech without holding the key - #73

Open
flxkm522 wants to merge 2 commits into
feature/speech-generate-toolfrom
feature/speech-endpoint
Open

flxkm522 wants to merge 2 commits into
feature/speech-generate-toolfrom
feature/speech-endpoint

Conversation

@flxkm522

@flxkm522 flxkm522 commented Oct 9, 2026

Copy link
Copy Markdown

Read-aloud, option (a) from the media-output plan: the engine speaks, and Redrob Cowork plays the result. Stacked on #71; base is feature/speech-generate-tool.

Why an engine route: read-aloud has to play audio in the app, but the Redrob key belongs to the engine. POST /redrob/speech speaks text through the gateway using the engine's own credential and returns the mp3. The key never leaves this process. Cowork's server will proxy this route to the app, the same way it already talks to the engine for /auth/redrob.

What changes

  • POST /redrob/speech in the control group (authorized root API, next to /auth/:providerID).
    • Payload: { text, voice?, model? }.
    • Success: audio/mpeg bytes (HttpApiSchema.asUint8Array).
  • One shared code path with speech_generate (src/tool/speech_generate.ts):
    • prepareSpeech validates the text and picks the model and voice.
    • speechKey finds the credential.
    • requestSpeech makes the single gateway call and checks the answer is audio.
    • Each fails with a typed SpeechError. The tool's behaviour is unchanged.
  • Declared errors:
Case Status Sent to gateway?
empty text, or over 4,096 code points 400 InvalidRequestError no
no Redrob credential 503 ServiceUnavailableError (the engine lacks it, not the caller) no
gateway refusal, or a non-audio answer 502 UpstreamError with the gateway's message yes
timeout 504 TimeoutError yes

Proof

Command Result
bun test test/server/httpapi-speech.test.ts 4 pass. Uses the production route tree on NodeHttpServer.layerTest with a gateway stand-in on its own Effect listener and fresh router. Checks: the mp3 bytes round-trip; the engine key goes only to the gateway as a bearer token; the exact gateway body; 400 and 503 with nothing sent; a 402 refusal and a JSON 200 both become 502 carrying the gateway's message
bun test test/tool/speech_generate.test.ts 4 pass, unchanged after the refactor
bun test test/server test/tool 740 pass / 1 fail. The failure is write.test.ts "OS denies write access", which also fails on develop here (runs as root). OpenAPI tests pass with the new route
bun typecheck (pre-push, 18 packages) exit 0
prettier --check, oxlint on the changed files clean

Not covered: a live call to the gateway (needs mckinley-and-rice/redrob-console#232 deployed).

…he key

Redrob Cowork's read-aloud has to play audio in the app, and the Redrob key that pays for it belongs to this engine. The route speaks text through the gateway on the engine's own credential and answers with the mp3, so the key never leaves this process.

It shares one path with speech_generate: prepareSpeech validates and picks the model and voice, speechKey finds the credential, and requestSpeech makes the one gateway call and checks the answer is audio. Each fails with a typed SpeechError, which the route maps to its declared errors: invalid text is a 400, no credential a 503 (the engine lacks it, not the caller), a gateway refusal or non-audio answer a 502 carrying the gateway's message, and a timeout a 504. The first two are found before anything is sent. The tool's behaviour is unchanged.
POST /redrob/speech exists for audio a client plays at once: a reply read
aloud and the voice conversation's answer, where the wait before the first
word is what the user feels. It now defaults to eleven-flash-v2.5 with the
same voice, sarah. speech_generate keeps eleven-multilingual-v2 for saved
files, and a caller can still name any model.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants