docs(http): correct serializeStream return type - follow-up to #641 by Ethan-Arrowood · Pull Request #655 · HarperFast/documentation

Ethan-Arrowood · 2026-08-28T19:08:21Z

Follow-up to #641, which is already merged. @kriszyp reviewed it after the merge and is right that the row it added is too strict:

Readable is not the complete return contract. serialize() invokes this method for any iterable response, including a plain array; the built-in MessagePack handler returns pack(data) for arrays, which is a Buffer rather than a stream. The core ContentTypeHandler declaration likewise permits Buffer | string. Please document the supported union (or restrict the statement to handlers that actually stream) so custom handlers are not given a stricter, incorrect contract.

Why #641 was wrong

#641 assumed the Buffer | string return paths were unreachable "since the callers gate on iterability first." That reasoning does not hold: an array is iterable, so the gate does not exclude it.

server/serverHelpers/contentTypes.ts:393-398 (harper origin/main):

if (
	typeof responseData === 'object' &&
	responseData &&
	(responseData[Symbol.iterator] || responseData[Symbol.asyncIterator]) &&
	serializer.serializer.serializeStream
) {

A plain array satisfies typeof === 'object' and has Symbol.iterator, so an array response reaches serializeStream, not serialize.

Evidence, per registered handler

Verified against HarperFast/harper at origin/main (ad9854bed).

mediaTypes map:

Handler	Source	`serializeStream` return
`application/json`, `/`, `''`	`contentTypes.ts:44`, `:194`	`streamAsJSON` -> `JSONStream extends Readable` (`JSONStream.ts:12-16`). Always a `Readable`.
`application/cbor`	`contentTypes.ts:53-56`	`new EncoderStream(...).end(data)` -> the transform stream. Always a stream.
`application/x-msgpack`	`contentTypes.ts:62-67`	`Readable` or `Buffer`. Explicit `&& !Array.isArray(data)` guard; an array falls through to `return pack(data)`.
`text/csv`	`contentTypes.ts:73-76`	`toCsvStream` -> `readStream.pipe(csvTransform)` (`:702`). Always a stream.
`text/plain`	`contentTypes.ts:88-90`	`Readable.from(...)`. Always a `Readable`.
`application/x-ndjson`, `application/ndjson`	`contentTypes.ts:106-111`	`Readable` or `string`. Non-iterable input returns `JSONStringify(data) + '\n'`.
`text/event-stream`	`contentTypes.ts:135-138`	`Readable.from(...)`. Always a `Readable`.

Fastify serializer list (registerContentHandlers, contentTypes.ts:211-241): same shapes - the msgpack entry at :226-231 carries the identical !Array.isArray(data) guard and return pack(data).

Custom handlers may also return a bare iterator: unitTests/testApp/resources.js:258-269 registers serializeStream as a generator / async generator, exercised by unitTests/apiTests/basicREST-test.mjs:587-611. server/http.ts:741-742 wraps a non-Readable iterable with Readable.from(), and :745-751 sends a Buffer/string body directly, so the whole union really is written to the response.

The core declaration agrees - server/Server.ts:100-105:

export interface ContentTypeHandler {
	serialize(data: any): Buffer | string;
	serializeStream(data: any): Buffer | string;
	deserialize(data: any): Buffer | string;
	q: number;
}

The compression branch is a latent bug

contentTypes.ts:410-425 pipes the serializeStream result unconditionally when compression is active:

let stream = serializer.serializer.serializeStream(responseData, responseObject);
if (canCompress) {
	responseObject.headers.set('Content-Encoding', 'br');
	stream = stream.pipe(createBrotliCompress({ ... }));
}

A Buffer has no .pipe, so Accept: application/x-msgpack + Accept-Encoding: br + an array response body throws TypeError: stream.pipe is not a function when http.compressionThreshold is non-zero (canCompress is gated on it at :373). The union is real, but that one branch does not currently handle it. Not fixed here - this is a docs PR - but the docs now warn custom-handler authors to return a stream if their handler may run with compression enabled, and it is worth a core issue.

Changes

serializeStream row: (any) => Readable | Buffer | string, plus a new Non-streaming returns from serializeStream subsection covering when each form occurs, that iterables are accepted too, and the compression caveat.
serialize row: added Readable. The built-in text/csv handler's non-streaming serialize (contentTypes.ts:77-81) also returns toCsvStream(...), a stream - so that row was over-strict in the same way.
deserialize row: was (Buffer | string) => any with "String for text/* types, Buffer for binary types." Harper always hands the handler a Buffer - getDeserializer streams to a buffer (contentTypes.ts:590), MQTT passes packet.payload (server/mqtt.ts:515), and the registered text/plain handler calls data.toString() itself (contentTypes.ts:91-93).

Verification

npm run format:write / npm run format:check clean.
npm run build succeeds with zero broken-link or broken-anchor warnings; the new #non-streaming-returns-from-serializestream anchor resolves in the built HTML.

🤖 Generated with Claude Code

gemini-code-assist

Code Review

This pull request updates the HTTP API Handler Interface documentation in reference/http/api.md. It refines the type definitions and descriptions for serialize, serializeStream, and deserialize, and introduces a new section explaining non-streaming returns from serializeStream. The reviewer suggested formatting the Brotli compression caveat as a :::caution block to make this critical warning more visible and maintain formatting consistency with the rest of the document.

gemini-code-assist · 2026-08-28T19:09:47Z

+- The built-in `application/x-msgpack` handler streams only non-array iterables. For a plain array it returns a packed `Buffer`.
+- The built-in NDJSON handler returns a string when it is handed a value that is not iterable.
+
+One caveat if you return something other than a stream: when Brotli compression applies to the response (the client sent `Accept-Encoding: br` and `http.compressionThreshold` is non-zero), Harper pipes the value returned by `serializeStream` into the compressor. A `Buffer` or string has no `pipe`, so a handler that must work with compression enabled should return a stream — wrap a buffered result with `Readable.from([buffer])` if there is nothing to stream incrementally.


Since this Brotli compression caveat describes a scenario that leads to a runtime crash (TypeError: stream.pipe is not a function), it is highly recommended to format it using a :::caution block. This aligns with the established formatting patterns on this page (such as on lines 159 and 186) and ensures that developers scanning the documentation do not miss this critical warning.

Suggested change

One caveat if you return something other than a stream: when Brotli compression applies to the response (the client sent `Accept-Encoding: br` and `http.compressionThreshold` is non-zero), Harper pipes the value returned by `serializeStream` into the compressor. A `Buffer` or string has no `pipe`, so a handler that must work with compression enabled should return a stream — wrap a buffered result with `Readable.from([buffer])` if there is nothing to stream incrementally.

:::caution

One caveat if you return something other than a stream: when Brotli compression applies to the response (the client sent Accept-Encoding: br and http.compressionThreshold is non-zero), Harper pipes the value returned by serializeStream into the compressor. A Buffer or string has no pipe, so a handler that must work with compression enabled should return a stream — wrap a buffered result with Readable.from([buffer]) if there is nothing to stream incrementally.

:::

References

When breaking down long, dense paragraphs or formatting warnings in Markdown documentation, maintain consistency with the existing formatting patterns of the page.

Applied — it is a :::caution now. I did not take the suggested text verbatim, because @kriszyp's comment on the same line asked for the caveat to be reduced: the block now leads with the streaming guidance and frames the TypeError: stream.pipe is not a function as a current limitation of Harper's compression path rather than a requirement the handler contract places on authors.

Two corrections to the suggested wording that I carried into the final text:

Compression is off by default. static/defaultConfig.yaml ships http.compressionThreshold: 0 and the gate is canCompress = COMPRESSION_THRESHOLD && ..., so the block says compression applies only when the threshold is set to a non-zero value.

Line 186 is not an admonition — it is a table. The page's only existing one is the :::caution at line 159, which is the pattern I matched.

_{sent with Claude Opus 5}

github-actions · 2026-08-28T19:11:19Z

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-655

This preview will update automatically when you push new commits.

kriszyp

We could probably go deeper with the return type, but I think this covers most of the cases.
_{🤖 Reviewed with Codex}

kriszyp · 2026-08-28T19:11:57Z

+- The built-in `application/x-msgpack` handler streams only non-array iterables. For a plain array it returns a packed `Buffer`.
+- The built-in NDJSON handler returns a string when it is handed a value that is not iterable.
+
+One caveat if you return something other than a stream: when Brotli compression applies to the response (the client sent `Accept-Encoding: br` and `http.compressionThreshold` is non-zero), Harper pipes the value returned by `serializeStream` into the compressor. A `Buffer` or string has no `pipe`, so a handler that must work with compression enabled should return a stream — wrap a buffered result with `Readable.from([buffer])` if there is nothing to stream incrementally.


This documents a framework bug as a handler requirement. Buffer and string are now advertised as supported serializeStream results, but a routine request with Accept-Encoding: br can fail because the compressor assumes .pipe(). With the default compression threshold and browser clients, custom handlers cannot reliably use part of the documented API. Harper should normalize a non-stream result with Readable.from([value]) before the compression branch, so the return contract is independent of a response-header/configuration combination; then this caveat can be removed or reduced to normal streaming guidance.

Agreed on the substance — the fix belongs in serialize(), not in every handler. Normalizing a non-stream result with Readable.from([value]) before the compression branch at server/serverHelpers/contentTypes.ts:410-425 makes the return contract independent of the request headers and the config, which is the right shape. I have reduced the caveat to ordinary streaming guidance in a :::caution and framed the .pipe() crash as a current limitation of Harper's compression path rather than something the handler contract demands.

One correction to the premise, because it changes how reachable the crash is: compression is off by default, not on. static/defaultConfig.yaml:4 ships http.compressionThreshold: 0, and the gate is canCompress = COMPRESSION_THRESHOLD && request.headers...includes('br') (contentTypes.ts:373), so a falsy threshold disables compression entirely — including the streaming branch. It has been 0 since that file was first added, and there is no code-level fallback; the only other DEFAULT_COMPRESSION_THRESHOLD in the tree is the unrelated storage one at resources/databases.ts:102. So reaching the TypeError needs a non-default compressionThreshold, plus Accept-Encoding: br, plus a non-stream return — not merely a browser client on a stock install. Still a real bug, just not one a default deployment hits.

That finding does expose a separate docs problem I am leaving out of this PR: several pages assert the opposite. reference/configuration/options.md:37, reference/http/configuration.md:111, and reference/http/overview.md:104 all state the default is 1200, with 1200 also in the sample snippets at reference/http/configuration.md:123, :318, and reference/configuration/operations.md:106. configuration.md and overview.md additionally claim streaming responses "are always compressed for supporting clients, regardless of this setting", which the canCompress gate contradicts. I will fix those separately rather than widening this PR.

Let me know if you would rather I open the core issue for the Readable.from([value]) normalization, or pick it up yourself.

_{sent with Claude Opus 5}

kriszyp · 2026-08-28T19:11:57Z

+| Property                | Type                                                  | Description                                                                                                                                                                                                                                                                                                                                                                                                                     |
+| ----------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `serialize(data)`       | `(any) => Buffer \| Uint8Array \| string \| Readable` | Serialize a complete response body. Used when the response body is not iterable, or when the handler defines no `serializeStream`. Most handlers return a string or a `Buffer`; the built-in `text/csv` handler returns a [`Readable`](https://nodejs.org/api/stream.html#class-streamreadable), which Harper streams to the response.                                                                                          |
+| `serializeStream(data)` | `(any) => Readable \| Buffer \| string`               | Serialize a streaming response body. Called once per response with the whole iterable (not once per chunk), and only when the response body is an object that is iterable or async iterable. A Node.js [`Readable`](https://nodejs.org/api/stream.html#class-streamreadable) is the usual return, but it is not the only one — see [Non-streaming returns from `serializeStream`](#non-streaming-returns-from-serializestream). |


The return type is incomplete relative to the subsection immediately below it: serializeStream is said to accept any iterable or async iterable and wrap it with Readable.from(), but neither is represented here. A typed custom handler returning a generator would therefore be rejected despite the documented runtime behavior. Add the iterable forms to the public type (and its core declaration), or narrow the prose to the supported contract.

Good catch. I added the iterable forms to the documented type rather than narrowing the prose, because the runtime behavior is deliberate and covered, not incidental: server/http.ts:778-779 wraps a non-Readable iterable with Readable.from(), and unitTests/testApp/resources.js:258-269 registers serializeStream as a generator and as an async generator, exercised end to end by unitTests/apiTests/basicREST-test.mjs:587-611. The cell now reads (any) => Readable | Iterable | AsyncIterable | Buffer | string.

On "and its core declaration" — I cannot change that from this repo, and it is worth flagging that it disagrees with both the docs and the runtime more broadly than the generator case. server/Server.ts:100-105 declares:

serializeStream(data: any): Buffer | string;

There is no Readable in it at all, so a typed handler returning Readable.from(...) — what every built-in streaming handler does — is rejected too, not just one returning a generator. Rather than silently documenting one side of the disagreement, the new subsection says the ContentTypeHandler interface is currently narrower than the runtime contract and a typed handler may need a cast.

Happy to open a core issue to widen the declaration to the full union if you would like that tracked separately.

_{sent with Claude Opus 5}

kriszyp · 2026-08-28T19:11:57Z

+The union is not theoretical. Harper picks `serializeStream` over `serialize` whenever the response body is iterable, and a plain array is iterable — so a resource that returns an array reaches `serializeStream`. A handler that has nothing to stream in that case can serialize the value in one shot instead:
+
+- The built-in `application/x-msgpack` handler streams only non-array iterables. For a plain array it returns a packed `Buffer`.
+- The built-in NDJSON handler returns a string when it is handed a value that is not iterable.


This is not a reachable serializeStream dispatch case: the preceding contract says Harper selects this method only when the response body is iterable or async iterable. A non-iterable response instead uses serialize, so this example does not demonstrate the new return union and can mislead readers about which method runs. Remove it, or explicitly frame it as a direct call outside Harper's normal response path.

Agreed, and this is the one that caught a real contradiction inside the PR — thanks. Removed the NDJSON bullet.

Verified at harper origin/main: both dispatch sites gate on iterability before selecting serializeStream — serialize() at server/serverHelpers/contentTypes.ts:393-398, and the Fastify preSerialization serializer at :275-293. The NDJSON handler's return JSONStringify(data) + '\n' (:106-111) only runs for a non-iterable argument, so Harper never reaches it; it is observable only on a direct call. I dropped it rather than reframing it — an "outside the normal response path" example does not earn its space in the handler-interface reference, and as written it pointed readers at the wrong method.

The msgpack case stays, since that is the reachable one and the actual justification for the union: an array satisfies typeof === 'object' and has Symbol.iterator, so an array response body reaches serializeStream, and the && !Array.isArray(data) guard at :62-67 falls through to return pack(data).

_{sent with Claude Opus 5}

PR #641 documented `serializeStream` as returning a Node `Readable`. That is too strict: `serialize()` selects `serializeStream` for any iterable response body, and a plain array is iterable, so handlers that cannot stream a given input return an already-serialized value instead. - `serializeStream` row now documents `Readable | Buffer | string`, matching the core `ContentTypeHandler` declaration, with a subsection covering when each occurs and the Brotli-compression caveat. - `serialize` row now includes `Readable` - the built-in `text/csv` handler's `serialize` returns a CSV transform stream. - `deserialize` row now states that Harper always passes a `Buffer`; a `text/*` handler calls `buffer.toString()` itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Review follow-up on the serializeStream return-union row. - Drop the NDJSON bullet. Its `JSONStringify(data) + '\n'` branch only runs for a non-iterable argument, and both dispatch sites gate on iterability first (`contentTypes.ts:393-398` and the Fastify `preSerialization` serializer at `:275-293`), so Harper never takes it. The msgpack/array case stays - an array is iterable, so it does reach `serializeStream`, where the `!Array.isArray(data)` guard returns `pack(data)`. - Add the iterable forms to the documented return type. `http.ts:778-779` wraps a non-`Readable` iterable with `Readable.from()`, and the generator handlers in `unitTests/testApp/resources.js:258-269` are covered end to end by `unitTests/apiTests/basicREST-test.mjs:587-611`, so this is a supported contract rather than an accident. Note that core's `ContentTypeHandler` interface is narrower than the runtime behavior. - Reduce the compression caveat to streaming guidance in a `:::caution`, and frame the `.pipe()` crash as a current limitation of Harper's compression path instead of a handler requirement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions · 2026-08-31T18:44:27Z

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-655

This preview will update automatically when you push new commits.

Ethan-Arrowood requested a review from a team as a code owner August 28, 2026 19:08

gemini-code-assist Bot reviewed Aug 28, 2026

View reviewed changes

github-actions Bot temporarily deployed to pr-655 August 28, 2026 19:11 Inactive

kriszyp approved these changes Aug 28, 2026

View reviewed changes

Ethan-Arrowood and others added 2 commits August 31, 2026 12:36

Ethan-Arrowood force-pushed the docs/serializestream-return-union branch from 21be454 to 335acc2 Compare August 31, 2026 18:41

Ethan-Arrowood mentioned this pull request Aug 31, 2026

http.compressionThreshold: docs say default 1200, shipped config sets 0 (compression off by default) #656

Open

github-actions Bot deployed to pr-655 August 31, 2026 18:44 View deployment

Navigation Menu

Uh oh!

docs(http): correct serializeStream return type - follow-up to #641 - #655

docs(http): correct serializeStream return type - follow-up to #641#655
Ethan-Arrowood wants to merge 2 commits into
mainfrom
docs/serializestream-return-union

Ethan-Arrowood commented Aug 28, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Aug 28, 2026

Uh oh!

Ethan-Arrowood Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

kriszyp left a comment

Uh oh!

kriszyp Aug 28, 2026

Uh oh!

Ethan-Arrowood Aug 31, 2026

Uh oh!

kriszyp Aug 28, 2026

Uh oh!

Ethan-Arrowood Aug 31, 2026

Uh oh!

kriszyp Aug 28, 2026

Uh oh!

Ethan-Arrowood Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-One caveat if you return something other than a stream: when Brotli compression applies to the response (the client sent `Accept-Encoding: br` and `http.compressionThreshold` is non-zero), Harper pipes the value returned by `serializeStream` into the compressor. A `Buffer` or string has no `pipe`, so a handler that must work with compression enabled should return a stream — wrap a buffered result with `Readable.from([buffer])` if there is nothing to stream incrementally.
+:::caution
+One caveat if you return something other than a stream: when Brotli compression applies to the response (the client sent Accept-Encoding: br and http.compressionThreshold is non-zero), Harper pipes the value returned by serializeStream into the compressor. A Buffer or string has no pipe, so a handler that must work with compression enabled should return a stream — wrap a buffered result with Readable.from([buffer]) if there is nothing to stream incrementally.
+:::

Uh oh!

Conversation

Ethan-Arrowood commented Aug 28, 2026

Why #641 was wrong

Evidence, per registered handler

The compression branch is a latent bug

Changes

Verification

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 28, 2026

🚀 Preview Deployment

Uh oh!

kriszyp left a comment

Choose a reason for hiding this comment

Uh oh!

kriszyp Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

kriszyp Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

kriszyp Aug 28, 2026

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 31, 2026

🚀 Preview Deployment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants