Add progress-event callbacks for evaluation requests - #28
Open
m-messer wants to merge 30 commits into
Open
Conversation
Implement a unified progress reporting framework that supports optional HTTP callbacks: - Define `Event` and `Reporter` abstractions in `internal/progress`. - Add `HTTPFactory` for building per-request HTTP callback reporters. - Introduce `Emit` convenience method to attach/report progress events via context. - Update supervisor and handler logic to emit lifecycle events. - Include comprehensive unit tests for reliability and correctness.
- Integrate callbackUrl field from µEd spec for progress reporting. - Replace progress callback headers with callbackUrl and request ID. - Include evaluation feedback payload in StageCompleted events. - Update tests to reflect callbackUrl usage and validation.
…ck delivery timeout
…k feature - Introduce `--progress-allowed-hosts` flag to restrict allowed callback hostnames. - Add `--progress-allow-private-networks` flag for optional private network access. - Implement automatic request ID generation for traceability and progress correlation. - Expand documentation with guidance on callback URL safety and SSRF prevention. - Update tests and internal logic for new configuration options and request IDs.
- Introduce IP filtering to block private, loopback, and link-local addresses. - Add hostname wildcards for fine-grained allowed host configuration. - Implement custom HTTP transport with DNS-based IP validation. - Add comprehensive unit tests to cover SSRF scenarios and configuration options.
- Introduce IP filtering to block private, loopback, and link-local addresses. - Add hostname wildcards for fine-grained allowed host configuration. - Implement custom HTTP transport with DNS-based IP validation. - Add comprehensive unit tests to cover SSRF scenarios and configuration options.
- Introduce `--progress-sidecar-unbind-grace-period` flag with default value of 250ms. - Add `UnbindAfterGrace` method to allow delayed unbinding with generation-safe logic. - Update supervisor adapter to utilize `UnbindAfterGrace` for improved POST handling.
MinEventInterval's default (200ms) rate-limited a fast evaluation function reporting two checkpoints from compareSets' evaluation function to at most one event per span: even with delivery now serialized on the client side, two closely-spaced report_progress() calls could still both arrive well under any single fixed interval, since arrival timing is governed by local HTTP round-trip cost, not real application-level delay. Add BurstSize (default 5): the first N events in a span bypass MinEventInterval spacing entirely (still bounded by MaxEventsPerSpan), so a handful of legitimate back-to-back checkpoints go through, while MinEventInterval keeps guarding against sustained event spam once the burst is used up. Also lower the MinEventInterval default itself from 200ms to 10ms, since 200ms had no real abuse-prevention basis and was overly aggressive for normal use.
- Add test coverage for new SSEReporter behavior and intermediate step deduplication. - Include tests for dependency graph validation in Lambda and standalone runtime modules. - Add multi-reporter tests to verify fan-out behavior and isolated child panics. - Improve overall test reliability with enhanced mocks and structured assertions.
- Implement SSE-based streaming for progress updates on `/evaluate` responses. - Add tests to validate SSE behavior, event streaming, and live frame correctness. - Collapse repeated lifecycle stages (`preparing`, `evaluating`) into single events per request. - Update README with detailed documentation on SSE usage, configuration, and behavior. - Introduce `--progress-stream-enabled` and `--progress-stream-heartbeat-seconds` flags for configuration. - Ensure terminal frames include accumulated steps and align with live frame data.
- Extract shared SSE streaming logic into `streamProgress` for reuse across `/evaluate` and `/chat`. - Add per-command terminal frame shapes (`sseEnvelope` for `/evaluate`, `sseChatEnvelope` for `/chat`). - Extend progress stages with `starting` and `thinking` for unified lifecycle reporting. - Refactor `/chat` to support streaming progress updates with callback compatibility.
- Extend progress lifecycle to include `starting` stage for improved stage granularity. - Add `/chat` endpoint compatibility with SSE progress streaming and event validation. - Refactor evaluation tests to use `starting` in place of `evaluating` at appropriate stages. - Update README to document new stages and `/chat` progress models. - Add corresponding unit tests for new lifecycle reporting and endpoint behavior.
- Introduce comprehensive test coverage for `/chat` endpoint's Server-Sent Events (SSE) streaming behavior. - Validate `thinking` and `starting` progress stages, terminal frame shapes, and fallback to JSON output. - Add tests for capability-disabled scenarios, callback handling, authentication errors, and heartbeat events.
- Extend `--progress-stream-enabled` flag and behavior to `/chat` responses. - Update README and CLI usage text to reflect new `/chat` SSE streaming support.
- Validate terminal frame payloads against OpenAPI schemas for `/chat` and `/evaluate` SSE responses. - Refactor middleware to rely on runtime response sniffing instead of preflight checks. - Update tests for new validation logic and streaming behavior. - Document terminal frame structure in the OpenAPI schema.
- Add tests to enforce parity between SSE frame structs and OpenAPI schema definitions. - Introduce `ValidateComponentSchema` for validating payloads against specific OpenAPI component schemas. - Refactor `validate_body.go` to centralize schema validation logic and improve testability.
- Standardize SSE terminal frames to exclude `command` and move failure details to `error` objects. - Add structured `ErrorInfo` for detailed failure representation, aligning with OpenAPI schemas. - Update `/chat` and `/evaluate` SSE handlers and tests to reflect the refined terminal frame structure. - Document streaming variant support and terminal frame changes in OpenAPI specifications. - Extend back-end validation to enforce parity between emitted events and schema definitions.
- Extend `MuEdToChatHealthResponse` to handle new `streamingEnabled` flag and expose `supportsStreaming` and `supportedProgressStages`. - Introduce `chatProgressStages` defining possible SSE progress stages for `/chat`. - Update `/chat` SSE capabilities to align with shimmy-layer features.
…hat messages - Simplify `MuEdChatMessage` by removing `MuEdChatRole` type in favor of untyped strings for role handling. - Update tests to align with the revised structure.
- Add `ErrorInfo` to payloads for `StageFailed` events, standardizing failure handling across callback and SSE responses. - Update `/chat` and `/evaluate` handlers and tests to validate structured error propagation. - Add unit tests ensuring error object correctness in HTTP callbacks and SSE terminal frames.
- Clarify the use of `ErrorResponse` in terminal `failed` stages for `/evaluate` and `/chat`. - Add examples showcasing the updated payload format and structured error propagation. - Refine descriptions of terminal frame handling and SSE response structure.
- Clarify comments on error propagation and structured error details in `StageFailed` events. - Improve descriptions of terminal frame shapes, progress events, and fallback mechanisms.
- Add validation for `ErrorResponse` objects in terminal `failed` events. - Improve structured error propagation by asserting `error.title` in SSE tests. - Clarify comments on `command` field usage in terminal frame shapes.
The standalone server wrapped the route mux with NormalizePath + the OpenAPI request/response validation middleware; the Lambda adapter wrapped it with NormalizePath only. That left Lambda requests unvalidated against the spec and, more importantly, Lambda responses unchecked — so a non-conforming response would 500 on standalone but ship as-is on Lambda. Extract the wrapped chain into server.NewMux (mux + NormalizePath + per-version OpenAPI validation) and a server.HandlerModule fx module. Both app/standalone and app/lambda now depend on it and serve the exact same *server.Mux, so the two deployments validate identically. The only remaining deployment difference is transport: standalone runs an http.Server + listener (and optional h2c); Lambda hands the same handler to httpadapter. NewHttpServer / NewLifecycleServer no longer build the chain or return an error. Added fx.ValidateApp coverage for the Lambda graph and a NewMux test asserting validation + path normalisation are applied. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAusDUAMwEVN8hAV4N8qGk
Reconcile the versioned µEd adapter/registry architecture with SSE progress streaming: - Handlers resolve X-Api-Version to a MuEdAdapter and drive decode/encode through it, while retaining the streaming flow (streamProgress, serveChatStream/serveEvaluateStream, terminalError, produceFeedback). - MuEdAdapter.EncodeHealth/EncodeChatHealth take a streamingEnabled flag so per-version health encoding still advertises shimmy's SSE capability. - runtime no longer imports internal/progress (stage-name literals inlined) and internal/server no longer imports runtime (nil-resolver default routes off loaded spec versions) — combining both branches otherwise formed a server -> runtime -> progress import cycle. - server.Module re-provides LoadOpenAPISpec for the standalone streaming handler's SSE terminal-frame validation. - openapi_test keeps both the SSE-bypass and version-selection tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAusDUAMwEVN8hAV4N8qGk
- Add versioned adapters and `SupportsStreaming` to conditionally enable SSE. - Update `/chat` and `/evaluate` handlers and tests to use `X-Api-Version`. - Refine OpenAPI specs and README to document `0.1.1-dev`-specific streaming behavior. - Enhance test coverage for version negotiation and SSE response validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /evaluateendpoint: shimmy POSTs a small JSON event (preparing→evaluating→completed/failed) to a caller-suppliedcallbackUrlat each stage of processing, in addition to the normal synchronous HTTP response.callbackUrlbody field andX-Request-Idheader (now echoed on every response, generated if the caller doesn't supply one, and reused as the progress correlation key).completedevent carries the actual feedback payload indata.feedback, so a caller usingcallbackUrlgets the final result delivered there too, converging with the spec's "deliver feedback results to this URL" wording even though shimmy always takes the synchronous 200 path rather than the spec's 202-Accepted deferred flow.--progress-allowed-hosts) and an explicit opt-out (--progress-allow-private-networks) for trusted private deployments.internal/progresspackage: a generic, reusable event/progress-reporter abstraction threaded through the dispatcher/supervisor/handler layers viacontext.Context, so future custom events (e.g. emitted by the evaluation function itself) are a natural extension rather than a rework./evaluatecallers that don't supplycallbackUrlsee no behavior change.🤖 Generated with Claude Code
https://claude.ai/code/session_0187xesBCDmvKLY1xYbSVfhA