Skip to content

refactor(eval): share cell execution with fleet workers - #5294

Open
likun666661 wants to merge 1 commit into
mainfrom
refactor/eval-shared-execution
Open

likun666661 wants to merge 1 commit into
mainfrom
refactor/eval-shared-execution

Conversation

@likun666661

Copy link
Copy Markdown
Member

Summary

The fleet worker previously consumed lifecycle evidence only from the simulator. Share executeExperimentCell and admission preparation with the existing local runner, and add createFleetGroupExecution to run existing SubjectAdapter/ExperimentExecutor implementations through that path. Evidence is recorded at execution, verification, and finalization boundaries; local result semantics are preserved.

The bridge prepares adapters once before worker registration, validates assigned cells, and emits individual results after explicit artifact collection. Missing metering does not invalidate verified work; token counts without settled cost retain unknown completeness. Verification-only recovery remains out of scope.

Refs #5284. Follow-up to the shared-execution integration concern in #5288.

Verification

  • 15 new integration tests exercise the existing ExternalSubjectAdapter with mocked process/verifier boundaries, including full worker-to-coordinator artifact commits, partial retries, timeout zeroes, lifecycle failures, admission rejection, and concurrent cell emission.
  • Eval TypeScript regression: 128 passed, 1 platform-dependent skip, including the 31 deterministic fleet tests.
  • Full repository build, typecheck, lint, format, and desktop/UI knip passed before transplanting this isolated commit onto current main; eval build, lint, format, and regression rerun on the final branch. Dependencies are reused from the existing checkout, which contains unrelated uncommitted model-catalog changes; none are in this PR.
  • No live VM, Docker/Harbor deployment, provider inference, or new Python changes.

Remaining deployment integration

  • Connect authenticated remote transport and local connectivity detection.
  • Wire real environment/resource preflight and deployment-specific artifact byte collection.

These callbacks are explicit in the new bridge. Tests establish the shared execution seam, not real VM admission.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex authored the implementation, tests, documentation, and description under human direction. Commit includes Generated-by: Codex.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

Capture lifecycle evidence in the existing runner and bridge prepared adapters into fleet group execution. Exercise the external subject path, partial retry, admission, and artifact collection through mocked executor boundaries.

Generated-by: Codex
@likun666661
likun666661 marked this pull request as ready for review September 14, 2026 09:42
@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant