Skip to content

feat(evals): compare writing skills with Humanizer - #7

Open
christso wants to merge 1 commit into
mainfrom
feat/humanizer-comparison
Open

christso wants to merge 1 commit into
mainfrom
feat/humanizer-comparison

Conversation

@christso

Copy link
Copy Markdown
Contributor

Summary

Add bun run eval:humanizer to compare the actual Humanizer skill with write-naturally on the same public cases, writer settings and graders.

The extension fetches a pinned skill revision and its MIT license, verifies both hashes, and prepares a separate temporary workspace. The runner changes the skill-loading instruction and checks while retaining the common rewrite instructions and grading criteria. It also supports caller-supplied suites without storing their fixtures in this repository. Documentation covers focused runs, provenance, host downloads and cleanup limits.

Verification

  • Catalogue and local links validated; all 21 existing unit tests passed.
  • Generated public and companion private configurations validated without model calls.
  • Temporary CLI fixtures preserved all cases and default/case-specific graders, checked pinned downloads and skill-read assertions, and verified cleanup and exit status on success and failure.
  • HTTP, transport and checksum failure fixtures stopped setup and cleaned the workspace.
  • Independent review found no remaining blockers.

No comparative model evals were run. These changes add the comparison setup; they do not establish which skill produces better writing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant