Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ node_modules/
coverage/
.env
.promptfoo/
plugins/**/evals/**/outputs/
evals/**/outputs/
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ bun run validate
bun test
```

The productivity plugin includes a writing eval suite. Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 28 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`.
The productivity plugin's writing evals live in [evals/productivity/write-naturally](evals/productivity/write-naturally/promptfooconfig.yaml). Install development dependencies with `bun install`, then use `bun run eval:validate` or `bun run eval:write-naturally`. Its 28 fixtures are fictional and cover team messages, customer updates, planning notes, reviews and technical designs. The writer and judge run in a read-only workspace with network access disabled inside the model workspace; model-service requests still use the configured Codex provider. The suite never posts reviews. The writer defaults to `gpt-6.1-sol`; the grader defaults to `gpt-5.6-luna`. Use `CODEX_MODEL` and `CODEX_JUDGE_MODEL` to override them separately. Focus on review comments with `bun run eval:write-naturally --filter-metadata group=review-regression --max-concurrency 2`.

Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';

const suiteRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');
const pluginRoot = resolve(suiteRoot, '../..');
const repositoryRoot = resolve(suiteRoot, '../../..');
const workspace = resolve(suiteRoot, 'workspace');
const workspaceAgents = resolve(workspace, '.agents');
const skillSource = resolve(pluginRoot, 'skills/write-naturally');
const skillSource = resolve(repositoryRoot, 'plugins/productivity/skills/write-naturally');
const skillTarget = resolve(workspaceAgents, 'skills/write-naturally');

export function workspaceHook(hookName, context) {
Expand Down
6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
"scripts": {
"validate": "bun run scripts/validate-catalogue.ts",
"test": "bun test",
"eval:validate": "promptfoo validate config -c plugins/productivity/evals/write-naturally/promptfooconfig.yaml",
"eval:write-naturally": "promptfoo eval -c plugins/productivity/evals/write-naturally/promptfooconfig.yaml --no-cache --no-share",
"eval:write-naturally:baseline": "bun run scripts/eval-writing-baseline.ts plugins/productivity/evals/write-naturally"
"eval:validate": "promptfoo validate config -c evals/productivity/write-naturally/promptfooconfig.yaml",
"eval:write-naturally": "promptfoo eval -c evals/productivity/write-naturally/promptfooconfig.yaml --no-cache --no-share",
"eval:write-naturally:baseline": "bun run scripts/eval-writing-baseline.ts evals/productivity/write-naturally"
},
"devDependencies": {
"@openai/codex-sdk": "0.162.1",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Productivity
# Write naturally

## Write naturally: practical gotchas
## Practical gotchas

- **Long conversations:** an agent can read the skill early and later drift from it. Before the final version, check which guidance it is applying. If the answer is vague or the prose has drifted, ask it to reread the skill and the relevant reference. A claim that it remembers the skill is weaker evidence than a fresh read.
- **Use your paraphrase:** if you need to ask “so you mean …?”, give the agent your plain-language interpretation, confirm the meaning, then ask it to put that explanation into the text. The next reader should understand it without repeating your clarification.
- **Separate wording from judgment:** clearer prose cannot establish whether a concern is a blocker. Settle the evidence and intended action, then ask for a final version that carries that decision and only the context its recipient needs.

For example: “So you mean a caller can send text where an integer is required, and validation accepts it?” Once confirmed: “Put that caller example directly in the comment, with what happens and the proposed follow-up.”

The [core skill](skills/write-naturally/SKILL.md) applies to general prose. It loads additional guidance for [reviews](skills/write-naturally/references/reviews.md) or [technical designs](skills/write-naturally/references/technical-designs.md) when needed. The [public eval suite](evals/write-naturally/promptfooconfig.yaml) uses fictional examples, including nontechnical writing. Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private).
The [core skill](SKILL.md) applies to general prose. It loads additional guidance for [reviews](references/reviews.md) or [technical designs](references/technical-designs.md) when needed. The [public eval suite](../../../../evals/productivity/write-naturally/promptfooconfig.yaml) uses fictional examples, including nontechnical writing. Additional coverage lives in the [private eval repository](https://github.com/allagentsdev/agent-plugins-evals-private).
2 changes: 1 addition & 1 deletion tests/unit/plugins/write-naturally-baseline.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { describe, expect, test } from 'bun:test';
import baselineNoTools from '../../../plugins/productivity/evals/write-naturally/assertions/baseline-no-tools.mjs';
import baselineNoTools from '../../../evals/productivity/write-naturally/assertions/baseline-no-tools.mjs';

const context = (items: unknown[]) => ({ providerResponse: { raw: JSON.stringify({ items }) } });
const response = { type: 'agent_message', text: 'Rewritten text.' };
Expand Down
Loading