Skip to content

fix: three silent-success bugs, plus lg report - #2

Merged
datj9 merged 16 commits into
mainfrom
fix/claude-array-json-and-report
Aug 16, 2026
Merged

datj9 merged 16 commits into
mainfrom
fix/claude-array-json-and-report

Conversation

@datj9

@datj9 datj9 commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Four bugs and one feature. Three of the four bugs are the same shape: loomgraph reported success while doing the wrong thing. Every one was found by running the tool, not by reading it — the unit suite was green throughout.

1. The claude adapter was broken against every real run

claude -p --output-format json now emits a JSON array of messages, not a single result object:

[ {type:"system",subtype:"init"}, {type:"assistant"}, {type:"rate_limit_event"},
  {type:"result",subtype:"success",is_error:false,total_cost_usd:0.2642395,result:"BANANA"} ]

parseClaudeJson read subtype off the array, got nothing, and failed every agent node with claude run ended with subtype unknown — recording $0.0000 for a run that had really spent $0.26.

The cost side is the dangerous half: maxUsd accounting went blind, so a retry loop could burn real money against a ceiling that believed it had spent nothing. This violated the AGENTS.md rule "Cost is recorded even when the run failed, because budget accounting depends on it."

Fix: take the last type: "result" element, keep the legacy single-object form working, and carry total_cost_usd onto every return path including failures. Fixtures are captured from a real invocation.

Worth knowing: claude --help still documents the old single-result shape. The contract and the behaviour disagree with no version note, which is exactly why AGENTS.md requires fixtures from a real binary rather than from docs.

2. Command nodes passed having run nothing

npm run lint --if-present exits 0 in a repo with no lint script. That node was recorded succeeded and satisfied a when: all_succeeded fan-in gate having verified nothing.

Adds two opt-in assertions to command nodes:

test:
  type: command
  run: "npm test"
  expectNonEmpty: true      # fails if the command wrote no output
  expect: "0 failed"        # fails if that substring is absent from stdout

3. A run could report success having blown its budget ceiling

checkBudget ran only before each dispatch batch, so the final batch could overshoot without limit. A real run:

budget  0.5371/0.3000 usd    status  succeeded    exit 0

179% of the ceiling, zero budget_exceeded events. The README promised "Hitting a ceiling stops the run with status failed... and exit code 3" — that never fired when the overshoot happened during the last batch rather than before one.

Fix: re-check all three ceilings before a run may finish succeeded. Completed nodes keep their results — the run fails, the work does not unwind:

a  succeeded   b  succeeded   c  succeeded
budget  3/3 node runs    status  failed    exit 3

4. Hyphenated node references passed through as literal text

The reference regex class was [A-Za-z0-9_.] — no hyphen. Node ids were unconstrained, so my-node was legal but its reference could not be tokenized:

consume received:  "hyphen=[{{nodes.my-node.output}}] underscore=[UNDERSCORE_OUTPUT]"

The node succeeded, the run exited 0. README.md promised "An unresolvable reference is an error, not an empty string" — it was neither. An agent node would have been billed to reason about a literal template placeholder.

Hyphenated ids are the natural style for this tool: run-tests, type-check, fix-bug.

Fix, in two parts: widen the class to accept -, and constrain node ids to [A-Za-z0-9_-]{1,64}. A dot is excluded because references split on ., making a dotted id structurally unrepresentable — now rejected at lg validate instead of meaning nothing at run time.

Together these give a property the README can now state: every template reference reachable from a valid graph either resolves or throws. No third outcome.

Feature: lg report

lg report <runId> [--out path] [--publish] [--title t] [--visibility private|org]

Renders a run's checkpoint and event log to one self-contained HTML file — zero external requests, every interpolated value escaped, deterministic output. --publish shells out to the enclave CLI to host it, the same way a node shells out to claude.

$ lg report hello-20260815-081338-k1t8 --publish
/…/.loomgraph/runs/hello-…/report/index.html
https://0c171730-….dat-nguyen.me/

A missing enclave binary is not a failed report — the file is still written, exit 0. This gives loomgraph a web view without a daemon, a server, or a cloud dependency, which keeps the "What this is not" section honest.

Verification

Tests 119 → 152, all passing, 15 files
Typecheck clean at every commit
Build clean at every commit
Live-verified agent adapter with a real billed run · lg report --publish with a real URL · budget gate at zero cost · hyphen fix with the original probe

Each of the 12 commits has a green npm run typecheck && npm test behind it. Escaping is tested against a hostile graph name and a node error containing a closing tag — a node's output reaches the HTML, so that was an injection path.

Notes for review

  • No new dependencies. No LLM SDK. Adapter contract { ok, text, costUsd, raw, error } unchanged.
  • src/adapters/types.ts untouched.
  • No test spawns a real agent CLI; enclave is never spawned in a test.
  • No cost number is estimated anywhere — adapters that report no price still record 0.
  • The node id restriction is the one change that can break an existing graph file. Only ids containing dots, spaces, or other non-[A-Za-z0-9_-] characters are affected; every id in examples/ is unaffected.

5. OpenCode was a permanent $0 blind spot inside every budget

The adapter used OpenCode's default output format, which prints prose to stdout, the model banner to stderr, and no price anywhere. So every opencode node recorded 0.0000 and could never move a maxUsd ceiling.

[RUN] Under --format json, opencode 1.18.17 emits JSONL and reports a real price:

{"type":"step_finish", "part":{"type":"step-finish",
  "tokens":{"total":58887,...}, "cost":0.003991708}}

The adapter now walks that stream — concatenating text events, summing part.cost from every step_finish, and keeping that cost on the failure paths too, because budget accounting depends on it. Fixture captured by hand from a real invocation.

This also corrects the README and the lg status footer, both of which claimed OpenCode reports no price.

6. You could not choose the model

AdapterInput gains an optional model, threaded to all three CLIs — --model for claude and codex, -m for opencode:

review:
  type: verifier
  adapter: opencode
  model: "opencode-go/deepseek-v4-flash"
  prompt: "Review the diff. Reply PASS or FAIL."
  pass: "PASS"

Without it, each CLI's own resolution decides — and that is not always what the config says. [RUN] With no -m, opencode ignored the configured model in opencode.json and fell through to claude-sonnet-4-5, which failed with No active credentials for provider: anthropic. OPENCODE_MODEL is ignored; the flag is the only way in.

For a tool that sells budgeted orchestration, this matters more than it looks: model choice is cost control.

A model on a command or human node is a validation error rather than a silently ignored key — zod strips unknown keys, so it would otherwise vanish without a word. That is the same silent-success family as bugs 1–4.

The OpenCode adapter is no longer experimental

It had never been executed against a real binary. It has now been, twice:

# before: could not run at all
ask  failed  opencode exited with code 1: No active credentials for provider: anthropic

# after: model selectable, cost captured
ask  succeeded  1  0.0001  39.4s
budget  0.0001/0.5000 usd

The first of those is worth keeping in mind during review: the adapter's error handling was already correct — it surfaced the real CLI failure verbatim rather than swallowing it. What was missing was the ability to pass a model at all.

Verification, updated

Tests 119 → 171
Commits 15, each independently green (verified by checking out each and running the suite)
Live-verified claude adapter (billed run) · lg report --publish (real URL) · budget gate (zero cost) · hyphen fix · opencode adapter with model + cost

datj9 added 16 commits August 15, 2026 07:55
execa 10 declares `engines: {"node": ">=22"}` and calls
`Set.prototype.union`, which does not exist before node 22. The ci matrix
still ran node 20, so every adapter test file crashed at import with
`TypeError: TEXT_ENCODINGS.union is not a function`.

Node 20 reached end of life in april 2026, so the fix is to drop it rather
than pin execa back.

- package.json engines: >=20 -> >=22
- ci matrix: [20, 22] -> [22, 24]
- tsup target: node20 -> node22
- readme and contributing state node >= 22
claude -p --output-format json now emits an array of message objects
instead of a single result. the parser read subtype off the array, failed
every agent node with "subtype unknown", and recorded 0 usd for a run
that had really spent money - so the maxUsd ceiling went blind.

take the last type=result element, keep the legacy single-object form,
and carry total_cost_usd onto every return path including failures.
a command that exits 0 having done nothing was recorded as succeeded.
npm run lint --if-present in a repo with no lint script is the case:
it passed a fan-in gate having verified nothing.
pure argv builder and json parser, tested against output captured from a
real enclave push. the binary is spawned without a shell so a title
cannot be reinterpreted as shell syntax.
lg report <runId> writes the checkpoint and event log to one html file
with no external requests. every interpolated value is escaped - a node
whose output contains a closing tag would otherwise inject markup.
--publish shells out to enclave; a missing binary still writes the file.
the ceiling was a pre-dispatch gate only, so the final batch could
overshoot without limit: a real run spent 0.5371 usd against a maxUsd of
0.30, finished succeeded, exited 0, and emitted no budget_exceeded event.

completed nodes keep their results - the run fails, the work does not
unwind.
the reference regex class excluded '-', so {{nodes.my-node.output}} never
matched and was left in the prompt verbatim. the node reported succeeded
and the run exited 0, contradicting the readme's promise that an
unresolvable reference is an error.
a dot collides with the reference syntax, which splits on '.', so a
dotted id is unrepresentable in a template. reject it at validate rather
than let it mean nothing at run time. the END check keeps its own message
and still runs first.
the model was whatever each cli's own resolution produced. with no -m,
opencode ignored the configured model and fell through to a provider with
no credentials, so the adapter could not run at all. OPENCODE_MODEL is
ignored - the flag is the only way in.

a model on a command or human node is a validation error: zod strips
unknown keys, so it would otherwise vanish silently.
… cost

opencode reports a price per step under --format json; the default format
prints prose and no price. loomgraph used the default, so every opencode
node recorded 0.0000 usd and was a permanent blind spot inside maxUsd.
@datj9
datj9 force-pushed the fix/claude-array-json-and-report branch from 0e567ed to ff3903f Compare August 16, 2026 23:17
@datj9
datj9 force-pushed the fix/node-22-floor branch from 75bdfd5 to 0f17c7c Compare August 16, 2026 23:17
@datj9
datj9 deleted the branch main August 16, 2026 23:38
@datj9 datj9 closed this Aug 16, 2026
@datj9 datj9 reopened this Aug 16, 2026
@datj9
datj9 changed the base branch from fix/node-22-floor to main August 16, 2026 23:39
@datj9
datj9 merged commit b0a4162 into main Aug 16, 2026
7 checks passed
@datj9
datj9 deleted the fix/claude-array-json-and-report branch August 16, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant