feat: cortex.py library — ledger entry with the results-index hash (results-library 5b)

AGENTS.md

@@ -77,7 +77,7 @@ not re-derive them.
   ledger file"). Title `# <key> — <one-line summary>`; header `Project:` and
   `Issue:` (`Repo#N`, an issue URL, or `none`); then exactly three sections:
   `## Now`, `## Runs` (`open | running`), `## Log` (newest first, kind
-  `run | result | lesson | note`). Every `status: active` row must have one.
+  `run | result | lesson | note | library`). Every `status: active` row must have one.
   `projects/AGENTS.md` is the doctrine for the folder.
 - **`archive/`** — frozen: the 2026-08/09 task, ruling and batch ledgers,
   kept because the science repos cite their ids. Nothing writes there; any
@@ -107,6 +107,11 @@ not re-derive them.
     runs whose tasks started become `running`, runs whose tasks all ended go
     through `done` with the task counts. Matched by SLURM job id; never writes
     a `result`, `lesson` or `## Now` (decision 64).
+  - `library <project> <index_dir> [--name NAME] [--dry-run]` — one `library`
+    entry naming the PyAutoFit results index (`autofit.results`) a library was
+    built from: its hash (equal to `autofit.export.index_hash`) and its state
+    counts, read from the index files. Facts only; never a `result`, `lesson`
+    or `## Now`; a re-run appends another line (decision 65).
   - `log <project> "<text>" [--kind note|result|lesson]` — the human's words,
     dated today, at the head of the log.
   - `now <project> "<text>"` — rewrites `## Now`.

REFERENCE.md

@@ -29,7 +29,7 @@ project.
 
 ## Log
 
-- <YYYY-MM-DD> — <run | result | lesson | note> — <text>
+- <YYYY-MM-DD> — <run | result | lesson | note | library> — <text>
   an optional continuation line, indented two spaces
 ```
 
@@ -52,14 +52,15 @@ A job id appears once per file. A retired project lists no runs.
 ### Log lines (`## Log`)
 
 Newest first — `check` refuses a date that is later than the entry above it.
-Four kinds:
+Five kinds:
 
 | Kind | Written by | What it is |
 |---|---|---|
 | `run` | `cortex.py run` / `done` / `sync-runs` | `<jobid> submitted: <what>` · `<jobid> running: <what> — <counts> (digest …)` · `<jobid> finished — wall H:MM: <what>` · `<jobid> failed: <what>` |
 | `result` | `cortex.py log --kind result` | what the human read off the results, in their words |
 | `lesson` | `cortex.py log --kind lesson` | something the human wants kept — a trap, a rule, a fact about the setup |
 | `note` | `cortex.py log` (default), `new`, `retire` | anything else: a thought, a plan, `ledger opened`, `retired: <why>` |
+| `library` | `cortex.py library` only | `<name>: index <hash12> — <N> results (<state> <n>, …)` — the results index a library was built from (decision 65) |
 
 The board and the issue block show the newest `LOG_WINDOW = 5` entries; the
 file keeps them all. An entry is never edited or removed: a wrong entry is
@@ -96,6 +97,7 @@ creates an issue; `Issue:` is set with `new --issue` or `link`.
 | `issue <project> [-n N]` | nothing (prints) | — |
 | `link <project> <ref>` | the `Issue:` header line | the ref is not `Repo#N`, an issue URL or `none` |
 | `retire <project> --why "…"` | `projects.yaml` (`status: retired`, `note:`), one `note` entry | the row is already retired; the ledger still lists a run; `--why` holds `"` |
+| `library <project> <index_dir> [--name NAME] [--dry-run]` | one `library` entry at the head of the log | the folder or its `index.json` is missing; `index.json` is not schema 1; the key is not a row; a `--today` earlier than the head entry |
 | `sync-runs <project> [--digest PATH] [--dry-run]` | each run line the digest settles: `open` → `running` (+ a `run` entry), or `done` with the task counts | the digest is not a schema-1 `digest.json` with a readable `tasks.jsonl` beside it; no `--digest` and no `digest:` in the row |
 
 Every verb takes `--root <dir>` (default: this checkout) and every dated verb
@@ -166,6 +168,41 @@ from the scheduler's accounting. What the runs *mean* is the human's: a
 `result` or `lesson` entry and `## Now` are only ever written on their ask, and
 `sync-runs` writes neither (schema decision 64).
 
+## The results index a library was built from (`library`)
+
+A results library — a catalogue, a data release, a paper's fits — is built
+from a PyAutoFit results index (`autofit.results.ResultsIndex`): a folder
+holding `index.json` (schema 1) and `records/<sha1>.json`, one `ResultRecord`
+per result. `library <project> <index_dir>` reads those files and prepends one
+entry naming exactly which results the library was built from:
+
+```
+- 2026-09-03 — library — dr1 v1.0: index 6369f0eaac33 — 5 results (completed 2, failed 1, superseded 1, never_started 1)
+```
+
+- **The hash is `autofit.export.index_hash`**, computed from the JSON with no
+  autofit import: sha256 over the records sorted by `output_path`, each
+  contributing `<output_path>|<state>|<completed_at>\n` (a missing completion
+  time is `None`). The entry shows its first 12 hex digits; the verb prints
+  the full hash. As in `ResultsIndex.load`, a record that does not read, lacks
+  `output_path`/`root`/`state`, or is not `schema` 1 is skipped.
+- **The counts are the records' states**, non-zero only, in `ResultState`
+  order (`completed`, `running`, `interrupted_valid_ckpt`, `corrupt_ckpt`,
+  `corrupt_zip`, `failed`, `superseded`, `never_started`); a state outside it
+  follows, alphabetically. The `counts` in `index.json` are not read: the
+  records are the truth, and the hash is taken over them.
+- `--name` is what the entry calls the library; by default the index folder's
+  name, or its parent's when it is autofit's default `.autofit_index`.
+- An index whose `records/` is empty or absent is 0 results. A missing folder,
+  a missing or non-schema-1 `index.json`, or a key with no row refuses, exit 1,
+  nothing written.
+- **One entry per call.** A re-run appends another dated line; no entry is
+  ever edited. `--dry-run` prints the entry and writes nothing.
+
+**The boundary** is decision 60's: the entry is facts read from the index
+files, never inferred. `library` writes no `result` or `lesson`, never touches
+`## Now` or `## Runs` (schema decision 65).
+
 ## Repository layout
 
 ```

docs/schema_decisions.md

@@ -522,3 +522,26 @@ Choices made in phase 2 — the conductor, the board and the workflows
     hand-read `squeue`; the digest is one small file the collector already
     writes. Decision 60 still holds: nothing in a ledger is inferred from
     results, and a run's start and end are cluster facts.
+
+65. **A `library` entry records which results a library was built from**
+    (2026-11-02, PyAutoCortex#84; results-library phase 5b — decide-and-flag,
+    awaiting the human's ruling at merge). The log gains a fifth kind,
+    `library`, written only by `cortex.py library <project> <index_dir>
+    [--name NAME]` (`log --kind` cannot write it): `<name>: index <hash12> —
+    <N> results (<state> <n>, …)`, read from a PyAutoFit results index
+    (`autofit.results`: `index.json` schema 1 and `records/*.json`). The hash
+    is `autofit.export.index_hash`, recomputed from the JSON without
+    importing autofit: sha256 over the records sorted by `output_path`, each
+    contributing `<output_path>|<state>|<completed_at>\n`, skipping a record
+    that does not read, lacks a required field or is not `schema` 1, as
+    `ResultsIndex.load` does. The counts are the records' non-zero states in
+    `ResultState` order. Facts read from the index files only, never inferred
+    (decision 60): it never writes a `result` or a `lesson` and never touches
+    `## Now` or `## Runs`. One entry per invocation — a re-run appends another
+    dated line rather than editing one, as every entry is append-only.
+    Why: a release (a catalogue, a paper's fits) has to say which results it
+    was built from, and the export package already names them by this hash;
+    a dated ledger line with the same 12 digits ties the science history to
+    the package without anyone retyping a number. Like a run's start and end
+    (decision 64), which results an index holds is a fact a script may read;
+    what they mean stays the human's.

scripts/cortex.py

@@ -34,13 +34,15 @@
     python3 scripts/cortex.py link <project> <Repo#N | URL | none>  # set `Issue:`
     python3 scripts/cortex.py retire <project> --why "<one line>"   # a project's row
     python3 scripts/cortex.py sync-runs <project> --digest <digest.json> [--dry-run]
+    python3 scripts/cortex.py library <project> <index_dir> [--name NAME] [--dry-run]
 
 Exit codes: 0 = done · 1 = drift or a refused edit · 2 = bad arguments.
 """
 
 from __future__ import annotations
 
 import argparse
+import hashlib
 import json
 import re
 import sys
@@ -58,8 +60,11 @@
 SECTIONS = ("Now", "Runs", "Log")
 HEADER_KEYS = ("Project", "Issue")
 #: what an entry is: a run set off or finished (recorded by this script), a
-#: result the human read off, a lesson they want kept, or anything else.
-KINDS = ("run", "result", "lesson", "note")
+#: result the human read off, a lesson they want kept, anything else, or the
+#: results index a library was built from (`library`, decision 65).
+KINDS = ("run", "result", "lesson", "note", "library")
+#: the kinds `log --kind` may write; `library` is written only by `library`.
+LOG_KINDS = ("run", "result", "lesson", "note")
 #: a run on the cluster is `open` (submitted, not seen running) or `running`;
 #: there is no third state — a finished run is a log entry, not a run line.
 RUN_STATES = ("open", "running")
@@ -97,7 +102,7 @@
     r"^- (?P<ident>\d+(?:_\d+|_\[[\d,\-]+\])?) — (?P<state>open|running) — "
     r"(?P<partition>[a-z][a-z0-9_-]*) — (?P<date>\d{4}-\d{2}-\d{2}) — (?P<what>\S.*)$")
 LOG_LINE_RE = re.compile(
-    r"^- (?P<date>\d{4}-\d{2}-\d{2}) — (?P<kind>run|result|lesson|note) — (?P<text>\S.*)$")
+    r"^- (?P<date>\d{4}-\d{2}-\d{2}) — (?P<kind>" + "|".join(KINDS) + r") — (?P<text>\S.*)$")
 CONT_RE = re.compile(r"^  (?P<text>\S.*)$")
 HEADER_KEY_RE = re.compile(r"^([A-Z][A-Za-z0-9-]*):(?:[ \t]+(.*?))?[ \t]*$")
 HEADER_LINES = 12
@@ -579,8 +584,8 @@ def _prepend_entry(text: str, entry_line: str) -> str:
 
 
 def add_entry(root: Path, key: str, kind: str, text: str, today: date) -> str:
-    if kind not in KINDS:
-        raise CortexError(f"--kind must be one of {', '.join(KINDS)}")
+    if kind not in LOG_KINDS:
+        raise CortexError(f"--kind must be one of {', '.join(LOG_KINDS)}")
     text = _one_line(text, "the entry")
     led = _ledger_at(root, key)
     raw = led.path.read_text(encoding="utf-8")
@@ -1153,6 +1158,122 @@ def sync_runs(root: Path, key: str, digest_path: Path, today: date, *,
     return [head] + changes + notes
 
 
+# --------------------------------------------------------------------------- #
+# the results index a library was built from (`library`, decision 65)
+# --------------------------------------------------------------------------- #
+#: the schema of a PyAutoFit results index (`autofit.results.record.SCHEMA`).
+INDEX_SCHEMA = 1
+#: the fields `ResultRecord.from_dict` cannot do without.
+INDEX_RECORD_FIELDS = ("output_path", "root", "state")
+#: `autofit.ResultState`, in its own order — the order the counts are shown in;
+#: a state outside it follows, alphabetically.
+RESULT_STATES = ("completed", "running", "interrupted_valid_ckpt", "corrupt_ckpt",
+                 "corrupt_zip", "failed", "superseded", "never_started")
+#: the folder name `ResultsIndex` defaults to, so a bare default name says nothing.
+DEFAULT_INDEX_DIRNAME = ".autofit_index"
+
+
+def read_index_records(index_dir: Path) -> "list[dict]":
+    """The records of a PyAutoFit results index, as `ResultsIndex.records()`
+    returns them: `records/*.json`, skipping a file that does not read, lacks a
+    required field or is not `schema` 1, one per `output_path`, sorted by it.
+    A CortexError when the folder or its `index.json` is missing or unreadable;
+    an empty or absent `records/` is zero records."""
+    index_dir = Path(index_dir)
+    if not index_dir.is_dir():
+        raise CortexError(f"{index_dir}: no such results index folder")
+    manifest_path = index_dir / "index.json"
+    if not manifest_path.is_file():
+        raise CortexError(f"{index_dir}: no index.json — not a results index "
+                          f"(`python -m autofit.results index` writes one)")
+    try:
+        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+    except (OSError, ValueError) as e:
+        raise CortexError(f"{manifest_path}: not readable JSON ({e})")
+    if not isinstance(manifest, dict) or manifest.get("schema") != INDEX_SCHEMA:
+        raise CortexError(f"{manifest_path}: not a schema-{INDEX_SCHEMA} results index")
+    by_path: "dict[str, dict]" = {}
+    records_dir = index_dir / "records"
+    if records_dir.is_dir():
+        for p in sorted(records_dir.glob("*.json")):
+            try:
+                rec = json.loads(p.read_text(encoding="utf-8"))
+            except (OSError, ValueError):
+                continue
+            if not isinstance(rec, dict) or any(f not in rec for f in INDEX_RECORD_FIELDS):
+                continue
+            if rec.get("schema", INDEX_SCHEMA) != INDEX_SCHEMA:
+                continue
+            by_path[rec["output_path"]] = rec
+    return [by_path[k] for k in sorted(by_path)]
+
+
+def index_hash(records: "list[dict]") -> str:
+    """`autofit.export.index_hash`, from the JSON: sha256 over each record, in
+    output-path order, of `<output_path>|<state>|<completed_at>\\n` (a missing
+    completion time is `None`, as Python formats it)."""
+    digest = hashlib.sha256()
+    for rec in records:
+        digest.update(f"{rec['output_path']}|{rec['state']}|{rec.get('completed_at')}\n"
+                      .encode())
+    return digest.hexdigest()
+
+
+def _state_counts(records: "list[dict]") -> str:
+    from collections import Counter
+    c: "Counter[str]" = Counter(str(rec["state"]) for rec in records)
+    order = [s for s in RESULT_STATES if c[s]] + sorted(s for s in c if s not in RESULT_STATES)
+    return ", ".join(f"{s} {c[s]}" for s in order)
+
+
+def _index_name(index_dir: Path) -> str:
+    """The default `--name`: the index folder's name, or its parent's when it is
+    the default `.autofit_index`."""
+    d = Path(index_dir).resolve()
+    return d.parent.name if d.name == DEFAULT_INDEX_DIRNAME else d.name
+
+
+def library_entry(index_dir: Path, name: str) -> "tuple[str, str]":
+    """The entry text for a results index and its full hash:
+    `<name>: index <hash12> — <N> results (<state> <n>, …)`."""
+    records = read_index_records(index_dir)
+    digest = index_hash(records)
+    n = len(records)
+    text = f"{name}: index {digest[:12]} — {n} result{'' if n == 1 else 's'}"
+    if records:
+        text += f" ({_state_counts(records)})"
+    return text, digest
+
+
+def record_library(root: Path, key: str, index_dir: Path, today: date, *,
+                   name: "str | None" = None, dry_run: bool = False) -> "list[str]":
+    """Append one `library` entry naming the results index a library was built
+    from — facts read from the index files only (decision 65). One entry per
+    call: a re-run appends another dated line, it never edits one. Made in
+    memory, read back, written once; never `result`, `lesson` or `## Now`.
+    Returns the lines to print, the first `library: <project>: <entry>`."""
+    rows, _ = load_projects(root)
+    if PROJECT_KEY_RE.match(key or "") and key not in rows:
+        raise CortexError(f"{key} is not a projects.yaml key")
+    name = _one_line(name if name is not None else _index_name(index_dir), "--name")
+    text, digest = library_entry(index_dir, name)
+    led = _ledger_at(root, key)
+    stamp = today.isoformat()
+    if not dry_run:
+        if led.log and stamp < led.log[0].date:
+            raise CortexError(f"the log's head is {led.log[0].date}; an entry dated {stamp} "
+                              f"would break newest-first — nothing written")
+        new = _prepend_entry(led.path.read_text(encoding="utf-8"),
+                             f"- {stamp} — library — {text}")
+        after = parse_ledger(new, led.key, led.path, led.rel)
+        if after.problems:
+            raise CortexError("the edit would not read back — nothing written: "
+                              + "; ".join(after.problems))
+        led.path.write_text(new, encoding="utf-8")
+    head = f"library: {key}: {text}" + (" (dry run)" if dry_run else "")
+    return [head, f"  index sha256 {digest}"]
+
+
 # --------------------------------------------------------------------------- #
 # CLI
 # --------------------------------------------------------------------------- #
@@ -1212,6 +1333,13 @@ def cmd_sync_runs(args) -> int:
     return 0
 
 
+def cmd_library(args) -> int:
+    for line in record_library(args.root, args.project, Path(args.index_dir), _today(args),
+                               name=args.name, dry_run=args.dry_run):
+        print(line)
+    return 0
+
+
 def cmd_retire(args) -> int:
     print(retire_project(args.root, args.project, args.why, _today(args)))
     return 0
@@ -1266,7 +1394,7 @@ def build_parser() -> argparse.ArgumentParser:
     p = sub.add_parser("log", help="write a dated entry in the human's words")
     p.add_argument("project")
     p.add_argument("text")
-    p.add_argument("--kind", default="note", choices=KINDS)
+    p.add_argument("--kind", default="note", choices=LOG_KINDS)
     _common(p, dated=True)
     p.set_defaults(func=cmd_log)
 
@@ -1317,6 +1445,15 @@ def build_parser() -> argparse.ArgumentParser:
     _common(p, dated=True)
     p.set_defaults(func=cmd_sync_runs)
 
+    p = sub.add_parser("library", help="log the results index a library was built from "
+                                       "(its hash and state counts)")
+    p.add_argument("project")
+    p.add_argument("index_dir", help="a PyAutoFit results index folder (holds index.json)")
+    p.add_argument("--name", help="what the entry calls it (default: the index folder's name)")
+    p.add_argument("--dry-run", action="store_true", help="print the entry; write nothing")
+    _common(p, dated=True)
+    p.set_defaults(func=cmd_library)
+
     p = sub.add_parser("retire", help="retire a project's row (with `birth`, the only verbs "
                                       "that write projects.yaml)")
     p.add_argument("project")

tests/test_cortex.py

@@ -419,7 +419,7 @@ def test_retire_preserves_every_other_byte_of_projects_yaml(tmp_path):
 def test_every_verb_is_registered():
     parser = cortex.build_parser()
     verbs = set(parser._subparsers._group_actions[0].choices)
-    assert verbs == {"check", "new", "run", "running", "done", "log", "now", "issue", "link", "retire", "birth", "sync-runs"}
+    assert verbs == {"check", "new", "run", "running", "done", "log", "now", "issue", "link", "retire", "birth", "sync-runs", "library"}
 
 
 def test_a_bad_root_is_a_usage_error(tmp_path):