AI package for Open BRF: releases

The AI package is an Open BRF plugin that gives a housing cooperative's board assistants built on a language model, on the association's own model provider account and within a monthly limit the board sets.

It is a paid module from Apteo AB. Its source is not in this repository. This is a release-only repository, as Open BRF ADR 0044 describes: it holds the release workflow, and each release is built from the commit of the private source that source.json, at this repository's root, pins.

Installing

An instance installs the package from the Open BRF catalog once it is listed there. Each version's tarball is an asset of this repository's releases, and the board enters the instance's licence on the instance's licence screen. No version has been released yet.

What a release proves

• The tarball is attested by this repository's release workflow, in a run on the release's tag. Check it with gh attestation verify <file>.tgz --repo openbrf/ai-package --source-ref refs/tags/v<version> --signer-workflow openbrf/ai-package/.github/workflows/release.yml --deny-self-hosted-runners, then check that the commit it was attested from is on main and holds main's release workflow. A tag of a commit off main can carry a workflow of its own that attests and publishes a tarball without the pin checks or the evaluation (see "Releasing"). So can a tag of an earlier commit of main whose release workflow differs from main's, such as a commit that a pull request merged without squashing brought in. The attestation alone does not prove that the workflow on main made the tarball. A tarball can carry several attestations, and every commit they name has to pass. gh fetches at most 30 attestations unless it is given a larger --limit, so the script counts the file's attestations first and fetches all of them. gh also leaves out, without an error, an attestation whose predicate is not the build provenance, so the script refuses a result with fewer records than attestations. GitHub may read a bare main as a tag of that name, so the script asks for the branch by its full ref and compares with the commit id it gets.

  Save the script to a file and run it with sh check.sh, or give it to sh on its input (sh <<'EOF'). Do not paste it into your shell: set -eu and exit 1 would end an interactive shell, and zsh does not split $commits into words as sh does.

  set -eu
  # No globbing: nothing in the script is a pattern.
  set -f
  file=<file>.tgz
  # A tag can be named main, so ask for the branch by its full ref, and use its commit id
  # wherever the script would otherwise write main.
  m=$(gh api repos/openbrf/ai-package/git/ref/heads/main --jq .object.sha)
  case $m in
    *[!0123456789abcdef]*) echo "REFUSED: main is not a commit id"; exit 1 ;;
  esac
  [ "${#m}" -eq 40 ] || { echo "REFUSED: main is not a commit id"; exit 1; }
  # gh fetches 30 attestations unless told more, so count the file's, and ask for all of them.
  digest=$(sha256sum "$file" 2>/dev/null || shasum -a 256 "$file")
  # gh writes a page before it asks for the next, so a pipe from it would count the pages
  # written before a failure and hide the failure. Write to a file, which set -e checks.
  out=$(mktemp)
  trap 'rm -f "$out"' EXIT
  gh api --paginate "repos/openbrf/ai-package/attestations/sha256:${digest%% *}" \
    --jq '.attestations[] | 1' > "$out"
  count=$(wc -l < "$out" | tr -d ' ')
  [ "$count" -gt 0 ] || { echo "REFUSED: no attestation"; exit 1; }
  gh attestation verify "$file" --repo openbrf/ai-package \
    --limit "$count" \
    --source-ref refs/tags/v<version> \
    --signer-workflow openbrf/ai-package/.github/workflows/release.yml \
    --deny-self-hosted-runners \
    --format json \
    --jq '.[].verificationResult.signature.certificate.sourceRepositoryDigest' > "$out"
  verified=$(wc -l < "$out" | tr -d ' ')
  [ "$verified" -eq "$count" ] || { echo "REFUSED: $verified of $count attestations verified"; exit 1; }
  commits=$(sort -u "$out")
  [ -n "$commits" ] || { echo "REFUSED: no attested commit"; exit 1; }
  # Every line is a commit id, so no empty or blank record is lost to word splitting.
  if printf '%s\n' "$commits" | LC_ALL=C grep -qvxE '[0-9a-f]{40}'; then
    echo "REFUSED: not a commit id"; exit 1
  fi
  f=.github/workflows/release.yml
  for commit in $commits; do
    status=$(gh api "repos/openbrf/ai-package/compare/$m...$commit" --jq .status)
    case $status in
      identical | behind) ;;
      *) echo "NOT ON MAIN: $commit"; exit 1 ;;
    esac
    a=$(gh api "repos/openbrf/ai-package/contents/$f?ref=$commit" --jq .sha)
    b=$(gh api "repos/openbrf/ai-package/contents/$f?ref=$m" --jq .sha)
    [ -n "$a" ] && [ "$a" = "$b" ] || { echo "DIFFERENT: $f at $commit"; exit 1; }
  done
  echo "OK: every attested commit is on main and holds main's $f"

  The script stops at the first failure and exits with a non-zero status unless it prints OK as its last line. A failure prints REFUSED, NOT ON MAIN or DIFFERENT, or the error of a gh command that failed. Once main changes its release workflow, a release that the earlier workflow made reads DIFFERENT: that version is not listed, and a new version is released from main instead. The catalog's check of an attestation does not include this yet, so the reviewer of the catalog pull request that lists a version runs it.

  The check was written against gh v2.103.0 and verified with that version; use that or a later one. --deny-self-hosted-runners makes the verification fail for an attestation made on a self-hosted runner, which the release workflow never uses. A gh that does not know the flag fails on it, so update gh rather than drop the flag.

• The source is named by the source.json packed in the tarball, a copy of this repository's source.json at the tagged commit. It is Apteo's statement of the commit the tarball was built from, kept in this repository's public history. The workflow builds only a commit on the main branch of apteoab/openbrf-modules. Whoever can read the source can check out that commit and build it with the same steps.

• The prompt-injection evaluation passed on that commit. Residents write text the assistants read, and such text may try to make the model act. The package's evaluation runs a corpus of such text against every model the board may choose, and passes on what the model does, not on what it says. The workflow runs it on the commit it builds, and a case that fails stops the release before anything is attested or published.

Releasing

1. Open a pull request that moves source.json to the commit of apteoab/openbrf-modules to release:

   {
     "repository": "https://github.com/apteoab/openbrf-modules",
     "commit": "<full commit id>"
   }

2. Merge it, then push the tag v<version> from main while source.json there still pins that commit. The version is the one the pinned commit's package states.

3. Approve the run in the release environment three times: before it fetches the source, before it evaluates it, and before it builds it once the evaluation has passed. Before each approval, check that the run's commit, shown on the run's page, is on main:

   m=$(gh api repos/openbrf/ai-package/git/ref/heads/main --jq .object.sha)
   echo "$m"
   gh api repos/openbrf/ai-package/compare/$m...<commit> --jq .status

   main is asked for by its full ref, so a tag named main cannot stand in for it. The first line prints 40 hexadecimal characters; if it prints anything else, approve nothing. Run the next command in the same shell, as it uses $m.

   It prints identical for the head of main and behind for an earlier commit of main; approve nothing else. For a behind commit, approve only if main has changed neither the release workflow nor source.json since. Compare the two files' blob ids, not the compare API's list of changed files, which stops at 300 files:

   for f in .github/workflows/release.yml source.json; do
     a=$(gh api "repos/openbrf/ai-package/contents/$f?ref=<commit>" --jq .sha) &&
       b=$(gh api "repos/openbrf/ai-package/contents/$f?ref=$m" --jq .sha) &&
       [ "$a" = "$b" ] && echo "$f: same" || echo "$f: DIFFERENT, approve nothing"
   done

   Approve only if both lines read same.

4. Once the run has ended, however it ended, replace SOURCE_ARCHIVE_KEY. First check that no other release run is queued, waiting for approval or running: runs of different tags are not serialized, and a job reads the environment's secrets when it starts, so a run that is still waiting would start with the new key and fail to open the source sealed with the old one. The command prints nothing when no run is active:

   gh run list --repo openbrf/ai-package --workflow release.yml --limit 50 \
     --json databaseId,status,headBranch --jq '.[] | select(.status != "completed")'

   Replace the key once it prints nothing, or after the runs it lists have ended:

   openssl rand -base64 48 | gh secret set SOURCE_ARCHIVE_KEY --env release --repo openbrf/ai-package

GitHub runs the release workflow as it is at the tagged commit, so the workflow's own refusals below hold only for a commit of main: a tag of any other commit can carry a workflow without them. The approvals in step 3 gate the release environment's secrets and nothing else. Their check keeps such a run away from the secrets, and so from the private source and the model provider key, but a workflow that uses no environment asks nobody for approval: it can attest a tarball of its own and publish it. The tag ruleset limits who can push such a tag, and the check under "What a release proves" is what tells such a release apart.

The workflow refuses a tag that is not on main, whose source.json is not main's, or that does not match the version, and a pin that names a repository other than apteoab/openbrf-modules, the only one it fetches from, or a commit that is not on that repository's main branch. It fetches the pinned commit in a job that runs none of the source's code, then runs the package's prompt-injection evaluation in a job of its own. Only once that has passed does it run the package's typecheck and tests in another job and pack it with the pin copied in unchanged, so no tarball is made of a commit that failed the evaluation. Then it attests the tarball and publishes the release. Its job summary gives the sha512 and size for the catalog entry. Every release pays for one run of the evaluation, on the model provider key below.

The run logs of this repository are public, so the output of the source's dependency install, typecheck, tests, evaluation and build stays out of them. A failed release says which step failed; running that step in a checkout of the pinned commit shows why. Of the pinned commit, only the workspace's root files, its CI workflow and the package's own directory pass from the fetch job to the jobs that install the package's dependencies, so no other module's source reaches them. That source passes from one job to the next only encrypted, the run attests only the tarball its build job packed, and it deletes its artifacts, the tarball among them, when it ends.

The workflow relies on these repository settings:

• releases are immutable;
• a tag ruleset lets only the maintainers create a v* tag, and nobody move or delete one;
• a branch ruleset on main requires a pull request approved by someone other than its author and the Test the release workflow check, from the GitHub Actions app so that no other check or status of that name stands in for it, and refuses force pushes and deletion;
• the release environment allows only v* tags, and a maintainer approves each run's jobs in it. It holds three secrets:
  • SOURCE_DEPLOY_KEY, a read-only deploy key on apteoab/openbrf-modules. The key reads every module in that repository, not only this package;
  • SOURCE_ARCHIVE_KEY, a key of at least 256 random bits, such as the 384 bits openssl rand -base64 48 gives, that encrypts the source the run hands from one job to the next. It is replaced after every run (step 4 above), so a key read in one run opens no later run's source;
  • ANTHROPIC_API_KEY, a key of its own for the evaluation, in an Anthropic workspace of its own with a spend limit;
• the repository and its organisation hold no secret of these names outside that environment, and the repository has no other environment. Branches from before the source moved have an evaluation workflow of their own that reads ANTHROPIC_API_KEY from an injection-eval environment; that environment does not exist and is not to be created.

Licence

The AI package is proprietary software of Apteo AB.

Its first versions were developed in this repository under the AGPL-3.0. The history published here stays public under that licence: the main branch up to and including commit 1359fa9, and the branches of the pull requests opened here. Later development is in the private source and is not covered by it.

Contributing

Commit messages and pull request titles follow Conventional Commits. pnpm test checks which jobs of the release workflow can read each secret, and what passes from one job to the next.