Skip to content

Add provenance-checked PaCMAP and the shared semantic text view #910

Description

@realmarcin

The fleet embedding audit was revalidated against main 40f3a900b9ad0155062f03a47eeeac5cb49afab8: all 305 current structure records have pinned MiniLM 384D vectors, a PCA map and cosine neighbors, but no PaCMAP implementation. The visible map and SVG aria-label explicitly say PCA. Existing PaCMAP issue search returned no matches.

Coordinate with CultureBotAI/culturebotai-claw#426. Add the shared semantic-text space (BAAI/bge-large-en-v1.5 revision d4aa6901d3a41ba39fb536a557fa166f842b0e09 plus PaCMAP), while preserving the useful pinned MiniLM/PCA and cosine-neighbor view with accurate names. PaCMAP projects vectors and is a separate choice from the encoder; do not relabel MiniLM vectors as BGE.

Acceptance:

  • Implement real PaCMAP with pinned runtime, deterministic input order/seed and complete vector-source/model/reducer provenance; a first PaCMAP canary can use existing committed vectors without neural inference.
  • Introduce a versioned structure semantic adapter for the shared BGE view, with normalized vectors and keyed incremental cache reuse. Preserve unchanged vectors when source semantic hashes and model/adapter metadata are unchanged.
  • Generate and publish full-current-corpus artifacts for the common view; keep source-space distinctions and existing useful views explicit in visible text, selector/links and accessibility labels.
  • Validate expected model dimensions, complete IDs/source hashes, finite coordinates, source vector fingerprint and reducer parameters. Provide refresh/build/offline-verify recipes and tests; ordinary CI should not run neural inference or redownload weights.
  • Verify rendered map pages and all record links, and keep neighbors defined in their original high-dimensional space.

This issue does not duplicate or take over #909 / PR #907, which address unchanged-vector churn in the cellular bud membrane PR. The underlying main refresh() still encodes every record; coordinate a durable incremental implementation without interfering with that active branch.

Implementation sites: scripts/build_text_embedding_map.py, data/embeddings/, src/cellstructuremech/templates/embedding_map.html, scripts/render_pages.py, justfile, pyproject.toml, scripts/run_qc.py and tests/test_text_embedding_map.py.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions