HabitatMech
HabitatFour habitat vocabularies harmonized into ENVO-grounded records that keep every source's attestation.
3,206habitat records · 684 reviewed
+
ENVONCBITaxonBTOUBERONFOODONGOLDBacDive
The Mechs are joinable because they ground records in the same public ontologies. The table counts identifier occurrences per vocabulary in each Mech's record corpus; darker cells mean more. Columns run from the most widely shared vocabulary to the least, so the left edge is the fleet's common ground and the right edge is what a single Mech needs alone. Click a Mech name to open it, a cell to list the records behind it, or a column heading to filter the graph to that vocabulary.
+ +This earlier September 2026 vocabulary census was not recomputed with the September 15 site refresh and covers nine Mechs; TaxonMech is a fleet member whose vocabulary census has not yet been measured. Counts are prefix occurrences in the canonical record directories (merged recipes for CultureMech, communities for CommunityMech, habitat records for HabitatMech) as of September 2026. Columns are ordered by how many Mechs ground anything in each vocabulary, then by the total records citing it across the fleet; PMID and DOI sit at the right because every Mech cites literature. ChEBI binds the chemistry arm (media, ingredients, antibiotics, proteins); NCBITaxon and ENVO bind the organism arm (habitat, community, traits); GO and METPO bridge phenotype, structure and protein.
+Four habitat vocabularies harmonized into ENVO-grounded records that keep every source's attestation.
Curated knowledge base of microbial communities, their interactions, cultivation conditions and evidence.
Microbial taxa and strains identified by NCBI Taxonomy, harmonized with GTDB, LPSN and BacDive, with evidence for strain-to-genome links.
Microbial ecophysiological trait knowledge base seeded from METPO, one curated YAML per trait, with causal mechanism graphs.
Organelles, envelope layers, appendages, microcompartments and complexes: components, distribution, function and causal mechanism.
Protein sequence, structure and function trait classes seeded from InterPro, Pfam, Rhea, CATH, SCOPe, CARD and more.
One record per natural product structure: who makes it, from which gene cluster, what it does, and the evidence for all three.
One record per antimicrobial chemical structure, harmonizing ChEBI and CARD with targets, mode of action, resistance and evidence.
LLM-assisted curation of media-ingredient ontology mappings with full audit trails; owns ingredient identity for the fleet.
Versioned, ontology-grounded knowledge base of culture-media recipes from MediaDive, TogoMedium, KOMODO and the major collections.
This earlier September 2026 vocabulary census was not recomputed with the September 15 site refresh and covers nine Mechs; TaxonMech is a fleet member whose vocabulary census has not yet been measured. Counts are prefix occurrences in the canonical record directories (merged recipes for CultureMech, communities for CommunityMech, habitat records for HabitatMech) as of September 2026. Columns are ordered by how many Mechs ground anything in each vocabulary, then by the total records citing it across the fleet; PMID and DOI sit at the right because every Mech cites literature. ChEBI binds the chemistry arm (media, ingredients, antibiotics, proteins); NCBITaxon and ENVO bind the organism arm (habitat, community, traits); GO and METPO bridge phenotype, structure and protein.
## How the Mechs reference each other diff --git a/mechs.md b/mechs.md index 3dff2ae..5ffddd9 100644 --- a/mechs.md +++ b/mechs.md @@ -19,6 +19,7 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases tha identify the nodes without relying on color alone. */ --mech-naturalproductmech: #0b5fa5; --mech-taxonmech: #5257C9; --mech-mediaingredientmech: #7E5BC4; --mech-culturemech: #4B9E5F; + --sphere-hi: .42; --sphere-hi-mid: .12; --sphere-lo-mid: .12; --sphere-lo: .21; --fleet-edge: rgba(90, 99, 94, .34); --fleet-heat-hue: 150 22%; --voc-chebi: #D9702F; --voc-ncbitaxon: #3B7DD8; --voc-go: #2FA36B; --voc-metpo: #7A5BC7; @@ -30,6 +31,7 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases tha --mech-cellstructuremech: #8a8ef5; --mech-proteintraitsmech: #3dbfb2; --mech-antibioticmech: #d9a94a; --mech-naturalproductmech: #79b8f3; --mech-taxonmech: #8b8fe8; --mech-mediaingredientmech: #b08cf2; --mech-culturemech: #63c46f; + --sphere-hi: .34; --sphere-hi-mid: .09; --sphere-lo-mid: .26; --sphere-lo: .44; --fleet-edge: rgba(160, 178, 168, .34); --fleet-heat-hue: 150 14%; --voc-chebi: #E8925A; --voc-ncbitaxon: #6FA6F2; --voc-go: #5CC48F; --voc-metpo: #A48CE8; @@ -41,6 +43,7 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases tha --mech-cellstructuremech: #8a8ef5; --mech-proteintraitsmech: #3dbfb2; --mech-antibioticmech: #d9a94a; --mech-naturalproductmech: #79b8f3; --mech-taxonmech: #8b8fe8; --mech-mediaingredientmech: #b08cf2; --mech-culturemech: #63c46f; + --sphere-hi: .34; --sphere-hi-mid: .09; --sphere-lo-mid: .26; --sphere-lo: .44; --fleet-edge: rgba(160, 178, 168, .34); --fleet-heat-hue: 150 14%; --voc-chebi: #E8925A; --voc-ncbitaxon: #6FA6F2; --voc-go: #5CC48F; --voc-metpo: #A48CE8; @@ -48,11 +51,16 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases tha } /* ---- Stat strip ---- */ - .fleet-stats { display: grid; grid-template-columns: repeat(4, 1fr); gap: .8rem; margin: 1.2rem 0 .4rem; } + /* Five tiles need explicit breakpoints, not auto-fit. auto-fit picks the most + columns that fit, which between roughly 700px and 1010px is four, leaving + the fifth tile alone on a row and stretched to the full width. Stepping + 5 -> 3 -> 2 keeps every row either full or short by one. */ + .fleet-stats { display: grid; grid-template-columns: repeat(5, 1fr); gap: .8rem; margin: 1.2rem 0 .4rem; } + @media (max-width: 1010px) { .fleet-stats { grid-template-columns: repeat(3, 1fr); } } .fleet-stats > div { padding: .9rem 1rem; background: var(--card); border: 1px solid var(--line); border-radius: 12px; } .fleet-stats b { display: block; font-size: 1.7rem; font-weight: 800; letter-spacing: -.02em; line-height: 1.1; font-variant-numeric: tabular-nums; color: var(--ink); } - .fleet-stats span { font-size: .8rem; color: var(--muted); text-transform: uppercase; letter-spacing: .06em; } - @media (max-width: 640px) { .fleet-stats { grid-template-columns: repeat(2, 1fr); } } + .fleet-stats span { display: block; font-size: .72rem; line-height: 1.35; color: var(--muted); text-transform: uppercase; letter-spacing: .04em; } + @media (max-width: 640px) { .fleet-stats { grid-template-columns: repeat(2, 1fr); } .fleet-stats b { font-size: 1.45rem; } } /* ---- Graph shell ---- */ .fleet-graph { margin: 1.4rem 0 2rem; background: var(--card); border: 1px solid var(--line); border-radius: var(--radius); box-shadow: var(--shadow); overflow: hidden; } @@ -83,6 +91,27 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases tha .fleet-stage a.node-link { outline: none; } .fleet-stage a.node-link:hover .node-label, .fleet-stage a.node-link:focus-visible .node-label { text-decoration: underline; text-decoration-thickness: 2px; text-underline-offset: 3px; } .fleet-stage .node circle { stroke: var(--card); stroke-width: 3; transition: r .2s ease, opacity .2s ease; } + /* The shading circles sit on top of the coloured disc and must not eat its + pointer events, or hovering a node would hit the overlay and the tooltip + would never open. They carry no stroke, so the base circle keeps drawing + the single outline. */ + .fleet-stage .node circle.shade { stroke: none; pointer-events: none; } + /* The gradients are shaped once in the defs and weighted here, because the + weighting has to change with the theme. Light-theme Mech colours are much + darker than dark-theme ones -- they carry their contrast against white, so + HabitatMech is #2b6a4d on light against #4fbf85 on dark -- and a limb + tuned for the dark theme turned them to mud. On light the highlight does + most of the modelling; on dark the shadow does. + + Both peaks are kept low because the highlight washes every node toward + white, which pulls the ten Mech colours toward each other: at .52 the + closest dark-theme pair fell to 57% of its flat separation, and the pair + it hurt most was the one already too close (#78). At .34 it holds 77% + while the limb still models the sphere (#79). */ + #fleet-sheen stop:nth-child(1) { stop-opacity: var(--sphere-hi); } + #fleet-sheen stop:nth-child(2) { stop-opacity: var(--sphere-hi-mid); } + #fleet-limb stop:nth-child(2) { stop-opacity: var(--sphere-lo-mid); } + #fleet-limb stop:nth-child(3) { stop-opacity: var(--sphere-lo); } .fleet-stage .node.adjacent circle { stroke-dasharray: 5 4; stroke: var(--muted); } .fleet-stage a.node-link:focus-visible circle { stroke: var(--neon); } .fleet-stage .node.dim, .fleet-stage .edge.dim { opacity: .18; } @@ -150,6 +179,10 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases tha .mech-card .num { display: flex; align-items: baseline; gap: .45rem; font-variant-numeric: tabular-nums; } .mech-card .num b { font-size: 1.45rem; font-weight: 800; letter-spacing: -.02em; color: var(--ink); } .mech-card .num span { font-size: .82rem; color: var(--muted); } + /* Small print under the headline number: what the corpus has been through + rather than how big it is. Muted and a size down, so it reads as a footnote + to the number above it and not as a second headline. */ + .mech-card .prov { margin: -.15rem 0 0; font-size: .76rem; color: var(--muted); font-variant-numeric: tabular-nums; } .mech-card .vocab { display: flex; flex-wrap: wrap; gap: .3rem; } .mech-card .vocab span { font-size: .72rem; font-weight: 700; padding: .25em .55em; border-radius: 6px; background: var(--wash-b); color: var(--ink); border: 1px solid var(--line); } .mech-card .row { display: flex; flex-wrap: wrap; gap: .35rem .5rem; align-items: center; margin-top: auto; padding-top: .3rem; } @@ -280,6 +313,7 @@ The X-Mech suite is a fleet of 10 curated, ontology-grounded knowledge bases thaThe Mechs are joinable because they ground records in the same public ontologies. The table counts identifier occurrences per vocabulary in each Mech's record corpus; darker cells mean more. Columns run from the most widely shared vocabulary to the least, so the left edge is the fleet's common ground and the right edge is what a single Mech needs alone. Click a Mech name to open it, a cell to list the records behind it, or a column heading to filter the graph to that vocabulary.
+ +This earlier September 2026 vocabulary census was not recomputed with the September 15 site refresh and covers nine Mechs; TaxonMech is a fleet member whose vocabulary census has not yet been measured. Counts are prefix occurrences in the canonical record directories (merged recipes for CultureMech, communities for CommunityMech, habitat records for HabitatMech) as of September 2026. Columns are ordered by how many Mechs ground anything in each vocabulary, then by the total records citing it across the fleet; PMID and DOI sit at the right because every Mech cites literature. ChEBI binds the chemistry arm (media, ingredients, antibiotics, proteins); NCBITaxon and ENVO bind the organism arm (habitat, community, traits); GO and METPO bridge phenotype, structure and protein.
+Four habitat vocabularies harmonized into ENVO-grounded records that keep every source's attestation.
684 reviewed · 130 merged PRs
Curated knowledge base of microbial communities, their interactions, cultivation conditions and evidence.
528 merged PRs
Microbial taxa and strains identified by NCBI Taxonomy, harmonized with GTDB, LPSN and BacDive, with evidence for strain-to-genome links.
0 reviewed · 15 merged PRs
Microbial ecophysiological trait knowledge base seeded from METPO, one curated YAML per trait, with causal mechanism graphs.
427 reviewed · 517 merged PRs
Organelles, envelope layers, appendages, microcompartments and complexes: components, distribution, function and causal mechanism.
0 reviewed · 425 merged PRs
Protein sequence, structure and function trait classes seeded from InterPro, Pfam, Rhea, CATH, SCOPe, CARD and more.
39,591 reviewed · 314 merged PRs
One record per natural product structure: who makes it, from which gene cluster, what it does, and the evidence for all three.
0 reviewed · 80 merged PRs
One record per antimicrobial chemical structure, harmonizing ChEBI and CARD with targets, mode of action, resistance and evidence.
80 reviewed · 160 merged PRs
LLM-assisted curation of media-ingredient ontology mappings with full audit trails; owns ingredient identity for the fleet.
359 merged PRs
Versioned, ontology-grounded knowledge base of culture-media recipes from MediaDive, TogoMedium, KOMODO and the major collections.
250 merged PRs
This earlier September 2026 vocabulary census was not recomputed with the September 15 site refresh and covers nine Mechs; TaxonMech is a fleet member whose vocabulary census has not yet been measured. Counts are prefix occurrences in the canonical record directories (merged recipes for CultureMech, communities for CommunityMech, habitat records for HabitatMech) as of September 2026. Columns are ordered by how many Mechs ground anything in each vocabulary, then by the total records citing it across the fleet; PMID and DOI sit at the right because every Mech cites literature. ChEBI binds the chemistry arm (media, ingredients, antibiotics, proteins); NCBITaxon and ENVO bind the organism arm (habitat, community, traits); GO and METPO bridge phenotype, structure and protein.
## How the Mechs reference each other diff --git a/scripts/fleet/assemble_page.py b/scripts/fleet/assemble_page.py index 18dca78..d1c6c79 100644 --- a/scripts/fleet/assemble_page.py +++ b/scripts/fleet/assemble_page.py @@ -36,7 +36,7 @@ def script_json(value): return json.dumps(value, ensure_ascii=False, separators=(",", ":")).replace("<", "\\u003c") -def assemble(template, fragment, data, snapshot): +def assemble(template, fragment, data, snapshot, stats): validate(snapshot) names = set(snapshot["mechs"]) badges = re.findall(r"", template) @@ -68,13 +68,27 @@ def assemble(template, fragment, data, snapshot): raise ValueError("Expected one fleet fragment") page = template.replace("", fragment) source = snapshot["source"] + # mech_stats.py follows the same manifest, so every card is covered; an + # admission that reached the manifest but not a recount would otherwise + # leave a stat line unfilled, which the placeholder sweep below catches. + counted = {m["mech"] for m in stats["mechs"]} + if counted != names: + raise ValueError("Mech stats must cover canonical fleet membership exactly") tokens = { "": str(len(names)), + "": f"{stats['merged_prs_total']:,}", "": str(snapshot["artifact_count"]), "": f'CLAW fleet manifest at {escape(source["revision"][:7])}', "": capability_rows(snapshot), } tokens.update({f"": 'in fleet manifest' for name in names}) + for mech in stats["mechs"]: + prs = f"{mech['merged_prs']:,} merged PRs" + # A null reviewed count means the Mech's schema has no status that can + # say REVIEWED, which is not the same as nothing having been reviewed, + # so the card says nothing rather than zero. See mech_stats.py. + line = prs if mech["reviewed"] is None else f"{mech['reviewed']:,} reviewed \u00b7 {prs}" + tokens[f""] = line for token, value in tokens.items(): if token not in page: raise ValueError(f"Missing source token {token}") @@ -93,7 +107,8 @@ def main(): page = assemble((FLEET / "mechs_template.md").read_text(), (FLEET / "fleet_fragment.html").read_text(), json.loads((FLEET / "data/fleet_data.json").read_text()), - json.loads((FLEET / "data/manifest.json").read_text())) + json.loads((FLEET / "data/manifest.json").read_text()), + json.loads((FLEET / "data/mech_stats.json").read_text())) target = REPO / "mechs.md" if args.check: if target.read_text() != page: diff --git a/scripts/fleet/mech_stats.py b/scripts/fleet/mech_stats.py new file mode 100644 index 0000000..e731cff --- /dev/null +++ b/scripts/fleet/mech_stats.py @@ -0,0 +1,158 @@ +"""Count reviewed records per Mech and merged pull requests per repository. + +Run from the site root: `python3 scripts/fleet/mech_stats.py`. Writes +_fleet/data/mech_stats.json, which assemble_page.py substitutes into the stat +strip and the Mech cards. Needs the checkouts (see roots.py) and a `gh` that +can read the CultureBotAI repositories; pass --no-prs to recount records only +and keep the pull-request numbers already on file. + +Two facts per Mech, and they come from different places: + +- Reviewed records are counted here, from the same record files the rest of the + pipeline reads. Whether a Mech tracks review at all is decided by its LinkML + schema, not by the values its records happen to carry: six declare a status + slot whose enum permits REVIEWED, and for those a count of zero is a real + zero. MediaIngredientMech and CultureMech do have a mapping_status, but its + enum runs UNMAPPED to AMBIGUOUS and never reaches REVIEWED, so it grades + mapping completeness rather than review; CommunityMech has no such slot. + Those three are recorded as null rather than zero, because "nobody has + reviewed one" and "this Mech does not track review" are different claims and + the page should not make the second look like the first. +- Merged pull requests are asked of GitHub, since the local checkout knows only + the branch it is on. The count is every merged pull request in the + repository's history, curation and automation alike: seeding runs, + regeneration and dependency updates land the same way human curation does. + It measures development activity on a Mech, not how much of it was human. +""" +from __future__ import annotations + +import datetime as _dt +import glob +import json +import os +import re +import subprocess +import sys + +try: + import yaml +except ModuleNotFoundError: # the only pipeline script that needs it; see #68 + raise SystemExit( + "mech_stats.py needs PyYAML to read each Mech's schema and find the slot " + "that records review. Install it with `pip install pyyaml`." + ) + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from roots import RECORD_GLOBS, mech_root, record_paths + +REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +OUT = os.path.join(REPO, "_fleet", "data", "mech_stats.json") + +# The repository name is not always the Mech name: ProteinTraitsMech publishes +# from a lowercase repo, and GitHub redirects the mixed-case form, which the +# search API does not follow. +GH_REPO = {"ProteinTraitsMech": "proteintraitsmech"} + +# Fleet membership comes from the manifest snapshot, the same source +# assemble_page.py validates the cards against, so a Mech admitted to the fleet +# cannot end up with a card and no stat line. roots.ORDER is the *census* order +# and is a narrower thing: it is the Mechs the vocabulary scans have measured. +MANIFEST = json.load(open(os.path.join(REPO, "_fleet", "data", "manifest.json"), encoding="utf-8")) +MEMBERS = list(MANIFEST["mechs"]) + +# Record globs for members the census has not reached yet. Counting reviewed +# records needs only a corpus, not a vocabulary scan, so these can be reported +# before prefix_census.py and build_subsets.py are rerun to include them. +EXTRA_GLOBS = {"TaxonMech": ["data/taxa/**/*.yaml"]} +RECORD_GLOBS.update({m: g for m, g in EXTRA_GLOBS.items() if m not in RECORD_GLOBS}) + +# Top-level only. A nested entry_status or a reviewed: true deeper in the file +# describes one entry inside a record, not the record, and counting those would +# report more reviewed records than a Mech has records. +STATUS = re.compile(r"^(mapping_status|curation_status)\s*:\s*[\"']?([A-Z_]+)[\"']?\s*$", re.M) + + +def review_slot(mech: str) -> str | None: + """The record field whose schema enum permits REVIEWED, if the Mech has one. + + Asking the schema rather than the records is what separates a Mech with + nothing reviewed yet from a Mech that grades something else entirely. + MediaIngredientMech and CultureMech would otherwise report zero reviewed + off a mapping_status that cannot take the value. + """ + root = mech_root(mech) + slots: dict = {} + enums: dict = {} + for path in sorted(glob.glob(os.path.join(root, "src", "**", "schema", "*.yaml"), recursive=True)): + try: + doc = yaml.safe_load(open(path, encoding="utf-8")) or {} + except yaml.YAMLError: + continue + slots.update(doc.get("slots") or {}) + enums.update(doc.get("enums") or {}) + for cls in (doc.get("classes") or {}).values(): + for name, defn in (cls.get("attributes") or {}).items(): + slots.setdefault(name, defn) + for field in ("mapping_status", "curation_status"): + defn = slots.get(field) + rng = defn.get("range") if isinstance(defn, dict) else None + allowed = (enums.get(rng) or {}).get("permissible_values") or {} + if "REVIEWED" in allowed: + return field + return None + + +def review_census(mech: str) -> tuple[int, int | None, str | None]: + """Records, reviewed records, and the field that said so.""" + paths = record_paths(mech) + field = review_slot(mech) + if field is None: + return len(paths), None, None + reviewed = 0 + for path in paths: + for key, value in STATUS.findall(open(path, encoding="utf-8", errors="replace").read()): + if key == field: + reviewed += value == "REVIEWED" + break + return len(paths), reviewed, field + + +def merged_prs(mech: str) -> int: + repo = GH_REPO.get(mech, mech) + out = subprocess.run( + ["gh", "api", "-X", "GET", "search/issues", "-f", + f"q=repo:CultureBotAI/{repo} is:pr is:merged", "--jq", ".total_count"], + capture_output=True, text=True, + ) + if out.returncode != 0: + raise SystemExit(f"{mech}: gh could not count merged PRs for {repo}: {out.stderr.strip()}") + return int(out.stdout.strip()) + + +def main() -> None: + keep_prs = "--no-prs" in sys.argv + old = {} + if keep_prs and os.path.exists(OUT): + old = {m["mech"]: m for m in json.load(open(OUT))["mechs"]} + + mechs = [] + for name in MEMBERS: + records, reviewed, field = review_census(name) + prs = old[name]["merged_prs"] if keep_prs and name in old else merged_prs(name) + mechs.append({"mech": name, "repo": GH_REPO.get(name, name), "records": records, + "reviewed": reviewed, "status_field": field, "merged_prs": prs}) + shown = "not tracked" if reviewed is None else f"{reviewed:,} reviewed" + print(f"{name:<22} {records:>8,} records {shown:<16} {prs:>5,} merged PRs") + + blob = { + "as_of": _dt.date.today().isoformat(), + "merged_prs_total": sum(m["merged_prs"] for m in mechs), + "mechs": mechs, + } + json.dump(blob, open(OUT, "w"), indent=1, ensure_ascii=False) + print(f"\n{blob['merged_prs_total']:,} merged PRs across {len(mechs)} Mechs -> {OUT}") + + +if __name__ == "__main__": + main() diff --git a/tests/test_fleet_page.py b/tests/test_fleet_page.py index 94b4467..ee678ae 100644 --- a/tests/test_fleet_page.py +++ b/tests/test_fleet_page.py @@ -19,9 +19,24 @@ def setUp(self): self.template = (ROOT / "_fleet/mechs_template.md").read_text() self.fragment = (ROOT / "_fleet/fleet_fragment.html").read_text() self.data = json.loads((ROOT / "_fleet/data/fleet_data.json").read_text()) + self.stats = json.loads((ROOT / "_fleet/data/mech_stats.json").read_text()) def render(self): - return assemble(self.template, self.fragment, self.data, self.snapshot) + return assemble(self.template, self.fragment, self.data, self.snapshot, self.stats) + + def test_card_stats_must_cover_every_fleet_member(self): + self.stats["mechs"] = [m for m in self.stats["mechs"] if m["mech"] != "TaxonMech"] + with self.assertRaisesRegex(ValueError, "Mech stats"): + self.render() + + def test_a_mech_that_cannot_record_review_shows_only_its_pull_requests(self): + page = self.render() + by_name = {m["mech"]: m for m in self.stats["mechs"]} + untracked = next(m for m in by_name.values() if m["reviewed"] is None) + tracked = next(m for m in by_name.values() if m["reviewed"] is not None) + self.assertIn(f'{untracked["merged_prs"]:,} merged PRs
', page) + self.assertIn(f'{tracked["reviewed"]:,} reviewed \u00b7 {tracked["merged_prs"]:,} merged PRs', page) + self.assertNotIn("0 reviewed \u00b7 " + f'{untracked["merged_prs"]:,}', page) def test_published_page_contains_both_new_members_with_distinct_capabilities(self): page = self.render()